Compression on IB Live data

bb2

Hello,

I am struggling with data collection from IB under some conditions. I want to receive data from IB. I have tried the following:

data = ibstore.getdata(dataname=instrumentName, timeframe=bt.TimeFrame.Seconds, compression=30)
cerebro.adddata(data)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=1)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=15)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=30)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=60)

The code above fetches the data from IB in 30 Seconds bars as defined just for historical data. However, once the data is live, it arrives like tick data(under 1 Sec).
I need both of the 30 Sec bars and replayed ones.
Is there any way to let backtrader fetch 30 Seconds bars only?
The real issue here is that this strategy runs on 100-150 products and the strategy doesn't need to wake up every few milliseconds.

Thanks

ab_trader

Probably (I am not familiar with live trading deep enough, but based on the example provided in the docs) cerebro.resampledata(...) need to be used instead of cerebro.replaydata(...).

backtrader

@ab_trader said in Compression on IB Live data:

Probably (I am not familiar with live trading deep enough, but based on the example provided in the docs) cerebro.resampledata(...) need to be used instead of cerebro.replaydata(...).

Indeed.

replaydata is called so, because it does replay what the data does.

Documentation:

@bb2 said in Compression on IB Live data:

Is there any way to let backtrader fetch 30 Seconds bars only?

Nothing to do with backtrader. Interactive Brokers doesn't provide that service for live data. That's only for the historical part (and backfilling where possible)

@bb2 said in Compression on IB Live data:

The real issue here is that this strategy runs on 100-150 products and the strategy doesn't need to wake up every few milliseconds.

Don't replay the data then.

bb2

@backtrader Frankly I do not have any performance issues when I do backtesting on a csv data. But when I run the same strategy on live IB data, after 1-2 minutes the data starts to stall and thus I am not able to do 1-min trades anymore. Have you tried getting live data of around 20-30 products from IB and just printing them out? Even without replaying or resampling, the 'live' data is out of sync and delayed about a minute and goes off 'live' status for a while. I would be happy to help to resolve this problem if this persists for others as well.

Best

backtrader

@bb2 said in Compression on IB Live data:

Have you tried getting live data of around 20-30 products from IB and just printing them out?

No. For such a feat I would use a different data provider/broker, given the many limitations present in the data service offered by Interactive. For example: https://interactivebrokers.github.io/tws-api/historical_limitations.html

bb2

@backtrader which one would you suggest for such an operation? (both for live data acquisition and making orders)

bb2

@backtrader not replaying is unfortunately not an option. However, the following would solve my problem, I am not sure if it is possible to do it though:

tick data is resampled to 30 seconds.
replay data is called on the resampled data ( 30 sec bars).

in this case replayed data can only wake up every 30 secs instead of every tick. How can I make this happen?

The strategy needs to have access to live data (doesn't need to be tick, but 20-30 secs is enough) on high timeframes, because it relies on indicator's live values in high timeframes, too. As far as I know, resampling only takes into account the closed bar, thus live values wouldn't be considered. Please correct me if I am wrong.

Best

backtrader

@bb2 said in Compression on IB Live data:

replay data is called on the resampled data ( 30 sec bars).
in this case replayed data can only wake up every 30 secs instead of every tick. How can I make this happen?

You seem to fail to understand that replaydata has the prefix replay because it delivers each and every tick received to actually replay what the market does.

If you only want the complete 30 seconds every 30 seconds, you have to resample.

Please the read the documentation linked above.

bb2

@backtrader seems I was not clear enough. how can I replay on the resampled data? Let's assume the IB tick data arrives every 250ms. Instead of directly replaying on this tick data, I want to first resample the tick data to 30 secs and run replay on 30 sec data. How would that be possible?

backtrader

Have you tried replaydata(resampledata(...))? It may of course work or bring the system down to its knees ...

bb2

@backtrader ... seriously? my approach could be wrong. though do you have anything else in mind as a solution? resampling everything for sure wouldn't work cos recent data plays important role. Is this doable with backtrader or am I simply wasting my time here?

bb2

if still not clear, the strategy has to work on open bar ( which just needs to be updated every 20-30 secs) in low and high timeframes, how would you make it happen with 250ms tick data on backtrader?

bb2

I have just noticed if I resample tick data to 30 secs first and then replaydata on higher timeframes, the replayed data only updates with 30 secs intervals together with resampled data instead of updating with tick data. Is this a bug or a feature? :D

backtrader

@bb2 said in Compression on IB Live data:

if still not clear,

No it's not clear.

You are again describing resampling but you say you want to replay.

@bb2 said in Compression on IB Live data:

I have just noticed if I resample tick data to 30 secs first and then replaydata on higher timeframes, the replayed data only updates with 30 secs intervals together with resampled data instead of updating with tick data. Is this a bug or a feature?

It is the intended behavior. A bar resampled to 30 seconds, will be delivered every 30 seconds. If you replay that 30 seconds bar, the lowest possible replay frequency is obviously 30 seconds.

Incredibly, this is what was recommended:

@backtrader said in Compression on IB Live data:

Have you tried replaydata(resampledata(...))?

@bb2 said in Compression on IB Live data:

@backtrader ... seriously?

Yes. Apparently this is what you want in spite of the obscurity of the posts.

bb2

referring to the code:

data = ibstore.getdata(dataname=instrumentName)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1)
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=5)

it is not obvious to the coder that the replay is done on the resampled data. I thought resampling and replaying were done independently on the tick data. You could probably update the doc to point this out. Thank you for your help.

ab_trader

@bb2 I might be wrong, but looks like your baseline data obtained from ib is resampled to 30 sec bars during data delivery. So there are no tick data available in the system.

NothingButHands

@bb2 It makes sense if you realize that resampledata() performs the resampling in-place; something not specified very clearly in the docs