Backtest vs live bars "off-by-1" discrepancy?
-
Hi,
Thanks for this useful platform, it has helped me accelerate time to market. One thing I'm noticing however is that the live bars and backtest bars seem to be offset by 1 period. You can see below that the bar for backtest
10:02:00
is equal to the live bar for10:04:00
and this pattern seems to continue for the rest of the data. What could cause this?I am using Interactive Brokers. Here is how I set up the data feeds:
ibstore = bt.stores.IBStore(host=params.gateway_host,port=params.gateway_port, clientId=params.live and 983 or 984, notifyall = params.verbose >= 4, reconnect=-1, # keep retrying forever _debug=params.verbose >= 4) ticker = params.ticker data = ibstore.getdata(dataname=ticker, qcheck = 5.0, # allow a 5 second delay in prices name=ticker, latethrough=True, timeframe=params.lowertimeframeunits, compression=params.lowertimeframe, backfill_start = True, sessionstart=params.sessionstarttime, sessionend=params.sessionendtime, historical=not params.live, fromdate=params.start, todate=params.end)
Here are the ohlc bars for the data from the live run:
datetime,close,low,high,open,volume,openinterest "2018-04-04 10:10:00.000000",1.28263,1.28261,1.28286,1.28269,0.0,0.0 "2018-04-04 10:08:00.000000",1.28271,1.28246,1.28277,1.28271,0.0,0.0 "2018-04-04 10:06:00.000000",1.28269,1.28264,1.28319,1.28299,0.0,0.0 "2018-04-04 10:04:00.000000",1.28297,1.28295,1.28322,1.2831,0.0,0.0 "2018-04-04 10:02:00.000000",1.28309,1.28285,1.28335,1.28289,0.0,0.0
And here is the same period for the backtest:
datetime,close,low,high,open,volume,openinterest "2018-04-04 10:10:00.000000",1.28294,1.28262,1.28309,1.28263,-1.0,0.0 "2018-04-04 10:08:00.000000",1.28263,1.28261,1.28286,1.28271,-1.0,0.0 "2018-04-04 10:06:00.000000",1.28271,1.28246,1.28277,1.28269,-1.0,0.0 "2018-04-04 10:04:00.000000",1.28269,1.28264,1.28319,1.28297,-1.0,0.0 "2018-04-04 10:02:00.000000",1.28297,1.28295,1.28323,1.28306,-1.0,0.0
-
Things to consider:
- The data you show runs backwards in time. That means those longs have not been produced directly with backtrader
- Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.
What we need to know:
- How are the live bars being produced? Which resampling and whatever could be relevant?
Take into account that IB may consolidate/resample the data using a different starting point, algorithm, ... how they do it ... it is unknown.
-
Thanks for your response.
@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:
The data you show runs backwards in time. That means those longs have not been produced directly with backtrader
This is true, I am exporting them from CSV into Excel and sorting and filtering. However, the data is unaltered from the backtrader output.
@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:
Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.
Ah, this seems like it could be relevant! Could you point me towards where the resampling is taking place? Or are you implying that when trading live, I should be manually resampling?
@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:
How are the live bars being produced? Which resampling and whatever could be relevant?
The live bars were actually resampled in the live+backtest data above, but I did not include that code:
data.resample(timeframe=params.lowertimeframeunits, compression=params.lowertimeframe)
That is, the data was resampled to match the timeframes in
getdata
.Is there something else I can look into? It seems to be key that the resampled bars are "roughly" offset-by-1.
-
@nooby_mcnoob said in Backtest vs live bars "off-by-1" discrepancy?:
The live bars were actually resampled in the live+backtest data above, but I did not include that code:
data.resample(timeframe=params.lowertimeframeunits,
compression=params.lowertimeframe)That is, the data was resampled to match the timeframes in getdata.
@backtrader could this be the cause? Both the live and backtest are resampled in the same way, yet the live data is off-by-1.
-
It is very difficult to follow your reasoning with live and something else.
-
Live data from Interactive Brokers has to be resampled unless you want to work with
Ticks
.If this is the data you have stored and you call live (which is no longer live because it is stored and was experienced live only once), it is already resampled
-
You run then some other test, but is unclear what is the source that you are using to resample.
-
-
@backtrader Thanks for responding.
Let's use the following nomenclature:
- live data = live tick data
- resampled live data = resampled live tick data
- resampled historical data = obvious
Here is the fact about what I'm experiencing:
- The resampled live data for period N matches the resampled historical data for period N-1
More specifically, given:
ohlc = [self.data0.open,self.data0.high,self.data0.low,self.data0.close]
This value is different for resampled live data vs resampled historical data when
self.data0.datetime.datetime()
are equal. I test by logging each value ofself.data0.datetime.datetime()
andohlc
during both live trading and back testing. By comparing the value ofself.data0.datetime.datetime()
during live and historic runs, I am able to see that the values ofohlc
for (at least) CASH instruments are off-by-1.Is it possible that the resampled historical data is resampled incorrectly (i.e., a bug)? This would match my observations that resampled historical data are a period BEHIND the resampled live data.
It should theoretically be possible to write a test case. I'll see if I can do this.
-
Did you try
resampledata
method parameters such asrightedge
orboundoff
? This might help. -
@ab_trader Thanks. Is
cb.resampledata
different thandata.resample(...)
? I've been using the latter. -
@nooby_mcnoob I don't know, I always used resampledata following the manuals.
-
@nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.
-
@nooby_mcnoob Did you find a solution for your problem? There is someone else having comparable issues. See https://github.com/ftomassetti/backtrader-oandav20/issues/13
-
@dasch I did not find a solution, but I have subscribed to the issue you linked.
-
@dasch said in Backtest vs live bars "off-by-1" discrepancy?:
@nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.
It doesn't appear to be the same result. Resampling with cerebro and running a backtest causes it to hang. Sigh.
-
@dasch I was able to reproduce with a slightly patched
ibtest.py
and created a github issue: https://github.com/backtrader/backtrader/issues/359 -
I found a workaround for this issue. When creating the data feed, i used the following params which generate the correct timestamps:
rightedge=True, boundoff=1
with live data from oanda and rightedge=False the data will get delivered when arriving instead of being resampled.
-
@dasch Trying this for live trading, will see if they match up. Thanks.
-
@dasch Looks like bars match backtest/live trading now. It's odd to get a bar labeled "10:56 AM" at 10:55, but it's understandable and reproducible which is more important than guessing :)
Thanks for the update!
-
I bumped into similar situation. Looks like by default, backtrader is not touching DELAYED data and therefore inherits whatever "rightedge" convention from the original data feed, and in case of ibstore, it is "rightedge=False"; while for LIVE data part it will use the default convention of "rightedge=True", which causes the discrepancy.
Many thanks for the tip from @dasch, now the bars match backteset/live trading.
-
Besides the different rightedge convention between delayed and live data, has anyone noticed that the last delayed data bar is created even before the time period is up?
I am using Interactive Brokers and am seeing this behaviour.
-
Just want to emphasize that the workaround solution, boundoff = 1, as proposed earlier solve the problem of last delayed bar being created before time period is up but then, that workaround solution creates another problem.
The new problem is that all the live signals and execution would be brought forward by one period.
How do I resolve the problem of "last delayed bar being created before time period is up" and not have all the live signals and execution being brought forward?