Backtest vs live bars "off-by-1" discrepancy?

nooby_mcnoob

Hi,

Thanks for this useful platform, it has helped me accelerate time to market. One thing I'm noticing however is that the live bars and backtest bars seem to be offset by 1 period. You can see below that the bar for backtest 10:02:00 is equal to the live bar for 10:04:00 and this pattern seems to continue for the rest of the data. What could cause this?

I am using Interactive Brokers. Here is how I set up the data feeds:

    ibstore = bt.stores.IBStore(host=params.gateway_host,port=params.gateway_port,
                                clientId=params.live and 983 or 984,
                                notifyall = params.verbose >= 4,
                                reconnect=-1, # keep retrying forever
                                _debug=params.verbose >= 4)

    ticker = params.ticker

    data = ibstore.getdata(dataname=ticker,
                            qcheck = 5.0, # allow a 5 second delay in prices
                            name=ticker,
                            latethrough=True,
                            timeframe=params.lowertimeframeunits,
                            compression=params.lowertimeframe,
                            backfill_start = True,
                            sessionstart=params.sessionstarttime,
                            sessionend=params.sessionendtime,
                            historical=not params.live,
                            fromdate=params.start,
                            todate=params.end)

Here are the ohlc bars for the data from the live run:

datetime,close,low,high,open,volume,openinterest
"2018-04-04 10:10:00.000000",1.28263,1.28261,1.28286,1.28269,0.0,0.0
"2018-04-04 10:08:00.000000",1.28271,1.28246,1.28277,1.28271,0.0,0.0
"2018-04-04 10:06:00.000000",1.28269,1.28264,1.28319,1.28299,0.0,0.0
"2018-04-04 10:04:00.000000",1.28297,1.28295,1.28322,1.2831,0.0,0.0
"2018-04-04 10:02:00.000000",1.28309,1.28285,1.28335,1.28289,0.0,0.0

And here is the same period for the backtest:

datetime,close,low,high,open,volume,openinterest
"2018-04-04 10:10:00.000000",1.28294,1.28262,1.28309,1.28263,-1.0,0.0
"2018-04-04 10:08:00.000000",1.28263,1.28261,1.28286,1.28271,-1.0,0.0
"2018-04-04 10:06:00.000000",1.28271,1.28246,1.28277,1.28269,-1.0,0.0
"2018-04-04 10:04:00.000000",1.28269,1.28264,1.28319,1.28297,-1.0,0.0
"2018-04-04 10:02:00.000000",1.28297,1.28295,1.28323,1.28306,-1.0,0.0

backtrader

Things to consider:

The data you show runs backwards in time. That means those longs have not been produced directly with backtrader
Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.

What we need to know:

How are the live bars being produced? Which resampling and whatever could be relevant?

Take into account that IB may consolidate/resample the data using a different starting point, algorithm, ... how they do it ... it is unknown.

nooby_mcnoob

Thanks for your response.

@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

The data you show runs backwards in time. That means those longs have not been produced directly with backtrader

This is true, I am exporting them from CSV into Excel and sorting and filtering. However, the data is unaltered from the backtrader output.

@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.

Ah, this seems like it could be relevant! Could you point me towards where the resampling is taking place? Or are you implying that when trading live, I should be manually resampling?

@backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

How are the live bars being produced? Which resampling and whatever could be relevant?

The live bars were actually resampled in the live+backtest data above, but I did not include that code:

    data.resample(timeframe=params.lowertimeframeunits,
                               compression=params.lowertimeframe)

That is, the data was resampled to match the timeframes in getdata.

Is there something else I can look into? It seems to be key that the resampled bars are "roughly" offset-by-1.

nooby_mcnoob

@nooby_mcnoob said in Backtest vs live bars "off-by-1" discrepancy?:

The live bars were actually resampled in the live+backtest data above, but I did not include that code:
data.resample(timeframe=params.lowertimeframeunits,
compression=params.lowertimeframe)

That is, the data was resampled to match the timeframes in getdata.

@backtrader could this be the cause? Both the live and backtest are resampled in the same way, yet the live data is off-by-1.

backtrader

It is very difficult to follow your reasoning with live and something else.

Live data from Interactive Brokers has to be resampled unless you want to work with Ticks.

If this is the data you have stored and you call live (which is no longer live because it is stored and was experienced live only once), it is already resampled
You run then some other test, but is unclear what is the source that you are using to resample.

nooby_mcnoob

@backtrader Thanks for responding.

Let's use the following nomenclature:

live data = live tick data
resampled live data = resampled live tick data
resampled historical data = obvious

Here is the fact about what I'm experiencing:

The resampled live data for period N matches the resampled historical data for period N-1

More specifically, given:

ohlc = [self.data0.open,self.data0.high,self.data0.low,self.data0.close]

This value is different for resampled live data vs resampled historical data when self.data0.datetime.datetime() are equal. I test by logging each value of self.data0.datetime.datetime() and ohlc during both live trading and back testing. By comparing the value of self.data0.datetime.datetime() during live and historic runs, I am able to see that the values of ohlc for (at least) CASH instruments are off-by-1.

Is it possible that the resampled historical data is resampled incorrectly (i.e., a bug)? This would match my observations that resampled historical data are a period BEHIND the resampled live data.

It should theoretically be possible to write a test case. I'll see if I can do this.

ab_trader

Did you try resampledata method parameters such as rightedge or boundoff? This might help.

Docs - Data Resampling

nooby_mcnoob

@ab_trader Thanks. Is cb.resampledata different than data.resample(...)? I've been using the latter.

ab_trader

@nooby_mcnoob I don't know, I always used resampledata following the manuals.

dasch

@nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.

dasch

@nooby_mcnoob Did you find a solution for your problem? There is someone else having comparable issues. See https://github.com/ftomassetti/backtrader-oandav20/issues/13

nooby_mcnoob

@dasch I did not find a solution, but I have subscribed to the issue you linked.

nooby_mcnoob

@dasch said in Backtest vs live bars "off-by-1" discrepancy?:

@nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.

It doesn't appear to be the same result. Resampling with cerebro and running a backtest causes it to hang. Sigh.

nooby_mcnoob

@dasch I was able to reproduce with a slightly patched ibtest.py and created a github issue: https://github.com/backtrader/backtrader/issues/359

dasch

I found a workaround for this issue. When creating the data feed, i used the following params which generate the correct timestamps:

rightedge=True, boundoff=1

with live data from oanda and rightedge=False the data will get delivered when arriving instead of being resampled.

nooby_mcnoob

@dasch Trying this for live trading, will see if they match up. Thanks.

nooby_mcnoob

@dasch Looks like bars match backtest/live trading now. It's odd to get a bar labeled "10:56 AM" at 10:55, but it's understandable and reproducible which is more important than guessing :)

Thanks for the update!

z_kvn

I bumped into similar situation. Looks like by default, backtrader is not touching DELAYED data and therefore inherits whatever "rightedge" convention from the original data feed, and in case of ibstore, it is "rightedge=False"; while for LIVE data part it will use the default convention of "rightedge=True", which causes the discrepancy.

Many thanks for the tip from @dasch, now the bars match backteset/live trading.

Squirrel Singing

Besides the different rightedge convention between delayed and live data, has anyone noticed that the last delayed data bar is created even before the time period is up?

I am using Interactive Brokers and am seeing this behaviour.

Squirrel Singing

Just want to emphasize that the workaround solution, boundoff = 1, as proposed earlier solve the problem of last delayed bar being created before time period is up but then, that workaround solution creates another problem.

The new problem is that all the live signals and execution would be brought forward by one period.

How do I resolve the problem of "last delayed bar being created before time period is up" and not have all the live signals and execution being brought forward?