Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Backtest vs live bars "off-by-1" discrepancy?

    General Discussion
    7
    29
    3127
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nooby_mcnoob last edited by nooby_mcnoob

      Hi,

      Thanks for this useful platform, it has helped me accelerate time to market. One thing I'm noticing however is that the live bars and backtest bars seem to be offset by 1 period. You can see below that the bar for backtest 10:02:00 is equal to the live bar for 10:04:00 and this pattern seems to continue for the rest of the data. What could cause this?

      I am using Interactive Brokers. Here is how I set up the data feeds:

          ibstore = bt.stores.IBStore(host=params.gateway_host,port=params.gateway_port,
                                      clientId=params.live and 983 or 984,
                                      notifyall = params.verbose >= 4,
                                      reconnect=-1, # keep retrying forever
                                      _debug=params.verbose >= 4)
      
          ticker = params.ticker
      
          data = ibstore.getdata(dataname=ticker,
                                  qcheck = 5.0, # allow a 5 second delay in prices
                                  name=ticker,
                                  latethrough=True,
                                  timeframe=params.lowertimeframeunits,
                                  compression=params.lowertimeframe,
                                  backfill_start = True,
                                  sessionstart=params.sessionstarttime,
                                  sessionend=params.sessionendtime,
                                  historical=not params.live,
                                  fromdate=params.start,
                                  todate=params.end)
      
      

      Here are the ohlc bars for the data from the live run:

      datetime,close,low,high,open,volume,openinterest
      "2018-04-04 10:10:00.000000",1.28263,1.28261,1.28286,1.28269,0.0,0.0
      "2018-04-04 10:08:00.000000",1.28271,1.28246,1.28277,1.28271,0.0,0.0
      "2018-04-04 10:06:00.000000",1.28269,1.28264,1.28319,1.28299,0.0,0.0
      "2018-04-04 10:04:00.000000",1.28297,1.28295,1.28322,1.2831,0.0,0.0
      "2018-04-04 10:02:00.000000",1.28309,1.28285,1.28335,1.28289,0.0,0.0
      

      And here is the same period for the backtest:

      datetime,close,low,high,open,volume,openinterest
      "2018-04-04 10:10:00.000000",1.28294,1.28262,1.28309,1.28263,-1.0,0.0
      "2018-04-04 10:08:00.000000",1.28263,1.28261,1.28286,1.28271,-1.0,0.0
      "2018-04-04 10:06:00.000000",1.28271,1.28246,1.28277,1.28269,-1.0,0.0
      "2018-04-04 10:04:00.000000",1.28269,1.28264,1.28319,1.28297,-1.0,0.0
      "2018-04-04 10:02:00.000000",1.28297,1.28295,1.28323,1.28306,-1.0,0.0
      
      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        Things to consider:

        • The data you show runs backwards in time. That means those longs have not been produced directly with backtrader
        • Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.

        What we need to know:

        • How are the live bars being produced? Which resampling and whatever could be relevant?

        Take into account that IB may consolidate/resample the data using a different starting point, algorithm, ... how they do it ... it is unknown.

        N 1 Reply Last reply Reply Quote 0
        • N
          nooby_mcnoob @backtrader last edited by

          Thanks for your response.

          @backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

          The data you show runs backwards in time. That means those longs have not been produced directly with backtrader

          This is true, I am exporting them from CSV into Excel and sorting and filtering. However, the data is unaltered from the backtrader output.

          @backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

          Live (unless you mean something else), involves resampling because Interactive Brokers provides no consolidated bars in real time. This isn't shown.

          Ah, this seems like it could be relevant! Could you point me towards where the resampling is taking place? Or are you implying that when trading live, I should be manually resampling?

          @backtrader said in Backtest vs live bars "off-by-1" discrepancy?:

          How are the live bars being produced? Which resampling and whatever could be relevant?

          The live bars were actually resampled in the live+backtest data above, but I did not include that code:

              data.resample(timeframe=params.lowertimeframeunits,
                                         compression=params.lowertimeframe)
          
          

          That is, the data was resampled to match the timeframes in getdata.

          Is there something else I can look into? It seems to be key that the resampled bars are "roughly" offset-by-1.

          N 1 Reply Last reply Reply Quote 0
          • N
            nooby_mcnoob @nooby_mcnoob last edited by

            @nooby_mcnoob said in Backtest vs live bars "off-by-1" discrepancy?:

            The live bars were actually resampled in the live+backtest data above, but I did not include that code:
            data.resample(timeframe=params.lowertimeframeunits,
            compression=params.lowertimeframe)

            That is, the data was resampled to match the timeframes in getdata.

            @backtrader could this be the cause? Both the live and backtest are resampled in the same way, yet the live data is off-by-1.

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              It is very difficult to follow your reasoning with live and something else.

              • Live data from Interactive Brokers has to be resampled unless you want to work with Ticks.

                If this is the data you have stored and you call live (which is no longer live because it is stored and was experienced live only once), it is already resampled

              • You run then some other test, but is unclear what is the source that you are using to resample.

              N 1 Reply Last reply Reply Quote 0
              • N
                nooby_mcnoob @backtrader last edited by nooby_mcnoob

                @backtrader Thanks for responding.

                Let's use the following nomenclature:

                • live data = live tick data
                • resampled live data = resampled live tick data
                • resampled historical data = obvious

                Here is the fact about what I'm experiencing:

                • The resampled live data for period N matches the resampled historical data for period N-1

                More specifically, given:

                ohlc = [self.data0.open,self.data0.high,self.data0.low,self.data0.close]

                This value is different for resampled live data vs resampled historical data when self.data0.datetime.datetime() are equal. I test by logging each value of self.data0.datetime.datetime() and ohlc during both live trading and back testing. By comparing the value of self.data0.datetime.datetime() during live and historic runs, I am able to see that the values of ohlc for (at least) CASH instruments are off-by-1.

                Is it possible that the resampled historical data is resampled incorrectly (i.e., a bug)? This would match my observations that resampled historical data are a period BEHIND the resampled live data.

                It should theoretically be possible to write a test case. I'll see if I can do this.

                1 Reply Last reply Reply Quote 0
                • A
                  ab_trader last edited by

                  Did you try resampledata method parameters such as rightedge or boundoff? This might help.

                  Docs - Data Resampling

                  N 1 Reply Last reply Reply Quote 1
                  • N
                    nooby_mcnoob @ab_trader last edited by

                    @ab_trader Thanks. Is cb.resampledata different than data.resample(...)? I've been using the latter.

                    A D 3 Replies Last reply Reply Quote 0
                    • A
                      ab_trader @nooby_mcnoob last edited by

                      @nooby_mcnoob I don't know, I always used resampledata following the manuals.

                      1 Reply Last reply Reply Quote 0
                      • D
                        dasch @nooby_mcnoob last edited by

                        @nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.

                        N 1 Reply Last reply Reply Quote 0
                        • D
                          dasch @nooby_mcnoob last edited by

                          @nooby_mcnoob Did you find a solution for your problem? There is someone else having comparable issues. See https://github.com/ftomassetti/backtrader-oandav20/issues/13

                          N 1 Reply Last reply Reply Quote 0
                          • N
                            nooby_mcnoob @dasch last edited by

                            @dasch I did not find a solution, but I have subscribed to the issue you linked.

                            1 Reply Last reply Reply Quote 0
                            • N
                              nooby_mcnoob @dasch last edited by nooby_mcnoob

                              @dasch said in Backtest vs live bars "off-by-1" discrepancy?:

                              @nooby_mcnoob The result of resampling should be the same. Its done by the class Resampler.

                              It doesn't appear to be the same result. Resampling with cerebro and running a backtest causes it to hang. Sigh.

                              1 Reply Last reply Reply Quote 0
                              • N
                                nooby_mcnoob last edited by

                                @dasch I was able to reproduce with a slightly patched ibtest.py and created a github issue: https://github.com/backtrader/backtrader/issues/359

                                1 Reply Last reply Reply Quote 0
                                • D
                                  dasch last edited by

                                  I found a workaround for this issue. When creating the data feed, i used the following params which generate the correct timestamps:

                                  rightedge=True, boundoff=1
                                  

                                  with live data from oanda and rightedge=False the data will get delivered when arriving instead of being resampled.

                                  N 2 Replies Last reply Reply Quote 1
                                  • N
                                    nooby_mcnoob @dasch last edited by

                                    @dasch Trying this for live trading, will see if they match up. Thanks.

                                    1 Reply Last reply Reply Quote 0
                                    • N
                                      nooby_mcnoob @dasch last edited by

                                      @dasch Looks like bars match backtest/live trading now. It's odd to get a bar labeled "10:56 AM" at 10:55, but it's understandable and reproducible which is more important than guessing :)

                                      Thanks for the update!

                                      1 Reply Last reply Reply Quote 0
                                      • Z
                                        z_kvn last edited by z_kvn

                                        I bumped into similar situation. Looks like by default, backtrader is not touching DELAYED data and therefore inherits whatever "rightedge" convention from the original data feed, and in case of ibstore, it is "rightedge=False"; while for LIVE data part it will use the default convention of "rightedge=True", which causes the discrepancy.

                                        Many thanks for the tip from @dasch, now the bars match backteset/live trading.

                                        1 Reply Last reply Reply Quote 1
                                        • Squirrel Singing
                                          Squirrel Singing last edited by

                                          Besides the different rightedge convention between delayed and live data, has anyone noticed that the last delayed data bar is created even before the time period is up?

                                          I am using Interactive Brokers and am seeing this behaviour.

                                          1 Reply Last reply Reply Quote 0
                                          • Squirrel Singing
                                            Squirrel Singing last edited by

                                            Just want to emphasize that the workaround solution, boundoff = 1, as proposed earlier solve the problem of last delayed bar being created before time period is up but then, that workaround solution creates another problem.

                                            The new problem is that all the live signals and execution would be brought forward by one period.

                                            How do I resolve the problem of "last delayed bar being created before time period is up" and not have all the live signals and execution being brought forward?

                                            D 3 Replies Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post
                                            Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                                            $(document).ready(function () { app.coldLoad(); }); }