Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Live data does not match historical data for ibtest.py #359

    General Discussion
    2
    6
    1527
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nooby_mcnoob last edited by

      I guess @backtrader does not consider https://github.com/backtrader/backtrader/issues/359 a bug so I am reposting here in case an expert can see what I'm doing wrong:

      Please note that you need the patch for ibtest.py below for it to work with these parameters.

      With the two sessions below, you can see that live bars don't match historical bars in a very particular pattern, where the pattern is not value-for-value, but approximately.

      There are two sessions below: 1 with 5 minute compression and one with 60 minute compression. You can see the same issue with both.

      In the snapshot below, you can see that 19:35 from the live data matches the historical data from 19:30 at least where it comes to high/low. In general, I expect open/close to not match exactly in live vs historical primarily because they're not exact, but I don't expect live bars and historical bars to regularly disagree on high/low with a one-off pattern.

      Live

      Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404
      Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454
      Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524
      Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768
      Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783
      Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966
      

      Historical

      Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434
      Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504
      Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688
      Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836
      Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968
      Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124
      

      Here are the commands I used with a patched ibtest.py to see the differences:

      # tracking live data
      $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5  --clientId=984 --fromdate=2018-04-08 --resample --todate=2019-04-08
      ...
      ***** DATA NOTIF: LIVE
      Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404
      Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454
      ***** STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.cashhmds>
      Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524
      Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768
      Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783
      Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966
      Data0, 0030,              736792.9861111111, 2018-04-08T19:40:00.000000, 1.27808, 1.27842, 1.27807, 1.27838, 0.0, 0, 1.278124
      Data0, 0031,              736792.9895833334, 2018-04-08T19:45:00.000000, 1.27839, 1.27839, 1.2782, 1.27821, 0.0, 0, 1.278228
      Data0, 0032,              736792.9930555555, 2018-04-08T19:50:00.000000, 1.2782, 1.27837, 1.27804, 1.27819, 0.0, 0, 1.278214
      Data0, 0033,              736792.9965277778, 2018-04-08T19:55:00.000000, 1.2782, 1.27827, 1.27804, 1.27804, 0.0, 0, 1.278178
      Data0, 0034,              736793.0034722222, 2018-04-08T20:05:00.000000, 1.27801, 1.27856, 1.27791, 1.27792, 0.0, 0, 1.278148
      Data0, 0035,              736793.0069444445, 2018-04-08T20:10:00.000000, 1.27795, 1.27806, 1.27773, 1.27804, 0.0, 0, 1.2780799999999999
      
      # Querying historical data
      $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host --port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5 --clientId=985 --fromdate=2018-04-8 --resample --historical --stopafter=10
      ...
      Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434
      Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504
      Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688
      Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836
      Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968
      Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124
      Data0, 0030,              736792.9861111111, 2018-04-08T19:40:00.000000, 1.27838, 1.27839, 1.2782, 1.27821, -1.0, 0, 1.278228
      Data0, 0031,              736792.9895833334, 2018-04-08T19:45:00.000000, 1.27821, 1.27837, 1.27804, 1.27819, -1.0, 0, 1.278214
      Data0, 0032,              736792.9930555555, 2018-04-08T19:50:00.000000, 1.27819, 1.27827, 1.27804, 1.27804, -1.0, 0, 1.278178
      Data0, 0033,              736792.9965277778, 2018-04-08T19:55:00.000000, 1.27804, 1.2783, 1.27797, 1.2783, -1.0, 0, 1.278224
      Data0, 0034,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27791, 1.27792, -1.0, 0, 1.278132
      Data0, 0035,              736793.0034722222, 2018-04-08T20:05:00.000000, 1.27792, 1.27806, 1.27773, 1.27806, -1.0, 0, 1.278102
      Data0, 0036,              736793.0069444445, 2018-04-08T20:10:00.000000, 1.27806, 1.27806, 1.27788, 1.27788, -1.0, 0, 1.27804
      

      Here is the same process, but run with compression=60 (so 1 hour). You can see that the resampled live bar at 23:00 is roughly equal (HLC are equal) to the historical bar at 22:00.

      Live

      Data0, 0001,              736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan
      Data0, 0002,              736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan
      Data0, 0003,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan
      Data0, 0004,              736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.2773, 1.27665, 1.27701, -1.0, 0, nan
      ***** DATA NOTIF: LIVE
      Data0, 0005,              736793.0833333334, 2018-04-08T22:00:00.000000, 1.27701, 1.27741, 1.27662, 1.27737, 0.0, 0, 1.2773379999999999
      Data0, 0006,                     736793.125, 2018-04-08T23:00:00.000000, 1.27736, 1.27781, 1.27707, 1.27727, 0.0, 0, 1.277324
      

      Historical

      Data0, 0001,              736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan
      Data0, 0002,              736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan
      Data0, 0003,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan
      Data0, 0004,              736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.27741, 1.27662, 1.27739, -1.0, 0, nan
      Data0, 0005,              736793.0833333334, 2018-04-08T22:00:00.000000, 1.27739, 1.27781, 1.27707, 1.27727, -1.0, 0, 1.277394
      Data0, 0006,                     736793.125, 2018-04-08T23:00:00.000000, 1.27727, 1.27746, 1.27721, 1.27738, -1.0, 0, 1.277402
      
      diff --git a/samples/ibtest/ibtest.py b/samples/ibtest/ibtest.py
      index 8000f41..4530be9 100644
      --- a/samples/ibtest/ibtest.py
      +++ b/samples/ibtest/ibtest.py
      @@ -93,7 +93,7 @@ class TestStrategy(bt.Strategy):
               txt.append('Data0')
               txt.append('%04d' % len(self.data0))
               dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
      -        txt.append('{}'.format(self.data.datetime[0]))
      +        txt.append('{:>30}'.format(self.data.datetime[0]))
               txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
               txt.append('{}'.format(self.data.open[0]))
               txt.append('{}'.format(self.data.high[0]))
      @@ -253,11 +253,18 @@ def runstrategy():
               dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate))
               fromdate = datetime.datetime.strptime(args.fromdate, dtformat)
       
      +    todate = None
      +    if args.todate:
      +        dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate))
      +        todate = datetime.datetime.strptime(args.todate,dtformat)
      +    else:
      +        todate = datetime.datetime.now()
      +
           IBDataFactory = ibstore.getdata if args.usestore else bt.feeds.IBData
       
           datakwargs = dict(
               timeframe=datatf, compression=datacomp,
      -        historical=args.historical, fromdate=fromdate,
      +        historical=args.historical, fromdate=fromdate, todate=todate,
               rtbar=args.rtbar,
               qcheck=args.qcheck,
               what=args.what,
      @@ -441,6 +448,11 @@ def parse_args():
                               help=('Starting date for historical download '
                                     'with format: YYYY-MM-DD[THH:MM:SS]'))
       
      +    parser.add_argument('--todate',
      +                        required=False, action='store',
      +                        help=('Ending date for historical download '
      +                              'with format: YYYY-MM-DD[THH:MM:SS]'))
      +
           parser.add_argument('--smaperiod', default=5, type=int,
                               required=False, action='store',
                               help='Period to apply to the Simple Moving Average')
      
      1 Reply Last reply Reply Quote 0
      • D
        dasch last edited by

        I did some checks:

        i used the following params in resample

        rightedge=False,
        adjbartime=True,
        bar2edge=True,

        When using rightedge=False, the tick will be immediately be delivered, the time will not be adjusted.

        This seems to be connected to _adjusttime in Resampler and the param greater. I replaced it with self.p.rightedge where it occured. This at least provided me correct timestamps in the resampled data. But one issue was, that the first incoming live data was not added to an existing bar. Everything else was correct after playing around with the greater param.

        @backtrader so if the rightedge param is False, the time is not being adjusted to the leftedge, instead is is delivered instantly. maybe this helps to find the issue.

        N 1 Reply Last reply Reply Quote 0
        • N
          nooby_mcnoob @dasch last edited by

          @dasch Do you think that it's an issue and not misuse of the API? I wanted to use ibtest.py to reproduce it so that there is no "user" code in there.

          1 Reply Last reply Reply Quote 0
          • D
            dasch last edited by

            It's not really a issue. Its how backtrader works. By default it adjusts the data to the rightedge of the current timeframe. You would need to use rightedge=False, so the data is aligned to the left edge.

            It may be that there is a bug in backtrader because rightedge=False is being ignored, but i am not sure.

            @backtrader it seems this code may be the culprint?

            File resamplerfilter.py

                def _adjusttime(self, greater=False, forcedata=None):
                    '''
                    Adjusts the time of calculated bar (from underlying data source) by
                    using the timeframe to the appropriate boundary, with compression taken
                    into account
            
                    Depending on param ``rightedge`` uses the starting boundary or the
                    ending one
                    '''
                    dtnum = self._calcadjtime(greater=greater)
                    if greater and dtnum <= self.bar.datetime:
                        return False
            
                    self.bar.datetime = dtnum
                    return True
            
            N 1 Reply Last reply Reply Quote 0
            • D
              dasch last edited by dasch

              It breaks when the adjusted time in live trade with rightedge = False is lower than the time of the last bar?

              I got a tick

              self.bar.datetime = DATE 2018-04-09T11:54:25.976293 (736793.4961339849)

              the adjusted time with rightedge = False is:
              dtnum = DATE 2018-04-09 11:54:00 (736793.495833333)
              with rightedge = True it would be:
              dtnum = DATE 2018-04-09 11:54:30 (this will be higher than self.bar.datetime)

              when _adjusttime is being called with greater=True then the codition:

                  if greater and dtnum <= self.bar.datetime:
              

              is False, so no bar time is being updated.

              1 Reply Last reply Reply Quote 0
              • N
                nooby_mcnoob @dasch last edited by nooby_mcnoob

                @dasch @backtrader

                Thanks for your help. The PEBKAC is resolved :-)

                I guess it is obvious to you, but perhaps a default mode of "backtest bars should match live bars" would follow the principle of least surprise? In any case, here are some live resampled bars, followed by the corresponding historical resampled bars with a timeframe of 5 minutes with rightedge=False. This shows that live bars match historical bars. Now the question I have to answer myself is: "why did righedge=False not work in my system when I tried it live?"

                Live

                Data0, 0196,                    736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366
                Data0, 0197,              736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632
                Data0, 0198,              736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775
                Data0, 0199,              736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784
                Data0, 0200,              736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684
                Data0, 0201,              736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775
                Data0, 0202,              736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176
                Data0, 0203,              736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711
                Data0, 0204,              736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002
                Data0, 0205,                   736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034
                

                Historical

                Data0, 0196,                    736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366
                Data0, 0197,              736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632
                Data0, 0198,              736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775
                Data0, 0199,              736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784
                Data0, 0200,              736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684
                Data0, 0201,              736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775
                Data0, 0202,              736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176
                Data0, 0203,              736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711
                Data0, 0204,              736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002
                Data0, 0205,                   736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034
                
                
                1 Reply Last reply Reply Quote 3
                • 1 / 1
                • First post
                  Last post
                Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors