For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Live data does not match historical data for ibtest.py #359



  • I guess @backtrader does not consider https://github.com/backtrader/backtrader/issues/359 a bug so I am reposting here in case an expert can see what I'm doing wrong:

    Please note that you need the patch for ibtest.py below for it to work with these parameters.

    With the two sessions below, you can see that live bars don't match historical bars in a very particular pattern, where the pattern is not value-for-value, but approximately.

    There are two sessions below: 1 with 5 minute compression and one with 60 minute compression. You can see the same issue with both.

    In the snapshot below, you can see that 19:35 from the live data matches the historical data from 19:30 at least where it comes to high/low. In general, I expect open/close to not match exactly in live vs historical primarily because they're not exact, but I don't expect live bars and historical bars to regularly disagree on high/low with a one-off pattern.

    Live

    Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404
    Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454
    Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524
    Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768
    Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783
    Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966
    

    Historical

    Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434
    Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504
    Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688
    Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836
    Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968
    Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124
    

    Here are the commands I used with a patched ibtest.py to see the differences:

    # tracking live data
    $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5  --clientId=984 --fromdate=2018-04-08 --resample --todate=2019-04-08
    ...
    ***** DATA NOTIF: LIVE
    Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404
    Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454
    ***** STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.cashhmds>
    Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524
    Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768
    Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783
    Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966
    Data0, 0030,              736792.9861111111, 2018-04-08T19:40:00.000000, 1.27808, 1.27842, 1.27807, 1.27838, 0.0, 0, 1.278124
    Data0, 0031,              736792.9895833334, 2018-04-08T19:45:00.000000, 1.27839, 1.27839, 1.2782, 1.27821, 0.0, 0, 1.278228
    Data0, 0032,              736792.9930555555, 2018-04-08T19:50:00.000000, 1.2782, 1.27837, 1.27804, 1.27819, 0.0, 0, 1.278214
    Data0, 0033,              736792.9965277778, 2018-04-08T19:55:00.000000, 1.2782, 1.27827, 1.27804, 1.27804, 0.0, 0, 1.278178
    Data0, 0034,              736793.0034722222, 2018-04-08T20:05:00.000000, 1.27801, 1.27856, 1.27791, 1.27792, 0.0, 0, 1.278148
    Data0, 0035,              736793.0069444445, 2018-04-08T20:10:00.000000, 1.27795, 1.27806, 1.27773, 1.27804, 0.0, 0, 1.2780799999999999
    
    # Querying historical data
    $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host --port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5 --clientId=985 --fromdate=2018-04-8 --resample --historical --stopafter=10
    ...
    Data0, 0024,              736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434
    Data0, 0025,                   736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504
    Data0, 0026,              736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688
    Data0, 0027,              736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836
    Data0, 0028,              736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968
    Data0, 0029,              736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124
    Data0, 0030,              736792.9861111111, 2018-04-08T19:40:00.000000, 1.27838, 1.27839, 1.2782, 1.27821, -1.0, 0, 1.278228
    Data0, 0031,              736792.9895833334, 2018-04-08T19:45:00.000000, 1.27821, 1.27837, 1.27804, 1.27819, -1.0, 0, 1.278214
    Data0, 0032,              736792.9930555555, 2018-04-08T19:50:00.000000, 1.27819, 1.27827, 1.27804, 1.27804, -1.0, 0, 1.278178
    Data0, 0033,              736792.9965277778, 2018-04-08T19:55:00.000000, 1.27804, 1.2783, 1.27797, 1.2783, -1.0, 0, 1.278224
    Data0, 0034,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27791, 1.27792, -1.0, 0, 1.278132
    Data0, 0035,              736793.0034722222, 2018-04-08T20:05:00.000000, 1.27792, 1.27806, 1.27773, 1.27806, -1.0, 0, 1.278102
    Data0, 0036,              736793.0069444445, 2018-04-08T20:10:00.000000, 1.27806, 1.27806, 1.27788, 1.27788, -1.0, 0, 1.27804
    

    Here is the same process, but run with compression=60 (so 1 hour). You can see that the resampled live bar at 23:00 is roughly equal (HLC are equal) to the historical bar at 22:00.

    Live

    Data0, 0001,              736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan
    Data0, 0002,              736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan
    Data0, 0003,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan
    Data0, 0004,              736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.2773, 1.27665, 1.27701, -1.0, 0, nan
    ***** DATA NOTIF: LIVE
    Data0, 0005,              736793.0833333334, 2018-04-08T22:00:00.000000, 1.27701, 1.27741, 1.27662, 1.27737, 0.0, 0, 1.2773379999999999
    Data0, 0006,                     736793.125, 2018-04-08T23:00:00.000000, 1.27736, 1.27781, 1.27707, 1.27727, 0.0, 0, 1.277324
    

    Historical

    Data0, 0001,              736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan
    Data0, 0002,              736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan
    Data0, 0003,                       736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan
    Data0, 0004,              736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.27741, 1.27662, 1.27739, -1.0, 0, nan
    Data0, 0005,              736793.0833333334, 2018-04-08T22:00:00.000000, 1.27739, 1.27781, 1.27707, 1.27727, -1.0, 0, 1.277394
    Data0, 0006,                     736793.125, 2018-04-08T23:00:00.000000, 1.27727, 1.27746, 1.27721, 1.27738, -1.0, 0, 1.277402
    
    diff --git a/samples/ibtest/ibtest.py b/samples/ibtest/ibtest.py
    index 8000f41..4530be9 100644
    --- a/samples/ibtest/ibtest.py
    +++ b/samples/ibtest/ibtest.py
    @@ -93,7 +93,7 @@ class TestStrategy(bt.Strategy):
             txt.append('Data0')
             txt.append('%04d' % len(self.data0))
             dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
    -        txt.append('{}'.format(self.data.datetime[0]))
    +        txt.append('{:>30}'.format(self.data.datetime[0]))
             txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
             txt.append('{}'.format(self.data.open[0]))
             txt.append('{}'.format(self.data.high[0]))
    @@ -253,11 +253,18 @@ def runstrategy():
             dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate))
             fromdate = datetime.datetime.strptime(args.fromdate, dtformat)
     
    +    todate = None
    +    if args.todate:
    +        dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate))
    +        todate = datetime.datetime.strptime(args.todate,dtformat)
    +    else:
    +        todate = datetime.datetime.now()
    +
         IBDataFactory = ibstore.getdata if args.usestore else bt.feeds.IBData
     
         datakwargs = dict(
             timeframe=datatf, compression=datacomp,
    -        historical=args.historical, fromdate=fromdate,
    +        historical=args.historical, fromdate=fromdate, todate=todate,
             rtbar=args.rtbar,
             qcheck=args.qcheck,
             what=args.what,
    @@ -441,6 +448,11 @@ def parse_args():
                             help=('Starting date for historical download '
                                   'with format: YYYY-MM-DD[THH:MM:SS]'))
     
    +    parser.add_argument('--todate',
    +                        required=False, action='store',
    +                        help=('Ending date for historical download '
    +                              'with format: YYYY-MM-DD[THH:MM:SS]'))
    +
         parser.add_argument('--smaperiod', default=5, type=int,
                             required=False, action='store',
                             help='Period to apply to the Simple Moving Average')
    


  • I did some checks:

    i used the following params in resample

    rightedge=False,
    adjbartime=True,
    bar2edge=True,

    When using rightedge=False, the tick will be immediately be delivered, the time will not be adjusted.

    This seems to be connected to _adjusttime in Resampler and the param greater. I replaced it with self.p.rightedge where it occured. This at least provided me correct timestamps in the resampled data. But one issue was, that the first incoming live data was not added to an existing bar. Everything else was correct after playing around with the greater param.

    @backtrader so if the rightedge param is False, the time is not being adjusted to the leftedge, instead is is delivered instantly. maybe this helps to find the issue.



  • @dasch Do you think that it's an issue and not misuse of the API? I wanted to use ibtest.py to reproduce it so that there is no "user" code in there.



  • It's not really a issue. Its how backtrader works. By default it adjusts the data to the rightedge of the current timeframe. You would need to use rightedge=False, so the data is aligned to the left edge.

    It may be that there is a bug in backtrader because rightedge=False is being ignored, but i am not sure.

    @backtrader it seems this code may be the culprint?

    File resamplerfilter.py

        def _adjusttime(self, greater=False, forcedata=None):
            '''
            Adjusts the time of calculated bar (from underlying data source) by
            using the timeframe to the appropriate boundary, with compression taken
            into account
    
            Depending on param ``rightedge`` uses the starting boundary or the
            ending one
            '''
            dtnum = self._calcadjtime(greater=greater)
            if greater and dtnum <= self.bar.datetime:
                return False
    
            self.bar.datetime = dtnum
            return True
    


  • It breaks when the adjusted time in live trade with rightedge = False is lower than the time of the last bar?

    I got a tick

    self.bar.datetime = DATE 2018-04-09T11:54:25.976293 (736793.4961339849)

    the adjusted time with rightedge = False is:
    dtnum = DATE 2018-04-09 11:54:00 (736793.495833333)
    with rightedge = True it would be:
    dtnum = DATE 2018-04-09 11:54:30 (this will be higher than self.bar.datetime)

    when _adjusttime is being called with greater=True then the codition:

        if greater and dtnum <= self.bar.datetime:
    

    is False, so no bar time is being updated.



  • @dasch @backtrader

    Thanks for your help. The PEBKAC is resolved :-)

    I guess it is obvious to you, but perhaps a default mode of "backtest bars should match live bars" would follow the principle of least surprise? In any case, here are some live resampled bars, followed by the corresponding historical resampled bars with a timeframe of 5 minutes with rightedge=False. This shows that live bars match historical bars. Now the question I have to answer myself is: "why did righedge=False not work in my system when I tried it live?"

    Live

    Data0, 0196,                    736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366
    Data0, 0197,              736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632
    Data0, 0198,              736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775
    Data0, 0199,              736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784
    Data0, 0200,              736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684
    Data0, 0201,              736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775
    Data0, 0202,              736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176
    Data0, 0203,              736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711
    Data0, 0204,              736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002
    Data0, 0205,                   736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034
    

    Historical

    Data0, 0196,                    736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366
    Data0, 0197,              736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632
    Data0, 0198,              736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775
    Data0, 0199,              736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784
    Data0, 0200,              736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684
    Data0, 0201,              736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775
    Data0, 0202,              736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176
    Data0, 0203,              736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711
    Data0, 0204,              736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002
    Data0, 0205,                   736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034