Live data does not match historical data for ibtest.py #359
-
I guess @backtrader does not consider https://github.com/backtrader/backtrader/issues/359 a bug so I am reposting here in case an expert can see what I'm doing wrong:
Please note that you need the patch for
ibtest.pybelow for it to work with these parameters.With the two sessions below, you can see that live bars don't match historical bars in a very particular pattern, where the pattern is not value-for-value, but approximately.
There are two sessions below: 1 with 5 minute compression and one with 60 minute compression. You can see the same issue with both.
In the snapshot below, you can see that
19:35from the live data matches the historical data from19:30at least where it comes to high/low. In general, I expect open/close to not match exactly in live vs historical primarily because they're not exact, but I don't expect live bars and historical bars to regularly disagree on high/low with a one-off pattern.Live
Data0, 0024, 736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404 Data0, 0025, 736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454 Data0, 0026, 736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524 Data0, 0027, 736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768 Data0, 0028, 736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783 Data0, 0029, 736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966Historical
Data0, 0024, 736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434 Data0, 0025, 736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504 Data0, 0026, 736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688 Data0, 0027, 736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836 Data0, 0028, 736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968 Data0, 0029, 736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124Here are the commands I used with a patched
ibtest.pyto see the differences:# tracking live data $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5 --clientId=984 --fromdate=2018-04-08 --resample --todate=2019-04-08 ... ***** DATA NOTIF: LIVE Data0, 0024, 736792.9652777778, 2018-04-08T19:10:00.000000, 1.27747, 1.27751, 1.27739, 1.27739, 0.0, 0, 1.277404 Data0, 0025, 736792.96875, 2018-04-08T19:15:00.000000, 1.27738, 1.2776, 1.27736, 1.27759, 0.0, 0, 1.277454 ***** STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.cashhmds> Data0, 0026, 736792.9722222222, 2018-04-08T19:20:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, 0.0, 0, 1.277524 Data0, 0027, 736792.9756944445, 2018-04-08T19:25:00.000000, 1.27773, 1.2783, 1.27767, 1.27826, 0.0, 0, 1.27768 Data0, 0028, 736792.9791666666, 2018-04-08T19:30:00.000000, 1.27825, 1.27837, 1.27817, 1.27822, 0.0, 0, 1.27783 Data0, 0029, 736792.9826388889, 2018-04-08T19:35:00.000000, 1.27821, 1.27821, 1.27788, 1.27807, 0.0, 0, 1.277966 Data0, 0030, 736792.9861111111, 2018-04-08T19:40:00.000000, 1.27808, 1.27842, 1.27807, 1.27838, 0.0, 0, 1.278124 Data0, 0031, 736792.9895833334, 2018-04-08T19:45:00.000000, 1.27839, 1.27839, 1.2782, 1.27821, 0.0, 0, 1.278228 Data0, 0032, 736792.9930555555, 2018-04-08T19:50:00.000000, 1.2782, 1.27837, 1.27804, 1.27819, 0.0, 0, 1.278214 Data0, 0033, 736792.9965277778, 2018-04-08T19:55:00.000000, 1.2782, 1.27827, 1.27804, 1.27804, 0.0, 0, 1.278178 Data0, 0034, 736793.0034722222, 2018-04-08T20:05:00.000000, 1.27801, 1.27856, 1.27791, 1.27792, 0.0, 0, 1.278148 Data0, 0035, 736793.0069444445, 2018-04-08T20:10:00.000000, 1.27795, 1.27806, 1.27773, 1.27804, 0.0, 0, 1.2780799999999999# Querying historical data $ python vendor/backtrader/samples/ibtest/ibtest.py --host=host --port=port --data0=USD.CAD-CASH-IDEALPRO --timeframe=Minutes --compression=5 --clientId=985 --fromdate=2018-04-8 --resample --historical --stopafter=10 ... Data0, 0024, 736792.9652777778, 2018-04-08T19:10:00.000000, 1.27741, 1.2776, 1.27736, 1.2776, -1.0, 0, 1.277434 Data0, 0025, 736792.96875, 2018-04-08T19:15:00.000000, 1.2776, 1.27775, 1.27757, 1.27769, -1.0, 0, 1.277504 Data0, 0026, 736792.9722222222, 2018-04-08T19:20:00.000000, 1.27769, 1.2783, 1.27767, 1.27826, -1.0, 0, 1.277688 Data0, 0027, 736792.9756944445, 2018-04-08T19:25:00.000000, 1.27826, 1.27837, 1.27817, 1.27822, -1.0, 0, 1.277836 Data0, 0028, 736792.9791666666, 2018-04-08T19:30:00.000000, 1.27822, 1.27823, 1.27788, 1.27807, -1.0, 0, 1.277968 Data0, 0029, 736792.9826388889, 2018-04-08T19:35:00.000000, 1.27807, 1.27842, 1.27807, 1.27838, -1.0, 0, 1.278124 Data0, 0030, 736792.9861111111, 2018-04-08T19:40:00.000000, 1.27838, 1.27839, 1.2782, 1.27821, -1.0, 0, 1.278228 Data0, 0031, 736792.9895833334, 2018-04-08T19:45:00.000000, 1.27821, 1.27837, 1.27804, 1.27819, -1.0, 0, 1.278214 Data0, 0032, 736792.9930555555, 2018-04-08T19:50:00.000000, 1.27819, 1.27827, 1.27804, 1.27804, -1.0, 0, 1.278178 Data0, 0033, 736792.9965277778, 2018-04-08T19:55:00.000000, 1.27804, 1.2783, 1.27797, 1.2783, -1.0, 0, 1.278224 Data0, 0034, 736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27791, 1.27792, -1.0, 0, 1.278132 Data0, 0035, 736793.0034722222, 2018-04-08T20:05:00.000000, 1.27792, 1.27806, 1.27773, 1.27806, -1.0, 0, 1.278102 Data0, 0036, 736793.0069444445, 2018-04-08T20:10:00.000000, 1.27806, 1.27806, 1.27788, 1.27788, -1.0, 0, 1.27804Here is the same process, but run with
compression=60(so 1 hour). You can see that the resampled live bar at23:00is roughly equal (HLC are equal) to the historical bar at22:00.Live
Data0, 0001, 736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan Data0, 0002, 736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan Data0, 0003, 736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan Data0, 0004, 736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.2773, 1.27665, 1.27701, -1.0, 0, nan ***** DATA NOTIF: LIVE Data0, 0005, 736793.0833333334, 2018-04-08T22:00:00.000000, 1.27701, 1.27741, 1.27662, 1.27737, 0.0, 0, 1.2773379999999999 Data0, 0006, 736793.125, 2018-04-08T23:00:00.000000, 1.27736, 1.27781, 1.27707, 1.27727, 0.0, 0, 1.277324Historical
Data0, 0001, 736792.9166666666, 2018-04-08T18:00:00.000000, 1.27684, 1.27823, 1.27623, 1.27734, -2.0, 0, nan Data0, 0002, 736792.9583333334, 2018-04-08T19:00:00.000000, 1.27734, 1.27842, 1.2772, 1.2783, -1.0, 0, nan Data0, 0003, 736793.0, 2018-04-08T20:00:00.000000, 1.2783, 1.27856, 1.27666, 1.27667, -1.0, 0, nan Data0, 0004, 736793.0416666666, 2018-04-08T21:00:00.000000, 1.27667, 1.27741, 1.27662, 1.27739, -1.0, 0, nan Data0, 0005, 736793.0833333334, 2018-04-08T22:00:00.000000, 1.27739, 1.27781, 1.27707, 1.27727, -1.0, 0, 1.277394 Data0, 0006, 736793.125, 2018-04-08T23:00:00.000000, 1.27727, 1.27746, 1.27721, 1.27738, -1.0, 0, 1.277402diff --git a/samples/ibtest/ibtest.py b/samples/ibtest/ibtest.py index 8000f41..4530be9 100644 --- a/samples/ibtest/ibtest.py +++ b/samples/ibtest/ibtest.py @@ -93,7 +93,7 @@ class TestStrategy(bt.Strategy): txt.append('Data0') txt.append('%04d' % len(self.data0)) dtfmt = '%Y-%m-%dT%H:%M:%S.%f' - txt.append('{}'.format(self.data.datetime[0])) + txt.append('{:>30}'.format(self.data.datetime[0])) txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt)) txt.append('{}'.format(self.data.open[0])) txt.append('{}'.format(self.data.high[0])) @@ -253,11 +253,18 @@ def runstrategy(): dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate)) fromdate = datetime.datetime.strptime(args.fromdate, dtformat) + todate = None + if args.todate: + dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate)) + todate = datetime.datetime.strptime(args.todate,dtformat) + else: + todate = datetime.datetime.now() + IBDataFactory = ibstore.getdata if args.usestore else bt.feeds.IBData datakwargs = dict( timeframe=datatf, compression=datacomp, - historical=args.historical, fromdate=fromdate, + historical=args.historical, fromdate=fromdate, todate=todate, rtbar=args.rtbar, qcheck=args.qcheck, what=args.what, @@ -441,6 +448,11 @@ def parse_args(): help=('Starting date for historical download ' 'with format: YYYY-MM-DD[THH:MM:SS]')) + parser.add_argument('--todate', + required=False, action='store', + help=('Ending date for historical download ' + 'with format: YYYY-MM-DD[THH:MM:SS]')) + parser.add_argument('--smaperiod', default=5, type=int, required=False, action='store', help='Period to apply to the Simple Moving Average') -
I did some checks:
i used the following params in resample
rightedge=False,
adjbartime=True,
bar2edge=True,When using rightedge=False, the tick will be immediately be delivered, the time will not be adjusted.
This seems to be connected to _adjusttime in Resampler and the param greater. I replaced it with self.p.rightedge where it occured. This at least provided me correct timestamps in the resampled data. But one issue was, that the first incoming live data was not added to an existing bar. Everything else was correct after playing around with the greater param.
@backtrader so if the rightedge param is False, the time is not being adjusted to the leftedge, instead is is delivered instantly. maybe this helps to find the issue.
-
@dasch Do you think that it's an issue and not misuse of the API? I wanted to use
ibtest.pyto reproduce it so that there is no "user" code in there. -
It's not really a issue. Its how backtrader works. By default it adjusts the data to the rightedge of the current timeframe. You would need to use rightedge=False, so the data is aligned to the left edge.
It may be that there is a bug in backtrader because rightedge=False is being ignored, but i am not sure.
@backtrader it seems this code may be the culprint?
File resamplerfilter.py
def _adjusttime(self, greater=False, forcedata=None): ''' Adjusts the time of calculated bar (from underlying data source) by using the timeframe to the appropriate boundary, with compression taken into account Depending on param ``rightedge`` uses the starting boundary or the ending one ''' dtnum = self._calcadjtime(greater=greater) if greater and dtnum <= self.bar.datetime: return False self.bar.datetime = dtnum return True -
It breaks when the adjusted time in live trade with rightedge = False is lower than the time of the last bar?
I got a tick
self.bar.datetime = DATE 2018-04-09T11:54:25.976293 (736793.4961339849)
the adjusted time with rightedge = False is:
dtnum = DATE 2018-04-09 11:54:00 (736793.495833333)
with rightedge = True it would be:
dtnum = DATE 2018-04-09 11:54:30 (this will be higher than self.bar.datetime)when _adjusttime is being called with greater=True then the codition:
if greater and dtnum <= self.bar.datetime:is False, so no bar time is being updated.
-
Thanks for your help. The PEBKAC is resolved :-)
I guess it is obvious to you, but perhaps a default mode of "backtest bars should match live bars" would follow the principle of least surprise? In any case, here are some live resampled bars, followed by the corresponding historical resampled bars with a timeframe of 5 minutes with
rightedge=False. This shows that live bars match historical bars. Now the question I have to answer myself is: "why didrighedge=Falsenot work in my system when I tried it live?"Live
Data0, 0196, 736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366 Data0, 0197, 736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632 Data0, 0198, 736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775 Data0, 0199, 736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784 Data0, 0200, 736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684 Data0, 0201, 736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775 Data0, 0202, 736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176 Data0, 0203, 736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711 Data0, 0204, 736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002 Data0, 0205, 736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034Historical
Data0, 0196, 736793.5625, 2018-04-09T09:30:00.000000, 1.27725, 1.27794, 1.27709, 1.27773, -1.0, 0, 1.277366 Data0, 0197, 736793.5659722222, 2018-04-09T09:35:00.000000, 1.27773, 1.27882, 1.27745, 1.27874, -1.0, 0, 1.277632 Data0, 0198, 736793.5694444445, 2018-04-09T09:40:00.000000, 1.27874, 1.27876, 1.27756, 1.27769, -1.0, 0, 1.27775 Data0, 0199, 736793.5729166666, 2018-04-09T09:45:00.000000, 1.27769, 1.27791, 1.27729, 1.27751, -1.0, 0, 1.277784 Data0, 0200, 736793.5763888889, 2018-04-09T09:50:00.000000, 1.27751, 1.27751, 1.2765, 1.27675, -1.0, 0, 1.277684 Data0, 0201, 736793.5798611111, 2018-04-09T09:55:00.000000, 1.27675, 1.27726, 1.27634, 1.27681, -1.0, 0, 1.2775 Data0, 0202, 736793.5833333334, 2018-04-09T10:00:00.000000, 1.27681, 1.27716, 1.27641, 1.27712, -1.0, 0, 1.277176 Data0, 0203, 736793.5868055555, 2018-04-09T10:05:00.000000, 1.27712, 1.27748, 1.27683, 1.27736, -1.0, 0, 1.27711 Data0, 0204, 736793.5902777778, 2018-04-09T10:10:00.000000, 1.27736, 1.27757, 1.27725, 1.27736, -1.0, 0, 1.2770800000000002 Data0, 0205, 736793.59375, 2018-04-09T10:15:00.000000, 1.27736, 1.2774, 1.27652, 1.27652, -1.0, 0, 1.277034