[Benchmark Observer] Issue when benchmarking with resampled data
-
Hello mementum/gents,
I am facing an issue when adding benchmark observer with resampled data.
Here is a short example.import backtrader as bt class Strat(bt.Strategy): def __init__(self): self.ema = bt.ind.ExponentialMovingAverage(period=2) def next(self): pass cerebro = bt.Cerebro() dfile = '/tmp/test_data.csv' data = bt.feeds.GenericCSVData(dataname=dfile) cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1) cerebro.addobserver(bt.observers.Benchmark, data=cerebro.datas[0], timeframe=bt.TimeFrame.NoTimeFrame) cerebro.addstrategy(Strat) import pdb; pdb.set_trace() cerebro.run()
and that is the data feeded to the platform:
2014-08-04 10:00:00,91.3808,91.4188,90.4414,90.356,551022849.0,288493 2014-08-04 11:00:00,90.4224,90.6407,90.3466,90.5268,972065815.0,177483 2014-08-04 12:00:00,90.5363,90.821,90.375,90.84,1245762598.0,80312 2014-08-04 13:00:00,90.8305,90.8314,90.6075,90.7251,1453963196.0,44641 2014-08-04 14:00:00,90.7545,90.9348,90.6312,90.9633,1628333296.0,46629 2014-08-04 15:00:00,90.9729,91.3171,90.783,90.7924,1894081674.0,297779 2014-08-04 16:00:00,90.802,90.6976,90.764,90.7071,36304379.0,927375 2014-08-05 09:00:00,90.5743,90.4319,89.5589,89.7013,119958953.0,150692 2014-08-05 10:00:00,89.7107,90.2516,89.6158,90.0048,757768836.0,144909 2014-08-05 11:00:00,90.0618,90.1662,89.8673,89.8866,1153074048.0,85007 2014-08-05 12:00:00,89.9859,90.0998,89.8151,90.0114,1437677659.0,26554 2014-08-05 13:00:00,90.0808,90.6881,89.91,90.318,1890843871.0,138986 2014-08-05 14:00:00,90.3465,90.3275,89.7202,90.2638,2557173264.0,56034 2014-08-05 15:00:00,90.2611,90.3845,90.0248,90.2706,2934212054.0,272037 2014-08-05 16:00:00,90.3086,90.2232,90.2706,90.2801,53558659.0,740163 2014-08-06 09:00:00,90.1472,90.5268,89.91,90.1188,110307201.0,176703 2014-08-06 10:00:00,90.1235,90.3086,89.9385,90.337,623044588.0,95524 2014-08-06 11:00:00,90.4129,90.4034,90.0903,90.3086,997681083.0,34679 2014-08-06 12:00:00,90.356,90.356,90.0902,90.1852,1240101549.0,21805 2014-08-06 13:00:00,90.2326,90.2801,90.0799,90.1273,1438332050.0,39124 2014-08-06 14:00:00,90.1378,90.2829,89.9764,90.2687,1662506155.0,42895 2014-08-06 15:00:00,90.2706,90.337,90.1188,90.1661,1933269928.0,182204 2014-08-06 16:00:00,90.1662,90.0903,90.1567,90.1093,35204618.0,470100 2014-08-07 09:00:00,90.5957,90.7101,90.3477,90.7673,105480730.0,95062 2014-08-07 10:00:00,90.8055,91.4349,90.5957,91.0677,661284861.0,120101 2014-08-07 11:00:00,91.0725,91.2346,90.6624,90.6481,1191828033.0,150209 2014-08-07 12:00:00,90.7387,90.8818,90.3954,90.5194,1602115067.0,80080 2014-08-07 13:00:00,90.5194,90.8551,90.5099,90.8344,1855378746.0,27218 2014-08-07 14:00:00,90.8532,90.8341,90.3668,90.3954,2029750367.0,38943 2014-08-07 15:00:00,90.405,90.4717,89.7469,89.9854,2379287631.0,169671 2014-08-07 16:00:00,90.0998,89.9758,89.9854,90.0998,43999921.0,430020 2014-08-08 09:00:00,90.0712,89.786,89.3273,89.7284,132831979.0,81711 2014-08-08 10:00:00,89.7279,89.7183,89.0603,89.5085,657399391.0,42598 2014-08-08 11:00:00,89.5467,89.5943,89.3021,89.4322,993243343.0,78475 2014-08-08 12:00:00,89.4609,89.5181,89.3273,89.5371,1198958712.0,27913 2014-08-08 13:00:00,89.5467,90.1952,89.4895,90.0903,1463132880.0,63001 2014-08-08 14:00:00,90.1067,90.3764,90.0855,90.3572,1806865549.0,66153 2014-08-08 15:00:00,90.362,90.3887,90.1093,90.3764,2095341481.0,268884 2014-08-08 16:00:00,90.424,90.3096,90.3668,90.3382,39785070.0,1019502.0
Exception:
/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/cerebro.py in _runnext(self, runstrats) 1555 self._check_timers(runstrats, dt0, cheat=False) 1556 for strat in runstrats: -> 1557 strat._next() 1558 if self._event_stop: # stop if requested 1559 return /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next(self) 326 minperstatus = self._getminperstatus() 327 self._next_analyzers(minperstatus) --> 328 self._next_observers(minperstatus) 329 330 self.clear() /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next_observers(self, minperstatus, once) 354 observer.prenext() 355 else: --> 356 observer._next() 357 358 def _next_analyzers(self, minperstatus, once=False): /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in _next(self) 262 self.next() 263 elif clock_len == self._minperiod: --> 264 self.nextstart() # only called for the 1st value 265 elif clock_len: 266 self.prenext() /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in nextstart(self) 327 328 # Called once for 1st full calculation - defaults to regular next --> 329 self.next() 330 331 def next(self): /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/benchmark.py in next(self) 101 102 def next(self): --> 103 super(Benchmark, self).next() 104 self.lines.benchmark[0] = self.tbench.rets[self.treturn.dtkey] 105 /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/timereturn.py in next(self) 76 77 def next(self): ---> 78 self.lines.timereturn[0] = self.treturn.rets[self.treturn.dtkey] KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)
Something is getting mixed with the slave timereturn analyzers of benchmark.
I have been looking at it for couple of hours but unable to pin point the exact issue.
The difference I am seeing when I remove the resampler and just feed the data is here:self._doreplay = self._doreplay or any(x.replaying for x in self.datas) if self._doreplay: # preloading is not supported with replay. full timeframe bars # are constructed in realtime self._dopreload = False
If one case dopreload is True and benchmark is working in the othercase due to the resampled data it is False and benchmark observer is getting confused.
BR,
Dimitar -
Anyone has an idea?
-
@dimitar-petrov said in [Benchmark Observer] Issue when benchmarking with resampled data:
KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)
This
datetime
is used as an initial key before the analyzer has seen enough data to calculate the returns. This shows that the analyzer has actually not calculated anything.It seems that the timeframe being considered is
NoTimeFrame
, which will be because the resampled data has no actual parametertimeframe
.It might be because the
_timeframe
internal attribute in the resampled data is being set late by the filter. Too late for the analyzer to catch up.Will be looked into.
-
Great, thanks!
-
Nothing to do with the assumption but rather with the distinction between
runonce=True
andrunonce=False
. It is actually a chain of coincidences in that changes to theBenchmark
analyzer to skip theprenext
phase and only make a comparison from the actual trading starting point, impact when the analyzer actually delivers values. Supporting the two aforementionedrunonce
modes for observers and analyzers is done at different points in the code (unluckily it cannot be done with the same internal tools) and the observer by skippingprenext
was coming out of sync with the strategy.Resampling data triggers this chain of events because it automatically disables preloading in the background, which in turns set
runonce=False
Soon to be corrected.
-
There is a fix in the latest commit of the development branch. It seems in any case to have a side effect for writers. So the release that was planned with it will be delayed.
-
Works for me,
Thanks