[Benchmark Observer] Issue when benchmarking with resampled data

dimitar-petrov

Hello mementum/gents,

I am facing an issue when adding benchmark observer with resampled data.
Here is a short example.

import backtrader as bt

class Strat(bt.Strategy):

  def __init__(self):
    self.ema = bt.ind.ExponentialMovingAverage(period=2)

  def next(self):
      pass

cerebro = bt.Cerebro()
dfile = '/tmp/test_data.csv'

data = bt.feeds.GenericCSVData(dataname=dfile)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1)
cerebro.addobserver(bt.observers.Benchmark,
                    data=cerebro.datas[0],
                    timeframe=bt.TimeFrame.NoTimeFrame)

cerebro.addstrategy(Strat)

import pdb; pdb.set_trace()
cerebro.run()

and that is the data feeded to the platform:

2014-08-04 10:00:00,91.3808,91.4188,90.4414,90.356,551022849.0,288493
2014-08-04 11:00:00,90.4224,90.6407,90.3466,90.5268,972065815.0,177483
2014-08-04 12:00:00,90.5363,90.821,90.375,90.84,1245762598.0,80312
2014-08-04 13:00:00,90.8305,90.8314,90.6075,90.7251,1453963196.0,44641
2014-08-04 14:00:00,90.7545,90.9348,90.6312,90.9633,1628333296.0,46629
2014-08-04 15:00:00,90.9729,91.3171,90.783,90.7924,1894081674.0,297779
2014-08-04 16:00:00,90.802,90.6976,90.764,90.7071,36304379.0,927375
2014-08-05 09:00:00,90.5743,90.4319,89.5589,89.7013,119958953.0,150692
2014-08-05 10:00:00,89.7107,90.2516,89.6158,90.0048,757768836.0,144909
2014-08-05 11:00:00,90.0618,90.1662,89.8673,89.8866,1153074048.0,85007
2014-08-05 12:00:00,89.9859,90.0998,89.8151,90.0114,1437677659.0,26554
2014-08-05 13:00:00,90.0808,90.6881,89.91,90.318,1890843871.0,138986
2014-08-05 14:00:00,90.3465,90.3275,89.7202,90.2638,2557173264.0,56034
2014-08-05 15:00:00,90.2611,90.3845,90.0248,90.2706,2934212054.0,272037
2014-08-05 16:00:00,90.3086,90.2232,90.2706,90.2801,53558659.0,740163
2014-08-06 09:00:00,90.1472,90.5268,89.91,90.1188,110307201.0,176703
2014-08-06 10:00:00,90.1235,90.3086,89.9385,90.337,623044588.0,95524
2014-08-06 11:00:00,90.4129,90.4034,90.0903,90.3086,997681083.0,34679
2014-08-06 12:00:00,90.356,90.356,90.0902,90.1852,1240101549.0,21805
2014-08-06 13:00:00,90.2326,90.2801,90.0799,90.1273,1438332050.0,39124
2014-08-06 14:00:00,90.1378,90.2829,89.9764,90.2687,1662506155.0,42895
2014-08-06 15:00:00,90.2706,90.337,90.1188,90.1661,1933269928.0,182204
2014-08-06 16:00:00,90.1662,90.0903,90.1567,90.1093,35204618.0,470100
2014-08-07 09:00:00,90.5957,90.7101,90.3477,90.7673,105480730.0,95062
2014-08-07 10:00:00,90.8055,91.4349,90.5957,91.0677,661284861.0,120101
2014-08-07 11:00:00,91.0725,91.2346,90.6624,90.6481,1191828033.0,150209
2014-08-07 12:00:00,90.7387,90.8818,90.3954,90.5194,1602115067.0,80080
2014-08-07 13:00:00,90.5194,90.8551,90.5099,90.8344,1855378746.0,27218
2014-08-07 14:00:00,90.8532,90.8341,90.3668,90.3954,2029750367.0,38943
2014-08-07 15:00:00,90.405,90.4717,89.7469,89.9854,2379287631.0,169671
2014-08-07 16:00:00,90.0998,89.9758,89.9854,90.0998,43999921.0,430020
2014-08-08 09:00:00,90.0712,89.786,89.3273,89.7284,132831979.0,81711
2014-08-08 10:00:00,89.7279,89.7183,89.0603,89.5085,657399391.0,42598
2014-08-08 11:00:00,89.5467,89.5943,89.3021,89.4322,993243343.0,78475
2014-08-08 12:00:00,89.4609,89.5181,89.3273,89.5371,1198958712.0,27913
2014-08-08 13:00:00,89.5467,90.1952,89.4895,90.0903,1463132880.0,63001
2014-08-08 14:00:00,90.1067,90.3764,90.0855,90.3572,1806865549.0,66153
2014-08-08 15:00:00,90.362,90.3887,90.1093,90.3764,2095341481.0,268884
2014-08-08 16:00:00,90.424,90.3096,90.3668,90.3382,39785070.0,1019502.0

Exception:


/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/cerebro.py in _runnext(self, runstrats)
   1555                 self._check_timers(runstrats, dt0, cheat=False)
   1556                 for strat in runstrats:
-> 1557                     strat._next()
   1558                     if self._event_stop:  # stop if requested
   1559                         return

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next(self)
    326         minperstatus = self._getminperstatus()
    327         self._next_analyzers(minperstatus)
--> 328         self._next_observers(minperstatus)
    329 
    330         self.clear()

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next_observers(self, minperstatus, once)
    354                     observer.prenext()
    355             else:
--> 356                 observer._next()
    357 
    358     def _next_analyzers(self, minperstatus, once=False):

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in _next(self)
    262                 self.next()
    263             elif clock_len == self._minperiod:
--> 264                 self.nextstart()  # only called for the 1st value
    265             elif clock_len:
    266                 self.prenext()

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in nextstart(self)
    327 
    328         # Called once for 1st full calculation - defaults to regular next
--> 329         self.next()
    330 
    331     def next(self):

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/benchmark.py in next(self)
    101 
    102     def next(self):
--> 103         super(Benchmark, self).next()
    104         self.lines.benchmark[0] = self.tbench.rets[self.treturn.dtkey]
    105 

/home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/timereturn.py in next(self)
     76 
     77     def next(self):
---> 78         self.lines.timereturn[0] = self.treturn.rets[self.treturn.dtkey]

KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)

Something is getting mixed with the slave timereturn analyzers of benchmark.
I have been looking at it for couple of hours but unable to pin point the exact issue.
The difference I am seeing when I remove the resampler and just feed the data is here:

        self._doreplay = self._doreplay or any(x.replaying for x in self.datas)
        if self._doreplay:
            # preloading is not supported with replay. full timeframe bars
            # are constructed in realtime
            self._dopreload = False

If one case dopreload is True and benchmark is working in the othercase due to the resampled data it is False and benchmark observer is getting confused.

BR,
Dimitar

dimitar-petrov

Anyone has an idea?

backtrader

@dimitar-petrov said in [Benchmark Observer] Issue when benchmarking with resampled data:

KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)

This datetime is used as an initial key before the analyzer has seen enough data to calculate the returns. This shows that the analyzer has actually not calculated anything.

It seems that the timeframe being considered is NoTimeFrame, which will be because the resampled data has no actual parameter timeframe.

It might be because the _timeframe internal attribute in the resampled data is being set late by the filter. Too late for the analyzer to catch up.

Will be looked into.

dimitar-petrov

Great, thanks!

backtrader

Nothing to do with the assumption but rather with the distinction between runonce=True and runonce=False. It is actually a chain of coincidences in that changes to the Benchmark analyzer to skip the prenext phase and only make a comparison from the actual trading starting point, impact when the analyzer actually delivers values. Supporting the two aforementioned runonce modes for observers and analyzers is done at different points in the code (unluckily it cannot be done with the same internal tools) and the observer by skipping prenext was coming out of sync with the strategy.

Resampling data triggers this chain of events because it automatically disables preloading in the background, which in turns set runonce=False

Soon to be corrected.

backtrader

There is a fix in the latest commit of the development branch. It seems in any case to have a side effect for writers. So the release that was planned with it will be delayed.

dimitar-petrov

Works for me,
Thanks