[Benchmark Observer] Issue when benchmarking with resampled data



  • Hello mementum/gents,

    I am facing an issue when adding benchmark observer with resampled data.
    Here is a short example.

    import backtrader as bt
    
    class Strat(bt.Strategy):
    
      def __init__(self):
        self.ema = bt.ind.ExponentialMovingAverage(period=2)
    
      def next(self):
          pass
    
    cerebro = bt.Cerebro()
    dfile = '/tmp/test_data.csv'
    
    data = bt.feeds.GenericCSVData(dataname=dfile)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1)
    cerebro.addobserver(bt.observers.Benchmark,
                        data=cerebro.datas[0],
                        timeframe=bt.TimeFrame.NoTimeFrame)
    
    cerebro.addstrategy(Strat)
    
    import pdb; pdb.set_trace()
    cerebro.run()
    

    and that is the data feeded to the platform:

    2014-08-04 10:00:00,91.3808,91.4188,90.4414,90.356,551022849.0,288493
    2014-08-04 11:00:00,90.4224,90.6407,90.3466,90.5268,972065815.0,177483
    2014-08-04 12:00:00,90.5363,90.821,90.375,90.84,1245762598.0,80312
    2014-08-04 13:00:00,90.8305,90.8314,90.6075,90.7251,1453963196.0,44641
    2014-08-04 14:00:00,90.7545,90.9348,90.6312,90.9633,1628333296.0,46629
    2014-08-04 15:00:00,90.9729,91.3171,90.783,90.7924,1894081674.0,297779
    2014-08-04 16:00:00,90.802,90.6976,90.764,90.7071,36304379.0,927375
    2014-08-05 09:00:00,90.5743,90.4319,89.5589,89.7013,119958953.0,150692
    2014-08-05 10:00:00,89.7107,90.2516,89.6158,90.0048,757768836.0,144909
    2014-08-05 11:00:00,90.0618,90.1662,89.8673,89.8866,1153074048.0,85007
    2014-08-05 12:00:00,89.9859,90.0998,89.8151,90.0114,1437677659.0,26554
    2014-08-05 13:00:00,90.0808,90.6881,89.91,90.318,1890843871.0,138986
    2014-08-05 14:00:00,90.3465,90.3275,89.7202,90.2638,2557173264.0,56034
    2014-08-05 15:00:00,90.2611,90.3845,90.0248,90.2706,2934212054.0,272037
    2014-08-05 16:00:00,90.3086,90.2232,90.2706,90.2801,53558659.0,740163
    2014-08-06 09:00:00,90.1472,90.5268,89.91,90.1188,110307201.0,176703
    2014-08-06 10:00:00,90.1235,90.3086,89.9385,90.337,623044588.0,95524
    2014-08-06 11:00:00,90.4129,90.4034,90.0903,90.3086,997681083.0,34679
    2014-08-06 12:00:00,90.356,90.356,90.0902,90.1852,1240101549.0,21805
    2014-08-06 13:00:00,90.2326,90.2801,90.0799,90.1273,1438332050.0,39124
    2014-08-06 14:00:00,90.1378,90.2829,89.9764,90.2687,1662506155.0,42895
    2014-08-06 15:00:00,90.2706,90.337,90.1188,90.1661,1933269928.0,182204
    2014-08-06 16:00:00,90.1662,90.0903,90.1567,90.1093,35204618.0,470100
    2014-08-07 09:00:00,90.5957,90.7101,90.3477,90.7673,105480730.0,95062
    2014-08-07 10:00:00,90.8055,91.4349,90.5957,91.0677,661284861.0,120101
    2014-08-07 11:00:00,91.0725,91.2346,90.6624,90.6481,1191828033.0,150209
    2014-08-07 12:00:00,90.7387,90.8818,90.3954,90.5194,1602115067.0,80080
    2014-08-07 13:00:00,90.5194,90.8551,90.5099,90.8344,1855378746.0,27218
    2014-08-07 14:00:00,90.8532,90.8341,90.3668,90.3954,2029750367.0,38943
    2014-08-07 15:00:00,90.405,90.4717,89.7469,89.9854,2379287631.0,169671
    2014-08-07 16:00:00,90.0998,89.9758,89.9854,90.0998,43999921.0,430020
    2014-08-08 09:00:00,90.0712,89.786,89.3273,89.7284,132831979.0,81711
    2014-08-08 10:00:00,89.7279,89.7183,89.0603,89.5085,657399391.0,42598
    2014-08-08 11:00:00,89.5467,89.5943,89.3021,89.4322,993243343.0,78475
    2014-08-08 12:00:00,89.4609,89.5181,89.3273,89.5371,1198958712.0,27913
    2014-08-08 13:00:00,89.5467,90.1952,89.4895,90.0903,1463132880.0,63001
    2014-08-08 14:00:00,90.1067,90.3764,90.0855,90.3572,1806865549.0,66153
    2014-08-08 15:00:00,90.362,90.3887,90.1093,90.3764,2095341481.0,268884
    2014-08-08 16:00:00,90.424,90.3096,90.3668,90.3382,39785070.0,1019502.0
    

    Exception:

    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/cerebro.py in _runnext(self, runstrats)
       1555                 self._check_timers(runstrats, dt0, cheat=False)
       1556                 for strat in runstrats:
    -> 1557                     strat._next()
       1558                     if self._event_stop:  # stop if requested
       1559                         return
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next(self)
        326         minperstatus = self._getminperstatus()
        327         self._next_analyzers(minperstatus)
    --> 328         self._next_observers(minperstatus)
        329 
        330         self.clear()
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/strategy.py in _next_observers(self, minperstatus, once)
        354                     observer.prenext()
        355             else:
    --> 356                 observer._next()
        357 
        358     def _next_analyzers(self, minperstatus, once=False):
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in _next(self)
        262                 self.next()
        263             elif clock_len == self._minperiod:
    --> 264                 self.nextstart()  # only called for the 1st value
        265             elif clock_len:
        266                 self.prenext()
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/lineiterator.py in nextstart(self)
        327 
        328         # Called once for 1st full calculation - defaults to regular next
    --> 329         self.next()
        330 
        331     def next(self):
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/benchmark.py in next(self)
        101 
        102     def next(self):
    --> 103         super(Benchmark, self).next()
        104         self.lines.benchmark[0] = self.tbench.rets[self.treturn.dtkey]
        105 
    
    /home/dpetrov/develop/python/py-envs/py3env/lib/python3.6/site-packages/backtrader/observers/timereturn.py in next(self)
         76 
         77     def next(self):
    ---> 78         self.lines.timereturn[0] = self.treturn.rets[self.treturn.dtkey]
    
    KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)
    

    Something is getting mixed with the slave timereturn analyzers of benchmark.
    I have been looking at it for couple of hours but unable to pin point the exact issue.
    The difference I am seeing when I remove the resampler and just feed the data is here:

            self._doreplay = self._doreplay or any(x.replaying for x in self.datas)
            if self._doreplay:
                # preloading is not supported with replay. full timeframe bars
                # are constructed in realtime
                self._dopreload = False
    
    

    If one case dopreload is True and benchmark is working in the othercase due to the resampled data it is False and benchmark observer is getting confused.

    BR,
    Dimitar



  • Anyone has an idea?


  • administrators

    @dimitar-petrov said in [Benchmark Observer] Issue when benchmarking with resampled data:

    KeyError: datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)
    

    This datetime is used as an initial key before the analyzer has seen enough data to calculate the returns. This shows that the analyzer has actually not calculated anything.

    It seems that the timeframe being considered is NoTimeFrame, which will be because the resampled data has no actual parameter timeframe.

    It might be because the _timeframe internal attribute in the resampled data is being set late by the filter. Too late for the analyzer to catch up.

    Will be looked into.



  • Great, thanks!


  • administrators

    Nothing to do with the assumption but rather with the distinction between runonce=True and runonce=False. It is actually a chain of coincidences in that changes to the Benchmark analyzer to skip the prenext phase and only make a comparison from the actual trading starting point, impact when the analyzer actually delivers values. Supporting the two aforementioned runonce modes for observers and analyzers is done at different points in the code (unluckily it cannot be done with the same internal tools) and the observer by skipping prenext was coming out of sync with the strategy.

    Resampling data triggers this chain of events because it automatically disables preloading in the background, which in turns set runonce=False

    Soon to be corrected.


  • administrators

    There is a fix in the latest commit of the development branch. It seems in any case to have a side effect for writers. So the release that was planned with it will be delayed.



  • Works for me,
    Thanks


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.