Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!

boddy33

Hi everyone,

this is a very weird one. I am using cerebro with a pretty straightforward strategy (cross of 3 EMAs) and multiple historical data feeds within one run() cycle. It is run by a timer and performs the following steps every couple of minutes:

recent crypto feed data is being downloaded from Binance and saved as CSV (from 40 to 100 data feeds each time)
data is added to cerebro
cerebro is run
run result is processed elsewhere

It works perfectly for ~ 100 run cycles until cerebro throws an exception - and it happens every time around 1:30 am GMT+1. After it crashes for the first time, all subsequent runs with updated data will crash as well. (of course, new cerebro instance is created every time)

When I debug it in the morning, with the same data set, it works fine.
When I stress-test it with the same data set 1000 times it works fine.
When I change the Windows clock to the time when the exception happened it still works fine.

I have run out of ideas - the only problem I can think of - some weird accumulation of some static property - but I am not an python expert to tell... Here is the stack trace and the code:

cerebro = bt.Cerebro()
    for symbol_name, filename in data.items():
        cerebro_data = BinanceCSV (
              dataname=filename,
              fromdate=from_date)
        cerebro.adddata(cerebro_data, name=symbol_name)
    cerebro.addstrategy(strategy)
    result = cerebro.run()
    signals = result[0].signals

File "D:\CRYPTO\backtrader\signal-notifier\notify_cycle_process.py", line 33, in process_data
  result = cerebro.run()
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
  runstrat = self.runstrategies(iterstrat)
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1293, in runstrategies
  self._runonce(runstrats)
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1652, in _runonce
  strat._once()
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once
  indicator._once()
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once
  indicator._once()
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
  self.oncestart(self._minperiod - 1, self._minperiod)
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
  super(ExponentialSmoothing, self).once(start, end)
File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
  dst[i] = math.fsum(src[i - period + 1:i + 1]) / period

Thank you.

run-out

Could you include the rest of the code? In particular your strategy class.

boddy33

@run-out here you go

class NotifyStrategy(bt.Strategy):

    params = ( ('ema1', config.signal_params['ema1']),
               ('ema2', config.signal_params['ema2']),
               ('ema3', config.signal_params['ema3']), )

    def __init__(self):
        self.inds = dict()
        self.signals = list()
        for i, d in enumerate(self.datas):
            self.inds[d] = dict()
            self.inds[d]['ema1'] = bt.indicators.ExponentialMovingAverage(
                d.close, period=self.params.ema1)
            self.inds[d]['ema2'] = bt.indicators.ExponentialMovingAverage(
                d.close, period=self.params.ema2)
            self.inds[d]['ema3'] = bt.indicators.ExponentialMovingAverage(
                d.close, period=self.params.ema3)
            self.inds[d]['cross'] = bt.indicators.CrossOver(self.inds[d]['ema1'],self.inds[d]['ema2'])
    
    def next(self):
        for i, d in enumerate(self.datas):
            dt, dn = self.datetime.date(), d._name
            pos = self.getposition(d).size    
            if self.inds[d]['cross'][0] == 1 and self.inds[d]['ema1'] > self.inds[d]['ema3']:
                price = d.close[0]
                signal = Signal(d._name, True, d.datetime.datetime(), price)
                self.signals.append(signal)
            elif self.inds[d]['cross'][0] == -1 and self.inds[d]['ema1'] < self.inds[d]['ema3']:
                signal = Signal(d._name, False, d.datetime.datetime(), d.close[0])
                self.signals.append(signal)

run-out

@boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
self.oncestart(self._minperiod - 1, self._minperiod)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
super(ExponentialSmoothing, self).once(start, end)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
dst[i] = math.fsum(src[i - period + 1:i + 1]) / period

Hard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.

boddy33

@run-out said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

@boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
self.oncestart(self._minperiod - 1, self._minperiod)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
super(ExponentialSmoothing, self).once(start, end)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
dst[i] = math.fsum(src[i - period + 1:i + 1]) / period

Hard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.

You were right. The problem lied in this class. I did not know that params will only evaluate once hence current timestamp after a few hours of running became obsolete.

class BinanceCSV(btfeed.GenericCSVData):

  params = (
    ('todate', datetime.datetime.today()),
    ('nullvalue', 0.0),
    ('dtformat', ('%Y-%m-%d %H:%M:%S')),
    ('datetime', 0),
    ('open', 1),
    ('high', 2),
    ('low', 3),
    ('close', 4),
    ('volume', 5),
    ('openinterest', -1),
    ('timeframe', bt.TimeFrame.Minutes)
)