Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!
-
Hi everyone,
this is a very weird one. I am using cerebro with a pretty straightforward strategy (cross of 3 EMAs) and multiple historical data feeds within one run() cycle. It is run by a timer and performs the following steps every couple of minutes:
- recent crypto feed data is being downloaded from Binance and saved as CSV (from 40 to 100 data feeds each time)
- data is added to cerebro
- cerebro is run
- run result is processed elsewhere
It works perfectly for ~ 100 run cycles until cerebro throws an exception - and it happens every time around 1:30 am GMT+1. After it crashes for the first time, all subsequent runs with updated data will crash as well. (of course, new cerebro instance is created every time)
When I debug it in the morning, with the same data set, it works fine.
When I stress-test it with the same data set 1000 times it works fine.
When I change the Windows clock to the time when the exception happened it still works fine.I have run out of ideas - the only problem I can think of - some weird accumulation of some static property - but I am not an python expert to tell... Here is the stack trace and the code:
cerebro = bt.Cerebro() for symbol_name, filename in data.items(): cerebro_data = BinanceCSV ( dataname=filename, fromdate=from_date) cerebro.adddata(cerebro_data, name=symbol_name) cerebro.addstrategy(strategy) result = cerebro.run() signals = result[0].signals
File "D:\CRYPTO\backtrader\signal-notifier\notify_cycle_process.py", line 33, in process_data result = cerebro.run() File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1293, in runstrategies self._runonce(runstrats) File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1652, in _runonce strat._once() File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once indicator._once() File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once indicator._once() File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once self.oncestart(self._minperiod - 1, self._minperiod) File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart super(ExponentialSmoothing, self).once(start, end) File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once dst[i] = math.fsum(src[i - period + 1:i + 1]) / period
Thank you.
-
Could you include the rest of the code? In particular your strategy class.
-
@run-out here you go
class NotifyStrategy(bt.Strategy): params = ( ('ema1', config.signal_params['ema1']), ('ema2', config.signal_params['ema2']), ('ema3', config.signal_params['ema3']), ) def __init__(self): self.inds = dict() self.signals = list() for i, d in enumerate(self.datas): self.inds[d] = dict() self.inds[d]['ema1'] = bt.indicators.ExponentialMovingAverage( d.close, period=self.params.ema1) self.inds[d]['ema2'] = bt.indicators.ExponentialMovingAverage( d.close, period=self.params.ema2) self.inds[d]['ema3'] = bt.indicators.ExponentialMovingAverage( d.close, period=self.params.ema3) self.inds[d]['cross'] = bt.indicators.CrossOver(self.inds[d]['ema1'],self.inds[d]['ema2']) def next(self): for i, d in enumerate(self.datas): dt, dn = self.datetime.date(), d._name pos = self.getposition(d).size if self.inds[d]['cross'][0] == 1 and self.inds[d]['ema1'] > self.inds[d]['ema3']: price = d.close[0] signal = Signal(d._name, True, d.datetime.datetime(), price) self.signals.append(signal) elif self.inds[d]['cross'][0] == -1 and self.inds[d]['ema1'] < self.inds[d]['ema3']: signal = Signal(d._name, False, d.datetime.datetime(), d.close[0]) self.signals.append(signal)
-
@boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
self.oncestart(self._minperiod - 1, self._minperiod)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
super(ExponentialSmoothing, self).once(start, end)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
dst[i] = math.fsum(src[i - period + 1:i + 1]) / periodHard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.
-
@run-out said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:
@boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
self.oncestart(self._minperiod - 1, self._minperiod)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
super(ExponentialSmoothing, self).once(start, end)
File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
dst[i] = math.fsum(src[i - period + 1:i + 1]) / periodHard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.
You were right. The problem lied in this class. I did not know that params will only evaluate once hence current timestamp after a few hours of running became obsolete.
class BinanceCSV(btfeed.GenericCSVData): params = ( ('todate', datetime.datetime.today()), ('nullvalue', 0.0), ('dtformat', ('%Y-%m-%d %H:%M:%S')), ('datetime', 0), ('open', 1), ('high', 2), ('low', 3), ('close', 4), ('volume', 5), ('openinterest', -1), ('timeframe', bt.TimeFrame.Minutes) )