Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!

    General Code/Help
    2
    5
    118
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • boddy33
      boddy33 last edited by

      Hi everyone,

      this is a very weird one. I am using cerebro with a pretty straightforward strategy (cross of 3 EMAs) and multiple historical data feeds within one run() cycle. It is run by a timer and performs the following steps every couple of minutes:

      • recent crypto feed data is being downloaded from Binance and saved as CSV (from 40 to 100 data feeds each time)
      • data is added to cerebro
      • cerebro is run
      • run result is processed elsewhere

      It works perfectly for ~ 100 run cycles until cerebro throws an exception - and it happens every time around 1:30 am GMT+1. After it crashes for the first time, all subsequent runs with updated data will crash as well. (of course, new cerebro instance is created every time)

      When I debug it in the morning, with the same data set, it works fine.
      When I stress-test it with the same data set 1000 times it works fine.
      When I change the Windows clock to the time when the exception happened it still works fine.

      I have run out of ideas - the only problem I can think of - some weird accumulation of some static property - but I am not an python expert to tell... Here is the stack trace and the code:

      cerebro = bt.Cerebro()
          for symbol_name, filename in data.items():
              cerebro_data = BinanceCSV (
                    dataname=filename,
                    fromdate=from_date)
              cerebro.adddata(cerebro_data, name=symbol_name)
          cerebro.addstrategy(strategy)
          result = cerebro.run()
          signals = result[0].signals
      
      File "D:\CRYPTO\backtrader\signal-notifier\notify_cycle_process.py", line 33, in process_data
        result = cerebro.run()
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1293, in runstrategies
        self._runonce(runstrats)
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\cerebro.py", line 1652, in _runonce
        strat._once()
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once
        indicator._once()
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 297, in _once
        indicator._once()
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
        self.oncestart(self._minperiod - 1, self._minperiod)
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
        super(ExponentialSmoothing, self).once(start, end)
      File "D:\CRYPTO\backtrader\.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
        dst[i] = math.fsum(src[i - period + 1:i + 1]) / period
      

      Thank you.

      run-out 1 Reply Last reply Reply Quote 0
      • run-out
        run-out last edited by

        Could you include the rest of the code? In particular your strategy class.

        RunBacktest.com

        1 Reply Last reply Reply Quote 0
        • boddy33
          boddy33 last edited by

          @run-out here you go

          class NotifyStrategy(bt.Strategy):
          
              params = ( ('ema1', config.signal_params['ema1']),
                         ('ema2', config.signal_params['ema2']),
                         ('ema3', config.signal_params['ema3']), )
          
              def __init__(self):
                  self.inds = dict()
                  self.signals = list()
                  for i, d in enumerate(self.datas):
                      self.inds[d] = dict()
                      self.inds[d]['ema1'] = bt.indicators.ExponentialMovingAverage(
                          d.close, period=self.params.ema1)
                      self.inds[d]['ema2'] = bt.indicators.ExponentialMovingAverage(
                          d.close, period=self.params.ema2)
                      self.inds[d]['ema3'] = bt.indicators.ExponentialMovingAverage(
                          d.close, period=self.params.ema3)
                      self.inds[d]['cross'] = bt.indicators.CrossOver(self.inds[d]['ema1'],self.inds[d]['ema2'])
              
              def next(self):
                  for i, d in enumerate(self.datas):
                      dt, dn = self.datetime.date(), d._name
                      pos = self.getposition(d).size    
                      if self.inds[d]['cross'][0] == 1 and self.inds[d]['ema1'] > self.inds[d]['ema3']:
                          price = d.close[0]
                          signal = Signal(d._name, True, d.datetime.datetime(), price)
                          self.signals.append(signal)
                      elif self.inds[d]['cross'][0] == -1 and self.inds[d]['ema1'] < self.inds[d]['ema3']:
                          signal = Signal(d._name, False, d.datetime.datetime(), d.close[0])
                          self.signals.append(signal)
          
          1 Reply Last reply Reply Quote 0
          • run-out
            run-out @boddy33 last edited by

            @boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

            File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
            self.oncestart(self._minperiod - 1, self._minperiod)
            File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
            super(ExponentialSmoothing, self).once(start, end)
            File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
            dst[i] = math.fsum(src[i - period + 1:i + 1]) / period

            Hard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.

            RunBacktest.com

            1 Reply Last reply Reply Quote 2
            • boddy33
              boddy33 last edited by

              @run-out said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

              @boddy33 said in Cerebro is crashing upon indicator value calculation in the middle of the night - not able to reproduce afterward!:

              File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\lineiterator.py", line 317, in _once
              self.oncestart(self._minperiod - 1, self._minperiod)
              File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 401, in oncestart
              super(ExponentialSmoothing, self).once(start, end)
              File "D:\CRYPTO\backtrader.venv\lib\site-packages\backtrader\indicators\basicops.py", line 364, in once
              dst[i] = math.fsum(src[i - period + 1:i + 1]) / period

              Hard to tell without the code running on my machine. But I would be inclined to put break points in each of the above three spots and look at the start and end times. If your ide supports running in debug mode but only stopping when an error occurs, you should be able to look at these values and see what's going on. Most likely IMHO the end is before the start time. Let us know how you get on.

              You were right. The problem lied in this class. I did not know that params will only evaluate once hence current timestamp after a few hours of running became obsolete.

              class BinanceCSV(btfeed.GenericCSVData):
              
                params = (
                  ('todate', datetime.datetime.today()),
                  ('nullvalue', 0.0),
                  ('dtformat', ('%Y-%m-%d %H:%M:%S')),
                  ('datetime', 0),
                  ('open', 1),
                  ('high', 2),
                  ('low', 3),
                  ('close', 4),
                  ('volume', 5),
                  ('openinterest', -1),
                  ('timeframe', bt.TimeFrame.Minutes)
              )
              
              1 Reply Last reply Reply Quote 1
              • 1 / 1
              • First post
                Last post
              Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors