Error: deque index out of range when running optstrategy with memory saving

Boning Gong

Dear community,

We are currently using Backtrader and so far it has been a great tool!!

Today we encountered a bug and after spending many hours on it, were not able to fix it ourselves. The error is as follows:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__
    return self.runstrategies(iterstrat, predata=predata)
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
    strat._next()
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 325, in _next
    super(Strategy, self)._next()
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\lineiterator.py", line 255, in _next
    clock_len = self._clk_update()
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in _clk_update
    for d in self.datas if len(d))
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in <genexpr>
    for d in self.datas if len(d))
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\linebuffer.py", line 165, in __getitem__
    return self.array[self.idx + ago]
IndexError: deque index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "Main.py", line 813, in <module>
    binance_run(res_dict)
  File "Main.py", line 772, in binance_run
    run_save_results(source, pair, res_dict)
  File "Main.py", line 722, in run_save_results
    run_strat(pair, source)
  File "Main.py", line 620, in run_strat
    call_all_EMA_combinations(list_train_test, dn1, dn2, dn3, source, pair, test_index)
  File "Main.py", line 527, in call_all_EMA_combinations
    res = trainer.run()
  File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1143, in run
    for r in pool.imap(self, iterstrats):
  File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 748, in next
    raise value
IndexError: deque index out of range

After some debugging within linebuffer.py we found out the following:

Trying to get index, idx is 0 and ago is 0 while size is 1
Trying to get index, idx is 11 and ago is 0 while size is 12
Trying to get index, idx is -1 and ago is 0 while size is 0

So somehow the collections.deque has a size of 0 and the function getitem within linebuffer.py tries to access it with an index of -1.

Some other information that might be useful:

 cerebro = bt.Cerebro(stdstats=False,
                            runonce=False,
                            preload=False,
                            cheat_on_open=True,
                            maxcpus=args.maxcpus,
                            exactbars=args.save)

Where we use 1 as exactbars input and None for maxcpus. Also we are running optstrategy of Cerebro with three data streams (smallest time frames loaded earlier than larger time frames) and optimize a set of 5 different values for one parameter (even though this error appears as well when running optstrategy with only 1 specific value for the parameter).

Thank you in advance!

backtrader

Yes, this is bound to happend because the data streams misalign. Disable exactbars.

Boning Gong

Thank you for the quick reply!

Is there another way to gain the same memory saving results as when exactbars = 1 when using multiple data streams within optstrategy? We are having out of memory exceptions when running the code for multiple hours.

backtrader

No, there isn't.