Error: deque index out of range when running optstrategy with memory saving
-
Dear community,
We are currently using Backtrader and so far it has been a great tool!!
Today we encountered a bug and after spending many hours on it, were not able to fix it ourselves. The error is as follows:
multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__ return self.runstrategies(iterstrat, predata=predata) File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext strat._next() File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 325, in _next super(Strategy, self)._next() File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\lineiterator.py", line 255, in _next clock_len = self._clk_update() File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in _clk_update for d in self.datas if len(d)) File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in <genexpr> for d in self.datas if len(d)) File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\linebuffer.py", line 165, in __getitem__ return self.array[self.idx + ago] IndexError: deque index out of range """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "Main.py", line 813, in <module> binance_run(res_dict) File "Main.py", line 772, in binance_run run_save_results(source, pair, res_dict) File "Main.py", line 722, in run_save_results run_strat(pair, source) File "Main.py", line 620, in run_strat call_all_EMA_combinations(list_train_test, dn1, dn2, dn3, source, pair, test_index) File "Main.py", line 527, in call_all_EMA_combinations res = trainer.run() File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1143, in run for r in pool.imap(self, iterstrats): File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 748, in next raise value IndexError: deque index out of range
After some debugging within linebuffer.py we found out the following:
Trying to get index, idx is 0 and ago is 0 while size is 1 Trying to get index, idx is 11 and ago is 0 while size is 12 Trying to get index, idx is -1 and ago is 0 while size is 0
So somehow the collections.deque has a size of 0 and the function getitem within linebuffer.py tries to access it with an index of -1.
Some other information that might be useful:
cerebro = bt.Cerebro(stdstats=False, runonce=False, preload=False, cheat_on_open=True, maxcpus=args.maxcpus, exactbars=args.save)
Where we use 1 as exactbars input and None for maxcpus. Also we are running optstrategy of Cerebro with three data streams (smallest time frames loaded earlier than larger time frames) and optimize a set of 5 different values for one parameter (even though this error appears as well when running optstrategy with only 1 specific value for the parameter).
Thank you in advance!
-
Yes, this is bound to happend because the data streams misalign. Disable
exactbars
. -
Thank you for the quick reply!
Is there another way to gain the same memory saving results as when exactbars = 1 when using multiple data streams within optstrategy? We are having out of memory exceptions when running the code for multiple hours.
-
No, there isn't.