For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Error: deque index out of range when running optstrategy with memory saving



  • Dear community,

    We are currently using Backtrader and so far it has been a great tool!!

    Today we encountered a bug and after spending many hours on it, were not able to fix it ourselves. The error is as follows:

    multiprocessing.pool.RemoteTraceback:
    """
    Traceback (most recent call last):
      File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__
        return self.runstrategies(iterstrat, predata=predata)
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
        self._runnext(runstrats)
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
        strat._next()
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 325, in _next
        super(Strategy, self)._next()
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\lineiterator.py", line 255, in _next
        clock_len = self._clk_update()
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in _clk_update
        for d in self.datas if len(d))
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\strategy.py", line 310, in <genexpr>
        for d in self.datas if len(d))
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\linebuffer.py", line 165, in __getitem__
        return self.array[self.idx + ago]
    IndexError: deque index out of range
    """
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "Main.py", line 813, in <module>
        binance_run(res_dict)
      File "Main.py", line 772, in binance_run
        run_save_results(source, pair, res_dict)
      File "Main.py", line 722, in run_save_results
        run_strat(pair, source)
      File "Main.py", line 620, in run_strat
        call_all_EMA_combinations(list_train_test, dn1, dn2, dn3, source, pair, test_index)
      File "Main.py", line 527, in call_all_EMA_combinations
        res = trainer.run()
      File "C:\Users\Boning\Documents\Agga\venv\lib\site-packages\backtrader\cerebro.py", line 1143, in run
        for r in pool.imap(self, iterstrats):
      File "c:\users\boning\appdata\local\programs\python\python37\Lib\multiprocessing\pool.py", line 748, in next
        raise value
    IndexError: deque index out of range
    

    After some debugging within linebuffer.py we found out the following:

    Trying to get index, idx is 0 and ago is 0 while size is 1
    Trying to get index, idx is 11 and ago is 0 while size is 12
    Trying to get index, idx is -1 and ago is 0 while size is 0
    

    So somehow the collections.deque has a size of 0 and the function getitem within linebuffer.py tries to access it with an index of -1.

    Some other information that might be useful:

     cerebro = bt.Cerebro(stdstats=False,
                                runonce=False,
                                preload=False,
                                cheat_on_open=True,
                                maxcpus=args.maxcpus,
                                exactbars=args.save)
    

    Where we use 1 as exactbars input and None for maxcpus. Also we are running optstrategy of Cerebro with three data streams (smallest time frames loaded earlier than larger time frames) and optimize a set of 5 different values for one parameter (even though this error appears as well when running optstrategy with only 1 specific value for the parameter).

    Thank you in advance!


  • administrators

    Yes, this is bound to happend because the data streams misalign. Disable exactbars.



  • Thank you for the quick reply!

    Is there another way to gain the same memory saving results as when exactbars = 1 when using multiple data streams within optstrategy? We are having out of memory exceptions when running the code for multiple hours.


  • administrators

    No, there isn't.


Log in to reply
 

});