IndexError in __getitem__ linebuffer (may be need to fix)

oalexandere

Hello.
One of my datafeed ended up at 2008-05-30:

2008-05-20,10.09,10.1,9.58,9.61,,1292600.0
2008-05-21,9.68,9.69,9.32,9.35,,709700.0
2008-05-22,9.35,9.67,9.28,9.47,,1738500.0
2008-05-23,9.45,9.45,9.23,9.29,,1455300.0
2008-05-27,9.34,9.57,9.3,9.46,,1822400.0
2008-05-28,9.46,9.57,9.22,9.38,,3348100.0
2008-05-29,9.21,9.67,9.2,9.55,,5303600.0
2008-05-30,9.6,9.6,9.33,9.33,,6776700.0

I added feed data to cerebro with parameter fromdate '2008-06-01':

                data = bte.feeds.PortfolioFeed(
                    dataname=os.path.join(path_name, file_name + '.csv'),
                    name=file_name,
                    plot=False,
                    fromdate=p.fromdate)
                cerebro.adddata(data)

After running got an error:

Traceback (most recent call last):
  File "C:/Users/-/-/Codes Python/Algo/Baskets/Baskets/_main.py", line 306, in <module>
    strats = cerebro.run()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\cerebro.py", line 1127, in run
sdf
    runstrat = self.runstrategies(iterstrat)
  File "C:\Users\-\-\Codes Python\_Frame\bt_external\cerebro_multi.py", line 146, in runstrategies
    self._runonce(runstrats)
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\cerebro.py", line 1695, in _runonce
    strat._oncepost(dt0)
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\strategy.py", line 293, in _oncepost
    self._next_analyzers(minperstatus, once=True)
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\strategy.py", line 366, in _next_analyzers
    analyzer._prenext()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzer.py", line 150, in _prenext
    child._prenext()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzer.py", line 152, in _prenext
    self.prenext()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzer.py", line 229, in prenext
    self.next()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzers\positions.py", line 78, in next
    pvals = [self.strategy.broker.get_value([d]) for d in self.datas]
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzers\positions.py", line 78, in <listcomp>
    pvals = [self.strategy.broker.get_value([d]) for d in self.datas]
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\brokers\bbroker.py", line 422, in get_value
    return self._get_value(datas=datas, lever=lever)
  File "C:\Users\-\YandexDisk\Codes Python\_Frame\bt_external\brokers\multibroker.py", line 90, in _get_value
    data.close[0])
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\linebuffer.py", line 165, in __getitem__
    return self.array[self.idx + ago]
IndexError: array index out of range

If to add checking for empty array in linebuffer.py all works correctly:

    def __getitem__(self, ago):
        if self.array:
            return self.array[self.idx + ago]

Is this the right solution? Maybe should fix this in backtrader?

backtrader

Let's summarize:

You feed wrong input and you get an error.
You attempt to access the wrong input at the wrong time and you get and error.

It seems obvious and sensible that wrong input and wrong access will produce an error.

@oalexandere said in IndexError in __getitem__ linebuffer (may be need to fix):

    self.prenext()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzer.py", line 229, in prenext
    self.next()
  File "C:\Users\-\envs\backtrader\lib\site-packages\backtrader\analyzers\positions.py", line 78, in next
    pvals = [self.strategy.broker.get_value([d]) for d in self.datas]

You are invoking next from prenext. In the latter there are no guarantees that neither the data feeds nor the indicators contain sensible values and can be accessed.

The approach (it is not a solution, because there is no error) is that if you want to access something from prenext, that you check the len of every data feed/indicator before attempting to access its values.

@oalexandere said in IndexError in __getitem__ linebuffer (may be need to fix):

(may be need to fix)

Yes, you need to fix your code and the input you provide to the platform.

See for example this entry in the community: https://community.backtrader.com/topic/461/date-issue-with-multiple-data/

oalexandere

@backtrader said in IndexError in __getitem__ linebuffer (may be need to fix):

that you check the len of every data feed/indicator before attempting to access its values.

Great, thanks, I understood.

@backtrader said in IndexError in __getitem__ linebuffer (may be need to fix):

you need to fix your code and the input you provide to the platform.

May be I wrong, but it seems impossible to fix the input to the platform, because it will be ineffectively every time checking up all .csv files (feeds) for availability of fromdate before added it to cerebro. I have more than 800 files of history, they all may be used in backtesting and they all updating periodically.

As a result one of possible aprroach for me will be to edit the file in framework backtrader\analyzers\positions.py for checking the len of the data:

    def next(self):
        pvals = [self.strategy.broker.get_value([d]) for d in self.datas if len(d)]