ERROR: __len__() should return >= 0 on liveFeed

LAbrk

Hello everyone. I need help

I am working on implementing a new data feed / live broker. I have taken Oanda as a model since I have similar methods to interact.

I receive the MarkerData in real time when subscribing to a Web Service (for each Data separately). Which is similar to Oanda's price streamer.
I receive the Excecution Reports of the transactions by WebSocket. Similar to the Oanda events Streamer.
I can execute actions on orders by REST. Also similar to what OandaBroker uses

Finally the strategies are only in Live mode, they do not use anything historical.

I adapted the broker, store and feed of oanda according to the new destination, but the architecture remained the same. The only thing I had to add is the TICK granularity:

#supported granularities
     _GRANULARITIES = {
         (bt.TimeFrame.Ticks, 1): 'T1',
     }

And everything works well with only 1 Contract, but when I use more than one I get the following error:

Traceback (most recent call last):
  File "/proyect/venv/code/test2.py", line 110, in <module>
    cerebro.run()
  File "/proyect/venv/lib/py/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/proyect/venv/lib/py/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/proyect/venv/lib/py/site-packages/backtrader/cerebro.py", line 1631, in _runnext
    strat._next()
  File "/proyect/venv/lib/py/site-packages/backtrader/strategy.py", line 325, in _next
    super(Strategy, self)._next()
  File "/proyect/venv/lib/py/site-packages/backtrader/lineiterator.py", line 255, in _next
    clock_len = self._clk_update()
  File "/proyect/venv/lib/py/site-packages/backtrader/strategy.py", line 305, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "/proyect/venv/lib/py/site-packages/backtrader/strategy.py", line 305, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "/proyect/venv/lib/py/site-packages/backtrader/lineseries.py", line 464, in __len__
    return len(self.lines)
  File "/proyect/venv/lib/py/site-packages/backtrader/lineseries.py", line 220, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

This is the strategy that I am running and the parameters with which I add the Datas:

import argparse
import datetime
import backtrader as bt



# Create a Stratey
class TestStrategy(bt.Strategy):

    def __init__(self):
        pass

    #Provides any noticficaitons about the data.
    def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)

    def notify_store(self, msg, *args, **kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    def notify_order(self, order):
        pass

    def notify_trade(self, trade):
        pass


    def next(self):

        # Simply log the closing price of the series from the reference
        print('O: {} H: {} L: {} C: {}'.format(
            self.data.open[0],
            self.data.high[0],
            self.data.low[0],
            self.data.close[0],
            ))
        if data1 is not None:
            print('NEXT 1 ==> O: {} H: {} L: {} C: {}'.format(
                self.data1.open[0],
                self.data1.high[0],
                self.data1.low[0],
                self.data1.close[0],
            ))


    def start(self):
        if self.data0.contractdetails is not None:
            print('-- Contract Details:')
            print(self.data0.contractdetails)
        print('Started')
        acc_cash = cerebro.broker.getcash()
        acc_val = cerebro.broker.getvalue()
        print('Account Cash = {}'.format(acc_cash))
        print('Account Value = {}'.format(acc_val))


if __name__ == '__main__':
    cerebro = bt.Cerebro()
    testreststore = bt.stores.TestrestStore(token=apikey, account=acc, practice=True)
    cerebro.broker = testreststore.getbroker()
    data0 = testreststore.getdata(dataname="contract1", timeframe=bt.TimeFrame.Ticks, compression=1, backfill_start=False, backfill=False)
    data1 = testreststore.getdata(dataname="contract2", timeframe=bt.TimeFrame.Ticks, compression=1,
                                 backfill_start=False, backfill=False)

    cerebro.adddata(data0)
    cerebro.adddata(data1)
    cerebro.addstrategy(TestStrategy)
    cerebro.run()

Thank you very much, I hope I can return the help soon

LAbrk

I was doing some more tests and it seems that the error occurs when 2 markets datas (one from Data0 and one from Data1) arrive very close in time (almost together).

@backtrader, what would be the configuration of parameters to be able to take MarketData in real time? Is that what is failing me?

Thank !

backtrader

@labrk said in ERROR: __len__() should return >= 0 on liveFeed:

what would be the configuration of parameters to be able to take MarketData in real time?

I don't know which parameters you mean. And without seeing the code it is impossible to really ascertain what happens.

The only long shot is not that things are coming together, but that two things (former and latter) come with timestamps that make no sense (the latter has a timestamp smaller than the former) which makes the system thing it has to rewind one of the data feeds, thus taking you back to -1.

Zachary Harris

I've seen this same error with essentially the same stack trace. It can happen when Cerebro._runnext loads a bar which arrives in the narrow window of time between the first check for new data:

drets.append(d.next(ticks=False))

and the second chance:

if d.next(datamaster=dmaster, ticks=False)

If the new bar's timestamp shows it is from the "future" with respect to dmaster then it is loaded once (in d.next()) but rewound twice: once within d.next() itself and again in _runnext a few lines after the above block:

di.rewind()

My guess is that the needed fix is in AbstractDataBase.next():

if self.lines.datetime[0] > datamaster.lines.datetime[0]:
    # can't deliver new bar, too early, go back
    self.rewind()
    return False # <-- This needs to be added since bar is not delivered

However, I'm relatively new to the code, so far all I know it could be intentional that AbstractDataBase.next() returns True to "tell the world there is a bar" even though it is too early to actually deliver the bar. In that case it would seem the logic in _runnext needs to be modified so that a second rewind does not occur.

vladisld

Thanks a lot @Zachary-Harris

I've got the same error during live trading through IB (see my post here: https://community.backtrader.com/topic/642/iqfeed-store-and-feed-help-needed/12 ) and your fix seems to be working great.

This seems to be a pretty serious issue that should be fixed in the official source.

I just wonder why there are so little reports about it - it should have happen pretty often to everyone using backtrader for live trading.

Alexandre Durand

Thanks a lot @Zachary-Harris, very useful !

cynthiameow

@Zachary-Harris Thanks a lot for your fix! It indeed solved the same issue i was experiencing.
@vladisld @backtrader may we check in this change to master release?