Exception in line series length at paper trading

J. Javier Gálvez

Hi, Backtraders!

I'm trying to run backtrader with Interactive Broker where I fed data for 10 equities. But I get an error related to the length of some of the data lines at the beginning of the simulation. When I reduce the number of equities to three all run ok. I searched on this community and found a related problem but any
solution is yet proposed.

Here the full exception trace:

Traceback (most recent call last):
  File "/home/galvez/repos/research_factory/20180822_PortafolioBacktrader/XBranch_NN/backTesting/run_backtest.py", line 335, in <module>
    run_live(equity)
  File "/home/galvez/repos/research_factory/20180822_PortafolioBacktrader/XBranch_NN/backTesting/run_backtest.py", line 325, in run_live
    cerebro.run()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1630, in _runnext
    strat._next()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 325, in _next
    super(Strategy, self)._next()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineiterator.py", line 255, in _next
    clock_len = self._clk_update()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 305, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 305, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineseries.py", line 464, in __len__
    return len(self.lines)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineseries.py", line 220, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

Process finished with exit code 1

Thanks in advance and happy trading.

backtrader

You are provoking pacing violations, due to the number of requests. That's why reducing the number of assets gets you up and running.

J. Javier Gálvez

Thank you for your prompt response. Any clue how to overcome the issue? Or is there a limited number of assets we can work with Interactive Brokers and Backtrader?

backtrader

@j-javier-gálvez said in Exception in line series length at paper trading:

Or is there a limited number of assets we can work with Interactive Brokers

The limit is there. Even if some of the limits have been softened over the years, IB praises to be a broker and not a data provider. That's why they also provide 250 ms snapshot-ticks of the market instead of real time ticks.

Using IB is of course fit for many purposes and trading situations, but working simultaneously with a large number of assets isn't one of them.

Your options:

Disable backfilling
Use data files (or any other source) as backfill_from

Because the pacing violations are hit when requesting historical data (which is the backfilling source)

There is an additional limit which is the number of simultanteous assets you can request real-time information from (NBBO), but this also has to do with your account commissions level, if you have acquired a boost package and other things.

Let me add: if you didn't know these things before ... you should know all the rules that apply to your trading (data provision and brokerage) before you trade. It is the 1st rule when gambling (sorry trading): "Know the house rules".

J. Javier Gálvez

@backtrader Thank you very much for your response and suggestions. I appreciate it.

J. Javier Gálvez

Hi, @backtrader and Bactraders!

I tried the second option but it didn't go as expected. I read the documentation and review every post about the backfill_from and tried several things from there without success.

The issue is that the timestamp I get from the data when the next() is called jump from 12:00 to 5:00 of the other day instead of continue to 12:01 and so on until 16:00 and then 9:30 of the next day. Here an extract of my prints:

2018-10-01T11:56:00
2018-10-01T11:57:00
2018-10-01T11:58:00
2018-10-01T11:59:00
2018-10-01T12:00:00
2018-10-02T05:31:00
2018-10-02T05:32:00
2018-10-02T05:33:00
2018-10-02T05:34:00
2018-10-02T05:35:00
2018-10-02T05:36:00
2018-10-02T05:37:00

I'm very sure that my local CSVs have data only between 9:30 and 16:00 hrs.

Let me detail about my code. This is how I'm connecting to IB:

cerebro = bt.Cerebro()
store = bt.stores.IBStore(host='127.0.0.1',port=4002, clientId=36, reconnect=-1)

My data is in minute bars and I'm reading it as follow:

data0 = btfeed.GenericCSVData(
        dataname=file_path,

        fromdate=datetime.datetime(2018, 10, 1, 9, 30, 0),
        todate=datetime.datetime(2018, 12, 12, 9, 30, 0),

        timeframe=bt.TimeFrame.Minutes,
        compression=1,

        dtformat='%Y%m%d',
        tmformat='%H:%M:%S',

        datetime=0,
        time=1,
        open=3,
        high=4,
        low=5,
        close=6,
        volume=8,
        openinterest=-1,
)

Then, adding the data:

contract = 'TWTR-STK-SMART-USD'
data = store.getdata(dataname=contract, rtbar=True,
                             timeframe=bt.TimeFrame.Ticks, historical=False,
                             #todate=datetime.datetime.now(),
                             fromdate=datetime.datetime(2018, 10, 1, 9, 30, 0),
                             tz=pytz.timezone('US/Eastern'),
                             backfill_from=data0,
                             #sessionstart=datetime.time(9, 30, 0),
                             #sessionend=datetime.time(16, 0, 0),
                             )

cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1)

Finally, I run cerebro as:

cerebro.broker = store.getbroker()

cerebro.addstrategy(TestStrategy)

cerebro.run(exactbars=1, runonce=False)

I have tried several combinations of parameters both, in store.getdata() and in btfeed.GenericCSVData() but none of them work. So, any help will be appreciated.

Thanks in advance.