Live trading data(IB) issue

cynthiameow

I added multiple symbols as IB data feeds but I got the following error during running time:

Traceback (most recent call last):
  File "E:/test/ib_test.py", line 64, in <module>
    run()
  File "E:/test/ib_test.py", line 60, in run
    cerebro.run()
  File "C:\Python38\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Python38\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "C:\Python38\lib\site-packages\backtrader\cerebro.py", line 1557, in _runnext
    dt0 = min((d for i, d in enumerate(dts)
ValueError: min() arg is an empty sequence

the way I did to add data feeds:

        data = store.getdata(dataname=symbol, timeframe=bt.TimeFrame.Seconds, compression=1, historical=False)
        cerebro.resampledata(data, name=ticker, timeframe=bt.TimeFrame.Seconds, compression=5)
        cerebro.replaydata(data, name='{}_5min'.format(ticker), timeframe=bt.TimeFrame.Minutes, compression=5)

If I don't do the resampling step, other weid things may happen, like the latest 5min bar will vanish for some tickers temporarily:

2020-09-16 11:32:57.385073
SPY_5min
18:30:00 342.34 342.74 342.21 342.42
18:25:00 342.3 342.42 342.11 342.34
18:20:00 342.28 342.43 342.22 342.29
18:15:00 341.89 342.67 341.84 342.28
TSLA_5min
18:32:55.565999 449.91 449.91 448.79 449.08
18:30:00 450.14 451.2 448.23 448.82
18:25:00 449.24 450.74 448.57 449.86
18:20:00 451.68 453.0 448.72 449.37
FB_5min
18:30:00 266.35 266.75 265.81 265.99
18:25:00 265.98 266.38 265.35 266.38
18:20:00 266.1 266.51 265.84 266.04
18:15:00 265.94 266.99 265.82 266.13


2020-09-16 11:32:57.385073
SPY_5min
18:32:55.817999 342.38 342.42 340.25 342.37
18:30:00 342.34 342.74 342.21 342.42
18:25:00 342.3 342.42 342.11 342.34
18:20:00 342.28 342.43 342.22 342.29
TSLA_5min
18:32:55.817999 449.91 449.91 448.79 449.07
18:30:00 450.14 451.2 448.23 448.82
18:25:00 449.24 450.74 448.57 449.86
18:20:00 451.68 453.0 448.72 449.37
FB_5min
18:32:55.817999 265.95 266.02 265.93 265.93
18:30:00 266.35 266.75 265.81 265.99
18:25:00 265.98 266.38 265.35 266.38
18:20:00 266.1 266.51 265.84 266.04


2020-09-16 11:32:58.130299
SPY_5min
18:30:00 342.34 342.74 342.21 342.42
18:25:00 342.3 342.42 342.11 342.34
18:20:00 342.28 342.43 342.22 342.29
18:15:00 341.89 342.67 341.84 342.28
TSLA_5min
18:32:55.817999 449.91 449.91 448.79 449.07
18:30:00 450.14 451.2 448.23 448.82
18:25:00 449.24 450.74 448.57 449.86
18:20:00 451.68 453.0 448.72 449.37
FB_5min
18:32:56.067998 265.95 266.02 265.9 265.9
18:30:00 266.35 266.75 265.81 265.99
18:25:00 265.98 266.38 265.35 266.38
18:20:00 266.1 266.51 265.84 266.04

in this example, the latest evolving 5min bar of SPY shows up and disappears during updates of other tickers. anyone knows if this is the expected behavior of replaydata?

In summary, either using resampledata or not with replaydata, I cannot proceed with stable live data feeds.

dasch

i experience the same issue using oanda in live mode, but i did not have the possibility to create a reporducable example and did not investigate that issue much. I collect some info about this issue here: https://github.com/backtrader2/backtrader/issues/32

Maybe you have some code which reproduces this issue?

This does not help you, i suppose.

I added a workaround in backtrader which may help you.

I changed the block to the following code, which works for me, but is a ugly solution:

                # Get index to minimum datetime
                if onlyresample or noresample:
                    dt0 = min((d for d in dts if d is not None))
                else:
                    #dt0 = min((d for i, d in enumerate(dts)
                    #           if d is not None and i not in rsonly))
                    dtstmp = {d for i, d in enumerate(dts)
                              if d is not None and i not in rsonly}
                    if len(dtstmp) == 0:
                        dtstmp = {d for i, d in enumerate(dts)
                                  if d is not None}
                    dt0 = min(dtstmp)

cynthiameow

Thanks a lot for your sharing! The code showed above equipped with an empty strategy which printing latest bars will reproduce my issue.
I will also take a look on the cerebro code to investigate the problem.

dasch

as i remember, @vladisld did experience this issue, too. maybe he has some insights into this? sorry, if i am wrong about this @vladisld

cynthiameow

@dasch actually i suspect two issues i got (replay with/without resample) may share the same problem:
all replay somehow lost all latest updates (the evolving bars) (second issue i showed)
then the following code

dt0 = min((d for i, d in enumerate(dts)
                  if d is not None and i not in rsonly))

became problematic since all replay feeds dont have updates datetimes which gave empty sequence in the min function

This may also be related to this issue:
https://community.backtrader.com/topic/1367/error-len-should-return-0-on-livefeed/4
I already applied @vladisld's patch to avoid the qcheck issue.