For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Running multiple symbols data feeds in live IB



  • Hi,
    I'm trying to run a strategy for live IB trading that runs on several symbols 1 second bars.
    The issue I'm encountering is that sometimes when I try to run I get the following error:

    Traceback (most recent call last):
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\core\interactiveshell.py", line 2722, in safe_execfile
        self.compile if shell_futures else None)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\utils\py3compat.py", line 168, in execfile
        exec(compiler(f.read(), fname, 'exec'), glob, loc)
      File "C:\Work\Python\Backtrader\load_issue_script.py", line 55, in <module>
        cerebro.run()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
        self._runnext(runstrats)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
        strat._next()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 347, in _next
        super(Strategy, self)._next()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineiterator.py", line 260, in _next
        clock_len = self._clk_update()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in _clk_update
        newdlens = [len(d) for d in self.datas]
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in <listcomp>
        newdlens = [len(d) for d in self.datas]
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 464, in __len__
        return len(self.lines)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 220, in __len__
        return len(self.lines[0])
    ValueError: __len__() should return >= 0
    
    

    It seems that the code doesn't always wait for all the data feeds to have data before trying to run on them. Sometimes it works without a problem but a lot of time it crashes, especially the more data feeds you add.

    Can someone please tell me why this is happening? I wrote a very simple script that replicates the issue:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import backtrader as bt
    import backtrader.stores.ibstore as ibstore
    import pytz
    
    
    class TestStrategy(bt.Strategy):
    
        def _print_current_bar_price(self, data):
            txt = list()
            txt.append(data._name)
            txt.append('%04d' % len(data))
            txt.append('%s' % data.datetime.datetime(0))
            txt.append('{}'.format(data.open[0]))
            txt.append('{}'.format(data.high[0]))
            txt.append('{}'.format(data.low[0]))
            txt.append('{}'.format(data.close[0]))
            txt.append('{}'.format(data.volume[0]))
            print(', '.join(txt))
    
        def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, data._name, ' data is ', data._getstatusname(status), *args)
    
        def notify_store(self, msg, *args, **kwargs):
            print('*' * 5, 'STORE NOTIF:', msg)
    
        def next(self):
            # run on the symbols
            for i, d in enumerate(self.datas):
                self._print_current_bar_price(d)
    
    
    if __name__ == '__main__':
        symbols = ["AAPL", "BBBY", "BA", "NVDA"]
        port = 7497
    
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        # Add a strategy
        cerebro.addstrategy(TestStrategy)
    
        store = ibstore.IBStore(port=port)
        for s in symbols:
            data = store.getdata(dataname=s + '-STK-SMART-USD', tz=pytz.timezone('US/Eastern'), timeframe=bt.TimeFrame.Seconds, compression=1)
            cerebro.adddata(data, name=s)
        cerebro.broker = store.getbroker()  # connect IB broker
    
        # Run over everything
        cerebro.run()
    
    

    The full output for it is:

    TWS Time at connection:20200221 11:39:43 EST
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
    ***** STORE NOTIF: <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefnj>
    ***** AAPL  data is  DELAYED
    ***** BBBY  data is  DELAYED
    ***** BA  data is  DELAYED
    ***** NVDA  data is  DELAYED
    Traceback (most recent call last):
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\core\interactiveshell.py", line 2722, in safe_execfile
        self.compile if shell_futures else None)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\utils\py3compat.py", line 168, in execfile
        exec(compiler(f.read(), fname, 'exec'), glob, loc)
      File "C:\Work\Python\Backtrader\load_issue_script.py", line 52, in <module>
        cerebro.run()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
        self._runnext(runstrats)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
        strat._next()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 347, in _next
        super(Strategy, self)._next()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineiterator.py", line 260, in _next
        clock_len = self._clk_update()
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in _clk_update
        newdlens = [len(d) for d in self.datas]
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in <listcomp>
        newdlens = [len(d) for d in self.datas]
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 464, in __len__
        return len(self.lines)
      File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 220, in __len__
        return len(self.lines[0])
    ValueError: __len__() should return >= 0
    
    

    Thank you



  • Please take a look at the following post: ERROR: len() should return >= 0 on liveFeed

    I've got the same error while trading live with IB and the fix provided in the above post solved the problem.



  • You may take a patch from my fork here.



  • @vladisld Thank you so much for your reply, I tried the fix and it indeed seems to solve the problem.

    I did notice that once you try to use more then three data feeds it can take a long time (minutes) for Backtrader to start running. notify_data sends message that Delayed data is used, but it takes minutes until the data actually start to arrive, or it never arrives at all.

    TWS Time at connection:20200224 12:27:20 EST
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for CAMP is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for CAMP is issued.
    ReqData: reqMktData for DL is issued.
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
    ***** STORE NOTIF: <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefnj>
    ***** BA  data is  DELAYED
    ***** BBBY  data is  DELAYED
    ***** CAMP  data is  DELAYED
    ***** DL  data is  DELAYED
    
    

    Any ideas what could be causing it?

    Thanks again!



  • Few notes:

    1. reqMktData is issued multiple times for each symbol:
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for CAMP is issued.
    ReqData: reqMktData for BA is issued.
    ReqData: reqMktData for BBBY is issued.
    ReqData: reqMktData for CAMP is issued.
    ReqData: reqMktData for DL is issued.
    

    It means that if more than certain number of symbold are used, it may easily cause the pace violation for market data requests for the same symbol within 15 seconds.
    See here for the full list of IB pace limitations.

    This may explain the case were no data will be sent back to you for some symbols after the subscription.
    Please see the post here.
    There is also a fix for this issue in my fork in 'lazytrader' branch.

    1. By default IBStore.getdata will set the backfill_start and backfill parameters to True.

    It means that historical data will be requested on start to "backfill" some amount of data automatically. This may explain the delays.

    If only a live data is needed, both the backfill_start and backfill parameters could be set to False. For example:

            datakwargs = dict(
                timeframe=cfg['timeframe'],
                compression=cfg['compression'],
                calendar=cfg['calendar'],
                tzinput=cfg['tzinput'],
                historical=False,
                fromdate=None,
                rtbar=cfg['rtbar'],
                qcheck=args.qcheck,
                what=None,
                backfill_start=False,
                backfill=False,
                latethrough=False
            )
            data = bt.stores.IBStore.getdata(dataname=cfg['dataname'], **datakwargs)
    


  • @vladisld Thank you again for your great help. I applied your fix and it indeed solved the problem.
    The other delay I've encountered is simply because it looks like Backtrader waits until it receives new data from all the data feeds before starting to run.

    Backfill doesn't give more delay (besides the time it takes to load the historical data), what's weird is that it seems like Backtrader still waits until it receives new data from all the data feeds, and only then starts to load the backfill data.



  • Few more things that my help or give a clue:

    1. According to your strategy code, only the next notification is processed. The next notification will be received only when each data will receive at least a single bar ( in other words all datas are synchronized). Please pay attention that I'm only talking about independent datas, not indicators ( in which case the warm-up period play its role) . If you want to process each data independently, you may want to use the 'prenext' notification as well.

    2. The backfill will only start after IB DataFeed will try to load at least the first bar (which may fail of case). Usually it will wait at least qcheck seconds ( by default it is 0.5 seconds, but it could be less if the bars from the previous data was already fetched.) before giving up.



  • @vladisld said in Running multiple symbols data feeds in live IB:

    The next notification will be received only when each data will receive at least a single bar

    You were right, it did present a problem, thanks for the heads up.

    What happens is that with the default qcheck at 0.5 'next' waits 0.5 a second before releasing the data he currently have if all the datas didn't update, then when it all the datas arrive the queue is released at once. The weird thing is that when I set qcheck = 0 I still got some delay that increased over time. So I set qcheck = 0.001 which does synchronizes the datas when they are all available, which seems to be the best solution.

    When using prenext I still get the 0.5 data delay until all data data start arriving, so there's probably a parameter I'm missing.

    For anyone wondering in the future, the corrected code is:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import backtrader as bt
    import backtrader.stores.ibstore as ibstore
    import datetime
    import pytz
    
    
    class TestStrategy(bt.Strategy):
    
        def _print_current_bar_price(self, data):
            txt = list()
            txt.append(data._name)
            txt.append('%04d' % len(data))
            txt.append('%s' % datetime.datetime.now().time())
            txt.append('%s' % data.datetime.datetime(0))
            txt.append('{}'.format(data.open[0]))
            txt.append('{}'.format(data.high[0]))
            txt.append('{}'.format(data.low[0]))
            txt.append('{}'.format(data.close[0]))
            txt.append('{}'.format(data.volume[0]))
            print(', '.join(txt))
    
        def __init__(self):
            self._count = [0] * len(self.datas)
    
        def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, data._name, ' data is ', data._getstatusname(status), *args)
    
        def notify_store(self, msg, *args, **kwargs):
            print('*' * 5, 'STORE NOTIF:', msg)
    
        def prenext(self):
            # call next() even when data is not available for all tickers
            self.next()
    
        def next(self):
            # run on the symbols
            for i, d in enumerate(self.datas):
                if len(d) > self._count[i]:
                    self._count[i] = len(d)
                    self._print_current_bar_price(d)
    
    
    if __name__ == '__main__':
        symbols = ["AAPL", "BA", "BBBY", "NVDA"]
        port = 7497
    
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        # Add a strategy
        cerebro.addstrategy(TestStrategy)
    
        store = ibstore.IBStore(port=port)
        for s in symbols:
            data = store.getdata(dataname=s + '-STK-SMART-USD', tz=pytz.timezone('US/Eastern'), timeframe=bt.TimeFrame.Seconds, compression=1, qcheck=0.001, backfill_start=False)
            cerebro.adddata(data, name=s)
        cerebro.broker = store.getbroker()  # connect IB broker
    
        # Run over everything
        cerebro.run()
    


  • @orenshkol said in Running multiple symbols data feeds in live IB:

    So I set qcheck = 0.001 which does synchronizes the datas when they are all available, which seems to be the best solution

    The problem (or back side) of such qcheck setting is a high CPU load during live trading. See my previous post about it (https://community.backtrader.com/topic/2346/high-cpu-load-during-pre-market-live-trading-and-adaptive-qcheck-logic).



  • @orenshkol @vladisld why when I am using with resample data instead adddata, strategy runs on the first symbol only?


Log in to reply
 

});