Backtrader Community
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Running multiple symbols data feeds in live IB

    General Code/Help
    3
    10
    1663
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • orenshkol
      orenshkol last edited by

      Hi,
      I'm trying to run a strategy for live IB trading that runs on several symbols 1 second bars.
      The issue I'm encountering is that sometimes when I try to run I get the following error:

      Traceback (most recent call last):
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\core\interactiveshell.py", line 2722, in safe_execfile
    self.compile if shell_futures else None)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\utils\py3compat.py", line 168, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "C:\Work\Python\Backtrader\load_issue_script.py", line 55, in <module>
    cerebro.run()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
    strat._next()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 347, in _next
    super(Strategy, self)._next()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineiterator.py", line 260, in _next
    clock_len = self._clk_update()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 464, in __len__
    return len(self.lines)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 220, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

      It seems that the code doesn't always wait for all the data feeds to have data before trying to run on them. Sometimes it works without a problem but a lot of time it crashes, especially the more data feeds you add.

      Can someone please tell me why this is happening? I wrote a very simple script that replicates the issue:

      from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import backtrader as bt
import backtrader.stores.ibstore as ibstore
import pytz


class TestStrategy(bt.Strategy):

    def _print_current_bar_price(self, data):
        txt = list()
        txt.append(data._name)
        txt.append('%04d' % len(data))
        txt.append('%s' % data.datetime.datetime(0))
        txt.append('{}'.format(data.open[0]))
        txt.append('{}'.format(data.high[0]))
        txt.append('{}'.format(data.low[0]))
        txt.append('{}'.format(data.close[0]))
        txt.append('{}'.format(data.volume[0]))
        print(', '.join(txt))

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, data._name, ' data is ', data._getstatusname(status), *args)

    def notify_store(self, msg, *args, **kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    def next(self):
        # run on the symbols
        for i, d in enumerate(self.datas):
            self._print_current_bar_price(d)


if __name__ == '__main__':
    symbols = ["AAPL", "BBBY", "BA", "NVDA"]
    port = 7497

    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    store = ibstore.IBStore(port=port)
    for s in symbols:
        data = store.getdata(dataname=s + '-STK-SMART-USD', tz=pytz.timezone('US/Eastern'), timeframe=bt.TimeFrame.Seconds, compression=1)
        cerebro.adddata(data, name=s)
    cerebro.broker = store.getbroker()  # connect IB broker

    # Run over everything
    cerebro.run()

      The full output for it is:

      TWS Time at connection:20200221 11:39:43 EST
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** STORE NOTIF: <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefnj>
***** AAPL  data is  DELAYED
***** BBBY  data is  DELAYED
***** BA  data is  DELAYED
***** NVDA  data is  DELAYED
Traceback (most recent call last):
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\core\interactiveshell.py", line 2722, in safe_execfile
    self.compile if shell_futures else None)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\IPython\utils\py3compat.py", line 168, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "C:\Work\Python\Backtrader\load_issue_script.py", line 52, in <module>
    cerebro.run()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
    strat._next()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 347, in _next
    super(Strategy, self)._next()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineiterator.py", line 260, in _next
    clock_len = self._clk_update()
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\strategy.py", line 327, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 464, in __len__
    return len(self.lines)
  File "C:\Work\Python\Backtrader\venv\lib\site-packages\backtrader\lineseries.py", line 220, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

      Thank you

      1 Reply Last reply Reply Quote 0
      • vladisld
        vladisld last edited by

        Please take a look at the following post: ERROR: len() should return >= 0 on liveFeed

        I've got the same error while trading live with IB and the fix provided in the above post solved the problem.

        1 Reply Last reply Reply Quote 0
        • vladisld
          vladisld last edited by

          You may take a patch from my fork here.

          orenshkol 1 Reply Last reply Reply Quote 0
          • orenshkol
            orenshkol @vladisld last edited by

            @vladisld Thank you so much for your reply, I tried the fix and it indeed seems to solve the problem.

            I did notice that once you try to use more then three data feeds it can take a long time (minutes) for Backtrader to start running. notify_data sends message that Delayed data is used, but it takes minutes until the data actually start to arrive, or it never arrives at all.

            TWS Time at connection:20200224 12:27:20 EST
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for CAMP is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for CAMP is issued.
ReqData: reqMktData for DL is issued.
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** STORE NOTIF: <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefnj>
***** BA  data is  DELAYED
***** BBBY  data is  DELAYED
***** CAMP  data is  DELAYED
***** DL  data is  DELAYED

            Any ideas what could be causing it?

            Thanks again!

            1 Reply Last reply Reply Quote 0
            • vladisld
              vladisld last edited by

              Few notes:

              1. reqMktData is issued multiple times for each symbol:
              ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for CAMP is issued.
ReqData: reqMktData for BA is issued.
ReqData: reqMktData for BBBY is issued.
ReqData: reqMktData for CAMP is issued.
ReqData: reqMktData for DL is issued.

              It means that if more than certain number of symbold are used, it may easily cause the pace violation for market data requests for the same symbol within 15 seconds.
              See here for the full list of IB pace limitations.

              This may explain the case were no data will be sent back to you for some symbols after the subscription.
              Please see the post here.
              There is also a fix for this issue in my fork in 'lazytrader' branch.

              1. By default IBStore.getdata will set the backfill_start and backfill parameters to True.

              It means that historical data will be requested on start to "backfill" some amount of data automatically. This may explain the delays.

              If only a live data is needed, both the backfill_start and backfill parameters could be set to False. For example:

                      datakwargs = dict(
            timeframe=cfg['timeframe'],
            compression=cfg['compression'],
            calendar=cfg['calendar'],
            tzinput=cfg['tzinput'],
            historical=False,
            fromdate=None,
            rtbar=cfg['rtbar'],
            qcheck=args.qcheck,
            what=None,
            backfill_start=False,
            backfill=False,
            latethrough=False
        )
        data = bt.stores.IBStore.getdata(dataname=cfg['dataname'], **datakwargs)
              orenshkol 1 Reply Last reply Reply Quote 0
              • orenshkol
                orenshkol @vladisld last edited by

                @vladisld Thank you again for your great help. I applied your fix and it indeed solved the problem.
                The other delay I've encountered is simply because it looks like Backtrader waits until it receives new data from all the data feeds before starting to run.

                Backfill doesn't give more delay (besides the time it takes to load the historical data), what's weird is that it seems like Backtrader still waits until it receives new data from all the data feeds, and only then starts to load the backfill data.

                1 Reply Last reply Reply Quote 0
                • vladisld
                  vladisld last edited by

                  Few more things that my help or give a clue:

                  1. According to your strategy code, only the next notification is processed. The next notification will be received only when each data will receive at least a single bar ( in other words all datas are synchronized). Please pay attention that I'm only talking about independent datas, not indicators ( in which case the warm-up period play its role) . If you want to process each data independently, you may want to use the 'prenext' notification as well.

                  2. The backfill will only start after IB DataFeed will try to load at least the first bar (which may fail of case). Usually it will wait at least qcheck seconds ( by default it is 0.5 seconds, but it could be less if the bars from the previous data was already fetched.) before giving up.

                  orenshkol 1 Reply Last reply Reply Quote 0
                  • orenshkol
                    orenshkol @vladisld last edited by

                    @vladisld said in Running multiple symbols data feeds in live IB:

                    The next notification will be received only when each data will receive at least a single bar

                    You were right, it did present a problem, thanks for the heads up.

                    What happens is that with the default qcheck at 0.5 'next' waits 0.5 a second before releasing the data he currently have if all the datas didn't update, then when it all the datas arrive the queue is released at once. The weird thing is that when I set qcheck = 0 I still got some delay that increased over time. So I set qcheck = 0.001 which does synchronizes the datas when they are all available, which seems to be the best solution.

                    When using prenext I still get the 0.5 data delay until all data data start arriving, so there's probably a parameter I'm missing.

                    For anyone wondering in the future, the corrected code is:

                    from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import pytz


class TestStrategy(bt.Strategy):

    def _print_current_bar_price(self, data):
        txt = list()
        txt.append(data._name)
        txt.append('%04d' % len(data))
        txt.append('%s' % datetime.datetime.now().time())
        txt.append('%s' % data.datetime.datetime(0))
        txt.append('{}'.format(data.open[0]))
        txt.append('{}'.format(data.high[0]))
        txt.append('{}'.format(data.low[0]))
        txt.append('{}'.format(data.close[0]))
        txt.append('{}'.format(data.volume[0]))
        print(', '.join(txt))

    def __init__(self):
        self._count = [0] * len(self.datas)

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, data._name, ' data is ', data._getstatusname(status), *args)

    def notify_store(self, msg, *args, **kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    def prenext(self):
        # call next() even when data is not available for all tickers
        self.next()

    def next(self):
        # run on the symbols
        for i, d in enumerate(self.datas):
            if len(d) > self._count[i]:
                self._count[i] = len(d)
                self._print_current_bar_price(d)


if __name__ == '__main__':
    symbols = ["AAPL", "BA", "BBBY", "NVDA"]
    port = 7497

    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    store = ibstore.IBStore(port=port)
    for s in symbols:
        data = store.getdata(dataname=s + '-STK-SMART-USD', tz=pytz.timezone('US/Eastern'), timeframe=bt.TimeFrame.Seconds, compression=1, qcheck=0.001, backfill_start=False)
        cerebro.adddata(data, name=s)
    cerebro.broker = store.getbroker()  # connect IB broker

    # Run over everything
    cerebro.run()
                    vladisld 1 Reply Last reply Reply Quote 0
                    • vladisld
                      vladisld @orenshkol last edited by

                      @orenshkol said in Running multiple symbols data feeds in live IB:

                      So I set qcheck = 0.001 which does synchronizes the datas when they are all available, which seems to be the best solution

                      The problem (or back side) of such qcheck setting is a high CPU load during live trading. See my previous post about it (https://community.backtrader.com/topic/2346/high-cpu-load-during-pre-market-live-trading-and-adaptive-qcheck-logic).

                      1 Reply Last reply Reply Quote 0
                      • SPECULARI FUND
                        SPECULARI FUND last edited by

                        @orenshkol @vladisld why when I am using with resample data instead adddata, strategy runs on the first symbol only?

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post
                        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors