Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    error in dispatch

    General Discussion
    3
    10
    2325
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rastegarr last edited by backtrader

      Hello,

      I connect to IB and add two symbols (for each symbol, 1-min and 3-min bars) to the store; total 4 data feed. However, I get this error very often (not always):

      13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historical
      Data' for 'historicalData' Traceback (most recent call last):
        File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__results.append(listener(message))
        File "C:\Users\Reza\trading\libs\backtrader\backtrader\stores\ibstore.py", line 920, in historicalData    q = self.qs[tickerId] KeyError: 16777221
      13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historicalData' for 'historicalData' Traceback (most recent call last):
        File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__    results.append(listener(message))
      
      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        @rastegarr said in error in disptach:

        in historicalData q = self.qs[tickerId] KeyError: 16777221

        This is the relevant part. It means that an incoming event for a historical data download with tickerId=16777221 came in and the system is not expecting that.

        The tickerId are purged from the queues expecting data when the historical download is finished or when the historical data is canceled.

        Some extra information is needed to understand why the tickerId is no longer there if a download is actually happening

        R 1 Reply Last reply Reply Quote 0
        • R
          rastegarr @backtrader last edited by

          @backtrader I updated to the last version of backtrader and I cannot get that dispatcher error however I'm getting a different issue. The code is very simple:

          from __future__ import (absolute_import, division, print_function, unicode_literals)
          import backtrader as bt
          import time
          
          
          class BuyOnGapStrategy(bt.Strategy):
              def log(self, txt, doprint=False):
                  if doprint:
                      print('%s' % (txt))
                  
              def __init__(self):
                  self.log('******************* Strategy Created *********************', doprint = True)
          
              def notify_data(self, data, status, *args, **kwargs):
                  # CONNECTED, DISCONNECTED, CONNBROKEN, NOTSUBSCRIBED, DELAYED, LIVE
                  self.log('DATA NOTIF: %s, %s' %( data._getstatusname(status), data._dataname), doprint = True)
          
          
              def notify_store(self, msg, *args, **kwargs):
                  self.log('-> STORE NOTIF: %s' %(msg), doprint= True)
          
              def notify_order(self, order):
                  pass
                     
              def notify_trade(self, trade):
                  pass
          
              
              def next(self):
                  for indx in range(0, len(self.datas), 2):
                      datax = self.datas[indx]
                      datax2 = self.datas[indx+1]
                      if datax is not None and len(datax.datetime) > 0:
                          self.log("Sym %s, Time %s, 1-min Close %.2f" %(datax._dataname, datax.datetime.time(), datax.close[0]), doprint =  True)
                      if datax2 is not None and len(datax2.datetime) > 0:
                          self.log("Sym %s, Time %s, 3-min Close %.2f" %(datax2._dataname, datax2.datetime.time(), datax2.close[0]), doprint =  True)
          
          
          if __name__ == '__main__':
              all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD',
               'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']
          
              storekwargs = dict(
                  host = "127.0.0.1", port = 4001,
                  clientId = 35, timeoffset = True,
                  reconnect = True, timeout = 10,
                  notifyall = False, _debug = False
              )
          
              ibstore = bt.stores.IBStore(**storekwargs)
              cerebro = bt.Cerebro(exactbars = 1)
              cerebro.setbroker(ibstore.getbroker())
                                  
              datakwargs = dict(
                                timeframe = bt.TimeFrame.Minutes, 
                                compression = 1,
                                qcheck = 0.5,
                                historical = False, 
                                backfill_start = True,
                                backfill= True,
                                latethrough = True
                                )
              
              for symbol in all_syms:
                  datax = ibstore.getdata(dataname=symbol, **datakwargs)
                  cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
                  cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
                     
              # Add the strategy
              cerebro.addstrategy(BuyOnGapStrategy)
              
              cerebro.run()
          

          Here is message I get:

          Server Version: 76
          TWS Time at connection:20170317 11:58:19 CST
          ******************* Strategy Created *********************
          -> STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connecti
          on is OK:usfarm>
          -> STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection
           is OK:ushmds>
          DATA NOTIF: DELAYED, YHOO-STK-SMART-USD
          DATA NOTIF: DELAYED, CTXS-STK-SMART-USD
          DATA NOTIF: DELAYED, ADSK-STK-SMART-USD
          DATA NOTIF: DELAYED, ETFC-STK-SMART-USD
          DATA NOTIF: DELAYED, DISCA-STK-SMART-USD
          DATA NOTIF: DELAYED, QCOM-STK-SMART-USD
          DATA NOTIF: DELAYED, ADS-STK-SMART-USD
          DATA NOTIF: DELAYED, PVH-STK-SMART-USD
          DATA NOTIF: DELAYED, AMG-STK-SMART-USD
          DATA NOTIF: DELAYED, CMA-STK-SMART-USD
          Traceback (most recent call last):
            File "ib_test.py", line 84, in <module>
              cerebro.run()
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 794,
          in run
              runstrat = self.runstrategies(iterstrat)
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 924,
          in runstrategies
              self._runnext(runstrats)
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 1240,
           in _runnext
              strat._next()
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 296,
           in _next
              super(Strategy, self)._next()
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineiterator.py", line
          240, in _next
              clock_len = self._clk_update()
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
           in _clk_update
              newdlens = [len(d) for d in self.datas]
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
           in <listcomp>
              newdlens = [len(d) for d in self.datas]
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 43
          2, in __len__
              return len(self.lines)
            File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 19
          9, in __len__
              return len(self.lines[0])
          ValueError: __len__() should return >= 0
          
          B 1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @rastegarr last edited by backtrader

            @rastegarr said in error in dispatch:

                datax = ibstore.getdata(dataname=symbol, **datakwargs)
                cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
                cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
            

            The recommendation is to create independent data feeds and then resample them individually.

            all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD', 'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']
            

            This is for sure bound to generate a pacing violation in the communication with Interactive Brokers which is going to prevent historical data download for many of the symbols (and each symbol is being downloaded twice)

            Hence

            ValueError: __len__() should return >= 0
            

            At least (probably many) of the data feeds has downloaded absolutely nothing and this has generated a synchronization problem.

            R 1 Reply Last reply Reply Quote 0
            • R
              rastegarr @backtrader last edited by

              @backtrader Thanks backorder!

              1. How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?
              2. Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?
                Thanks for your help!
              B 1 Reply Last reply Reply Quote 0
              • B
                backtrader administrators @rastegarr last edited by

                @rastegarr said in error in dispatch:

                First, use a data feed for each resampling instance you wish. To avoid having duplicate historical requests which are later resampled by the platform. A pacing violation (seeing the amount of symbols) is also probably the cause for the original error.

                1. How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?

                It's not considered in the platform. backtrader is not meant to account for the pacing violations from IB. It tries to automatically get just the amount of data for a single symbol which fits into a request. Reducing the size of a single request will not reduce the amount of requests.

                1. Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?

                By downloading offline, disabling the normal backfilling and using backfill_from, to give you pass a data feed which is reading from the on-disk stored data.

                This of course doesn't allow backfilling after a disconnect/reconnect cycle. That would again hit a pacing limit and trigger a pacing violation with 190 requests (x2, since you hare inputting each symbol twice)

                Interactive Brokers deems itself as a broker and not as a data provider, hence the pacing violations and the recommendation by IB itself to use a real data provider if such scenarios are needed.

                R pablomarin 2 Replies Last reply Reply Quote 0
                • R
                  rastegarr @backtrader last edited by

                  @backtrader Thanks, for your informative comments. What company would you recommend as a minute-bar data provider that is compatible with backtrader? I'd still like to use IB as the broker since my business partner likes them due to their cost structure.

                  1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators last edited by

                    If you can write it down to disk, anything will be compatible.

                    If you are looking for live minute data with backfilling, the only other such source which is implemented is *[VisualChart][https://www.visualchart.com]. This has the drawback that it only runs under Windows and may be out of your area of influence.

                    Communication over COM has also small issues and data download may have to be restarted (COM is not one of the strongest aspects in Python)

                    1 Reply Last reply Reply Quote 0
                    • pablomarin
                      pablomarin @backtrader last edited by pablomarin

                      @backtrader is it possible to put a sleep(15) somewhere in the code to overcome this limitation instead of downloading offline and use backfill_from? when requesting the historical data for many symbols, a sleep(15) between each request might do the trick. Let us know.

                      1 Reply Last reply Reply Quote 0
                      • B
                        backtrader administrators last edited by

                        You can also shoot yourself in the knee and then see if it hurts. The problem is the data provider and the amount of symbols requested from it. Blocking a thread 15 seconds to try to overcome a structural problem is not the way.

                        Feel free to do it yourself.

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post
                        Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                        $(document).ready(function () { app.coldLoad(); }); }