error in dispatch



  • Hello,

    I connect to IB and add two symbols (for each symbol, 1-min and 3-min bars) to the store; total 4 data feed. However, I get this error very often (not always):

    13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historical
    Data' for 'historicalData' Traceback (most recent call last):
      File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__results.append(listener(message))
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\stores\ibstore.py", line 920, in historicalData    q = self.qs[tickerId] KeyError: 16777221
    13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historicalData' for 'historicalData' Traceback (most recent call last):
      File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__    results.append(listener(message))
    

  • administrators

    @rastegarr said in error in disptach:

    in historicalData q = self.qs[tickerId] KeyError: 16777221

    This is the relevant part. It means that an incoming event for a historical data download with tickerId=16777221 came in and the system is not expecting that.

    The tickerId are purged from the queues expecting data when the historical download is finished or when the historical data is canceled.

    Some extra information is needed to understand why the tickerId is no longer there if a download is actually happening



  • @backtrader I updated to the last version of backtrader and I cannot get that dispatcher error however I'm getting a different issue. The code is very simple:

    from __future__ import (absolute_import, division, print_function, unicode_literals)
    import backtrader as bt
    import time
    
    
    class BuyOnGapStrategy(bt.Strategy):
        def log(self, txt, doprint=False):
            if doprint:
                print('%s' % (txt))
            
        def __init__(self):
            self.log('******************* Strategy Created *********************', doprint = True)
    
        def notify_data(self, data, status, *args, **kwargs):
            # CONNECTED, DISCONNECTED, CONNBROKEN, NOTSUBSCRIBED, DELAYED, LIVE
            self.log('DATA NOTIF: %s, %s' %( data._getstatusname(status), data._dataname), doprint = True)
    
    
        def notify_store(self, msg, *args, **kwargs):
            self.log('-> STORE NOTIF: %s' %(msg), doprint= True)
    
        def notify_order(self, order):
            pass
               
        def notify_trade(self, trade):
            pass
    
        
        def next(self):
            for indx in range(0, len(self.datas), 2):
                datax = self.datas[indx]
                datax2 = self.datas[indx+1]
                if datax is not None and len(datax.datetime) > 0:
                    self.log("Sym %s, Time %s, 1-min Close %.2f" %(datax._dataname, datax.datetime.time(), datax.close[0]), doprint =  True)
                if datax2 is not None and len(datax2.datetime) > 0:
                    self.log("Sym %s, Time %s, 3-min Close %.2f" %(datax2._dataname, datax2.datetime.time(), datax2.close[0]), doprint =  True)
    
    
    if __name__ == '__main__':
        all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD',
         'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']
    
        storekwargs = dict(
            host = "127.0.0.1", port = 4001,
            clientId = 35, timeoffset = True,
            reconnect = True, timeout = 10,
            notifyall = False, _debug = False
        )
    
        ibstore = bt.stores.IBStore(**storekwargs)
        cerebro = bt.Cerebro(exactbars = 1)
        cerebro.setbroker(ibstore.getbroker())
                            
        datakwargs = dict(
                          timeframe = bt.TimeFrame.Minutes, 
                          compression = 1,
                          qcheck = 0.5,
                          historical = False, 
                          backfill_start = True,
                          backfill= True,
                          latethrough = True
                          )
        
        for symbol in all_syms:
            datax = ibstore.getdata(dataname=symbol, **datakwargs)
            cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
            cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
               
        # Add the strategy
        cerebro.addstrategy(BuyOnGapStrategy)
        
        cerebro.run()
    

    Here is message I get:

    Server Version: 76
    TWS Time at connection:20170317 11:58:19 CST
    ******************* Strategy Created *********************
    -> STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connecti
    on is OK:usfarm>
    -> STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection
     is OK:ushmds>
    DATA NOTIF: DELAYED, YHOO-STK-SMART-USD
    DATA NOTIF: DELAYED, CTXS-STK-SMART-USD
    DATA NOTIF: DELAYED, ADSK-STK-SMART-USD
    DATA NOTIF: DELAYED, ETFC-STK-SMART-USD
    DATA NOTIF: DELAYED, DISCA-STK-SMART-USD
    DATA NOTIF: DELAYED, QCOM-STK-SMART-USD
    DATA NOTIF: DELAYED, ADS-STK-SMART-USD
    DATA NOTIF: DELAYED, PVH-STK-SMART-USD
    DATA NOTIF: DELAYED, AMG-STK-SMART-USD
    DATA NOTIF: DELAYED, CMA-STK-SMART-USD
    Traceback (most recent call last):
      File "ib_test.py", line 84, in <module>
        cerebro.run()
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 794,
    in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 924,
    in runstrategies
        self._runnext(runstrats)
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 1240,
     in _runnext
        strat._next()
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 296,
     in _next
        super(Strategy, self)._next()
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineiterator.py", line
    240, in _next
        clock_len = self._clk_update()
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
     in _clk_update
        newdlens = [len(d) for d in self.datas]
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
     in <listcomp>
        newdlens = [len(d) for d in self.datas]
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 43
    2, in __len__
        return len(self.lines)
      File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 19
    9, in __len__
        return len(self.lines[0])
    ValueError: __len__() should return >= 0
    

  • administrators

    @rastegarr said in error in dispatch:

        datax = ibstore.getdata(dataname=symbol, **datakwargs)
        cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
        cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
    

    The recommendation is to create independent data feeds and then resample them individually.

    all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD', 'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']
    

    This is for sure bound to generate a pacing violation in the communication with Interactive Brokers which is going to prevent historical data download for many of the symbols (and each symbol is being downloaded twice)

    Hence

    ValueError: __len__() should return >= 0
    

    At least (probably many) of the data feeds has downloaded absolutely nothing and this has generated a synchronization problem.



  • @backtrader Thanks backorder!

    1. How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?
    2. Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?
      Thanks for your help!

  • administrators

    @rastegarr said in error in dispatch:

    First, use a data feed for each resampling instance you wish. To avoid having duplicate historical requests which are later resampled by the platform. A pacing violation (seeing the amount of symbols) is also probably the cause for the original error.

    1. How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?

    It's not considered in the platform. backtrader is not meant to account for the pacing violations from IB. It tries to automatically get just the amount of data for a single symbol which fits into a request. Reducing the size of a single request will not reduce the amount of requests.

    1. Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?

    By downloading offline, disabling the normal backfilling and using backfill_from, to give you pass a data feed which is reading from the on-disk stored data.

    This of course doesn't allow backfilling after a disconnect/reconnect cycle. That would again hit a pacing limit and trigger a pacing violation with 190 requests (x2, since you hare inputting each symbol twice)

    Interactive Brokers deems itself as a broker and not as a data provider, hence the pacing violations and the recommendation by IB itself to use a real data provider if such scenarios are needed.



  • @backtrader Thanks, for your informative comments. What company would you recommend as a minute-bar data provider that is compatible with backtrader? I'd still like to use IB as the broker since my business partner likes them due to their cost structure.


  • administrators

    If you can write it down to disk, anything will be compatible.

    If you are looking for live minute data with backfilling, the only other such source which is implemented is *[VisualChart][https://www.visualchart.com]. This has the drawback that it only runs under Windows and may be out of your area of influence.

    Communication over COM has also small issues and data download may have to be restarted (COM is not one of the strongest aspects in Python)


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.