error in dispatch

rastegarr

Hello,

I connect to IB and add two symbols (for each symbol, 1-min and 3-min bars) to the store; total 4 data feed. However, I get this error very often (not always):

13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historical
Data' for 'historicalData' Traceback (most recent call last):
  File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__results.append(listener(message))
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\stores\ibstore.py", line 920, in historicalData    q = self.qs[tickerId] KeyError: 16777221
13-Mar-17 15:25:03 ERROR     Exception in message dispatch.  Handler 'historicalData' for 'historicalData' Traceback (most recent call last):
  File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__    results.append(listener(message))

backtrader

@rastegarr said in error in disptach:

in historicalData q = self.qs[tickerId] KeyError: 16777221

This is the relevant part. It means that an incoming event for a historical data download with tickerId=16777221 came in and the system is not expecting that.

The tickerId are purged from the queues expecting data when the historical download is finished or when the historical data is canceled.

Some extra information is needed to understand why the tickerId is no longer there if a download is actually happening

rastegarr

@backtrader I updated to the last version of backtrader and I cannot get that dispatcher error however I'm getting a different issue. The code is very simple:

from __future__ import (absolute_import, division, print_function, unicode_literals)
import backtrader as bt
import time


class BuyOnGapStrategy(bt.Strategy):
    def log(self, txt, doprint=False):
        if doprint:
            print('%s' % (txt))
        
    def __init__(self):
        self.log('******************* Strategy Created *********************', doprint = True)

    def notify_data(self, data, status, *args, **kwargs):
        # CONNECTED, DISCONNECTED, CONNBROKEN, NOTSUBSCRIBED, DELAYED, LIVE
        self.log('DATA NOTIF: %s, %s' %( data._getstatusname(status), data._dataname), doprint = True)


    def notify_store(self, msg, *args, **kwargs):
        self.log('-> STORE NOTIF: %s' %(msg), doprint= True)

    def notify_order(self, order):
        pass
           
    def notify_trade(self, trade):
        pass

    
    def next(self):
        for indx in range(0, len(self.datas), 2):
            datax = self.datas[indx]
            datax2 = self.datas[indx+1]
            if datax is not None and len(datax.datetime) > 0:
                self.log("Sym %s, Time %s, 1-min Close %.2f" %(datax._dataname, datax.datetime.time(), datax.close[0]), doprint =  True)
            if datax2 is not None and len(datax2.datetime) > 0:
                self.log("Sym %s, Time %s, 3-min Close %.2f" %(datax2._dataname, datax2.datetime.time(), datax2.close[0]), doprint =  True)


if __name__ == '__main__':
    all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD',
     'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']

    storekwargs = dict(
        host = "127.0.0.1", port = 4001,
        clientId = 35, timeoffset = True,
        reconnect = True, timeout = 10,
        notifyall = False, _debug = False
    )

    ibstore = bt.stores.IBStore(**storekwargs)
    cerebro = bt.Cerebro(exactbars = 1)
    cerebro.setbroker(ibstore.getbroker())
                        
    datakwargs = dict(
                      timeframe = bt.TimeFrame.Minutes, 
                      compression = 1,
                      qcheck = 0.5,
                      historical = False, 
                      backfill_start = True,
                      backfill= True,
                      latethrough = True
                      )
    
    for symbol in all_syms:
        datax = ibstore.getdata(dataname=symbol, **datakwargs)
        cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
        cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
           
    # Add the strategy
    cerebro.addstrategy(BuyOnGapStrategy)
    
    cerebro.run()

Here is message I get:

Server Version: 76
TWS Time at connection:20170317 11:58:19 CST
******************* Strategy Created *********************
-> STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connecti
on is OK:usfarm>
-> STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection
 is OK:ushmds>
DATA NOTIF: DELAYED, YHOO-STK-SMART-USD
DATA NOTIF: DELAYED, CTXS-STK-SMART-USD
DATA NOTIF: DELAYED, ADSK-STK-SMART-USD
DATA NOTIF: DELAYED, ETFC-STK-SMART-USD
DATA NOTIF: DELAYED, DISCA-STK-SMART-USD
DATA NOTIF: DELAYED, QCOM-STK-SMART-USD
DATA NOTIF: DELAYED, ADS-STK-SMART-USD
DATA NOTIF: DELAYED, PVH-STK-SMART-USD
DATA NOTIF: DELAYED, AMG-STK-SMART-USD
DATA NOTIF: DELAYED, CMA-STK-SMART-USD
Traceback (most recent call last):
  File "ib_test.py", line 84, in <module>
    cerebro.run()
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 794,
in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 924,
in runstrategies
    self._runnext(runstrats)
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 1240,
 in _runnext
    strat._next()
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 296,
 in _next
    super(Strategy, self)._next()
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineiterator.py", line
240, in _next
    clock_len = self._clk_update()
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
 in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285,
 in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 43
2, in __len__
    return len(self.lines)
  File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 19
9, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

backtrader

@rastegarr said in error in dispatch:

    datax = ibstore.getdata(dataname=symbol, **datakwargs)
    cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1)
    cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)

The recommendation is to create independent data feeds and then resample them individually.

all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD', 'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']

This is for sure bound to generate a pacing violation in the communication with Interactive Brokers which is going to prevent historical data download for many of the symbols (and each symbol is being downloaded twice)

Hence

ValueError: __len__() should return >= 0

At least (probably many) of the data feeds has downloaded absolutely nothing and this has generated a synchronization problem.

rastegarr

@backtrader Thanks backorder!

How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?
Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?
Thanks for your help!

backtrader

@rastegarr said in error in dispatch:

First, use a data feed for each resampling instance you wish. To avoid having duplicate historical requests which are later resampled by the platform. A pacing violation (seeing the amount of symbols) is also probably the cause for the original error.

How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?

It's not considered in the platform. backtrader is not meant to account for the pacing violations from IB. It tries to automatically get just the amount of data for a single symbol which fits into a request. Reducing the size of a single request will not reduce the amount of requests.

Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?

By downloading offline, disabling the normal backfilling and using backfill_from, to give you pass a data feed which is reading from the on-disk stored data.

This of course doesn't allow backfilling after a disconnect/reconnect cycle. That would again hit a pacing limit and trigger a pacing violation with 190 requests (x2, since you hare inputting each symbol twice)

Interactive Brokers deems itself as a broker and not as a data provider, hence the pacing violations and the recommendation by IB itself to use a real data provider if such scenarios are needed.

rastegarr

@backtrader Thanks, for your informative comments. What company would you recommend as a minute-bar data provider that is compatible with backtrader? I'd still like to use IB as the broker since my business partner likes them due to their cost structure.

backtrader

If you can write it down to disk, anything will be compatible.

If you are looking for live minute data with backfilling, the only other such source which is implemented is *[VisualChart][https://www.visualchart.com]. This has the drawback that it only runs under Windows and may be out of your area of influence.

Communication over COM has also small issues and data download may have to be restarted (COM is not one of the strongest aspects in Python)

pablomarin

@backtrader is it possible to put a sleep(15) somewhere in the code to overcome this limitation instead of downloading offline and use backfill_from? when requesting the historical data for many symbols, a sleep(15) between each request might do the trick. Let us know.

backtrader

You can also shoot yourself in the knee and then see if it hurts. The problem is the data provider and the amount of symbols requested from it. Blocking a thread 15 seconds to try to overcome a structural problem is not the way.

Feel free to do it yourself.