error in dispatch
-
Hello,
I connect to IB and add two symbols (for each symbol, 1-min and 3-min bars) to the store; total 4 data feed. However, I get this error very often (not always):
13-Mar-17 15:25:03 ERROR Exception in message dispatch. Handler 'historical Data' for 'historicalData' Traceback (most recent call last): File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__results.append(listener(message)) File "C:\Users\Reza\trading\libs\backtrader\backtrader\stores\ibstore.py", line 920, in historicalData q = self.qs[tickerId] KeyError: 16777221 13-Mar-17 15:25:03 ERROR Exception in message dispatch. Handler 'historicalData' for 'historicalData' Traceback (most recent call last): File "C:\Users\Reza\Anaconda3\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message))
-
@rastegarr said in error in disptach:
in historicalData q = self.qs[tickerId] KeyError: 16777221
This is the relevant part. It means that an incoming event for a historical data download with
tickerId=16777221
came in and the system is not expecting that.The
tickerId
are purged from the queues expecting data when the historical download is finished or when the historical data is canceled.Some extra information is needed to understand why the
tickerId
is no longer there if a download is actually happening -
@backtrader I updated to the last version of backtrader and I cannot get that dispatcher error however I'm getting a different issue. The code is very simple:
from __future__ import (absolute_import, division, print_function, unicode_literals) import backtrader as bt import time class BuyOnGapStrategy(bt.Strategy): def log(self, txt, doprint=False): if doprint: print('%s' % (txt)) def __init__(self): self.log('******************* Strategy Created *********************', doprint = True) def notify_data(self, data, status, *args, **kwargs): # CONNECTED, DISCONNECTED, CONNBROKEN, NOTSUBSCRIBED, DELAYED, LIVE self.log('DATA NOTIF: %s, %s' %( data._getstatusname(status), data._dataname), doprint = True) def notify_store(self, msg, *args, **kwargs): self.log('-> STORE NOTIF: %s' %(msg), doprint= True) def notify_order(self, order): pass def notify_trade(self, trade): pass def next(self): for indx in range(0, len(self.datas), 2): datax = self.datas[indx] datax2 = self.datas[indx+1] if datax is not None and len(datax.datetime) > 0: self.log("Sym %s, Time %s, 1-min Close %.2f" %(datax._dataname, datax.datetime.time(), datax.close[0]), doprint = True) if datax2 is not None and len(datax2.datetime) > 0: self.log("Sym %s, Time %s, 3-min Close %.2f" %(datax2._dataname, datax2.datetime.time(), datax2.close[0]), doprint = True) if __name__ == '__main__': all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD', 'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD'] storekwargs = dict( host = "127.0.0.1", port = 4001, clientId = 35, timeoffset = True, reconnect = True, timeout = 10, notifyall = False, _debug = False ) ibstore = bt.stores.IBStore(**storekwargs) cerebro = bt.Cerebro(exactbars = 1) cerebro.setbroker(ibstore.getbroker()) datakwargs = dict( timeframe = bt.TimeFrame.Minutes, compression = 1, qcheck = 0.5, historical = False, backfill_start = True, backfill= True, latethrough = True ) for symbol in all_syms: datax = ibstore.getdata(dataname=symbol, **datakwargs) cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1) cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3) # Add the strategy cerebro.addstrategy(BuyOnGapStrategy) cerebro.run()
Here is message I get:
Server Version: 76 TWS Time at connection:20170317 11:58:19 CST ******************* Strategy Created ********************* -> STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connecti on is OK:usfarm> -> STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds> DATA NOTIF: DELAYED, YHOO-STK-SMART-USD DATA NOTIF: DELAYED, CTXS-STK-SMART-USD DATA NOTIF: DELAYED, ADSK-STK-SMART-USD DATA NOTIF: DELAYED, ETFC-STK-SMART-USD DATA NOTIF: DELAYED, DISCA-STK-SMART-USD DATA NOTIF: DELAYED, QCOM-STK-SMART-USD DATA NOTIF: DELAYED, ADS-STK-SMART-USD DATA NOTIF: DELAYED, PVH-STK-SMART-USD DATA NOTIF: DELAYED, AMG-STK-SMART-USD DATA NOTIF: DELAYED, CMA-STK-SMART-USD Traceback (most recent call last): File "ib_test.py", line 84, in <module> cerebro.run() File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 794, in run runstrat = self.runstrategies(iterstrat) File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 924, in runstrategies self._runnext(runstrats) File "C:\Users\Reza\trading\libs\backtrader\backtrader\cerebro.py", line 1240, in _runnext strat._next() File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 296, in _next super(Strategy, self)._next() File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineiterator.py", line 240, in _next clock_len = self._clk_update() File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285, in _clk_update newdlens = [len(d) for d in self.datas] File "C:\Users\Reza\trading\libs\backtrader\backtrader\strategy.py", line 285, in <listcomp> newdlens = [len(d) for d in self.datas] File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 43 2, in __len__ return len(self.lines) File "C:\Users\Reza\trading\libs\backtrader\backtrader\lineseries.py", line 19 9, in __len__ return len(self.lines[0]) ValueError: __len__() should return >= 0
-
@rastegarr said in error in dispatch:
datax = ibstore.getdata(dataname=symbol, **datakwargs) cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 1) cerebro.resampledata(dataname=datax, timeframe = bt.TimeFrame.Minutes, compression = 3)
The recommendation is to create independent data feeds and then resample them individually.
all_syms = ['YHOO-STK-SMART-USD', 'CTXS-STK-SMART-USD', 'ADSK-STK-SMART-USD', 'ETFC-STK-SMART-USD', 'DISCA-STK-SMART-USD', 'QCOM-STK-SMART-USD', 'ADS-STK-SMART-USD', 'PVH-STK-SMART-USD', 'AMG-STK-SMART-USD', 'CMA-STK-SMART-USD']
This is for sure bound to generate a pacing violation in the communication with Interactive Brokers which is going to prevent historical data download for many of the symbols (and each symbol is being downloaded twice)
Hence
ValueError: __len__() should return >= 0
At least (probably many) of the data feeds has downloaded absolutely nothing and this has generated a synchronization problem.
-
@backtrader Thanks backorder!
- How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?
- Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?
Thanks for your help!
-
@rastegarr said in error in dispatch:
First, use a data feed for each resampling instance you wish. To avoid having duplicate historical requests which are later resampled by the platform. A pacing violation (seeing the amount of symbols) is also probably the cause for the original error.
- How can I get only last 30 minutes of the historical data so this way I won't violate the IB download constraints?
It's not considered in the platform.
backtrader
is not meant to account for the pacing violations from IB. It tries to automatically get just the amount of data for a single symbol which fits into a request. Reducing the size of a single request will not reduce the amount of requests.- Since I have about 190 symbols, how can I have refill (in the beginning and upon the disconnect) without violating pacing violation?
By downloading offline, disabling the normal backfilling and using
backfill_from
, to give you pass a data feed which is reading from the on-disk stored data.This of course doesn't allow backfilling after a disconnect/reconnect cycle. That would again hit a pacing limit and trigger a pacing violation with 190 requests (x2, since you hare inputting each symbol twice)
Interactive Brokers deems itself as a broker and not as a data provider, hence the pacing violations and the recommendation by IB itself to use a real data provider if such scenarios are needed.
-
@backtrader Thanks, for your informative comments. What company would you recommend as a minute-bar data provider that is compatible with backtrader? I'd still like to use IB as the broker since my business partner likes them due to their cost structure.
-
If you can write it down to disk, anything will be compatible.
If you are looking for live minute data with backfilling, the only other such source which is implemented is *[VisualChart][https://www.visualchart.com]. This has the drawback that it only runs under Windows and may be out of your area of influence.
Communication over
COM
has also small issues and data download may have to be restarted (COM
is not one of the strongest aspects in Python) -
@backtrader is it possible to put a sleep(15) somewhere in the code to overcome this limitation instead of downloading offline and use backfill_from? when requesting the historical data for many symbols, a sleep(15) between each request might do the trick. Let us know.
-
You can also shoot yourself in the knee and then see if it hurts. The problem is the data provider and the amount of symbols requested from it. Blocking a thread 15 seconds to try to overcome a structural problem is not the way.
Feel free to do it yourself.