Posts made by vladisld

After further check - it seems the problem was caused by my own changes. The warning messages were pushed by mistake to the same message queue that was used by 'reqContractDetails'.

My apologies. Please ignore this topic.

Got a strange error in today's live trading session (still paper IB account) upon requesting contract details for some of my subscriptions (It happened just today, however the strategy is already running for few days without any changes in its code)

The error:

Ambiguous contract: none/multiple answers received

is reported after 'reqContractDetails' request is issued. At first I suspected that something is wrong with the contract definition, however it was not the case (the same contract info is used for more than a month already without any change).

after a short investigation it seems that it is a bug in IBStore.getContractDetails code:

    def getContractDetails(self, contract, maxcount=None):
        cds = list()
        q = self.reqContractDetails(contract)
        while True:
            msg = q.get()
            if msg is None:
                break
            cds.append(msg)

        if not cds or (maxcount and len(cds) > maxcount):
            err = 'Ambiguous contract: none/multiple answers received'
            self.notifs.put((err, cds, {}))
            return None

        return cds

The problem is that the above code assumes that after requesting the contract details ('reqContractDetails' call), the only messages that are going to be returned are the replies with the contract details. However this appears not always to be the case. Here the logs:

14:01:11:754 <- 9-7-16777220-0-MU-STK--0---SMART-USD---0---

14:01:12:035 -> 4-2--1-2106-HMDS data farm connection is OK:ushmds-

14:01:12:151 -> 10-8-16777220-MU-STK--0--SMART-USD-MU-NMS-NMS-9939-0.01--ACTIVETIM,AD,ADJUST,ALERT,ALGO,ALLOC,AVGCOST,BASKET,BENCHPX,COND,CONDORDER,DARKONLY,DARKPOLL,DAY,DEACT,DEACTDIS,DEACTEOD,DIS,GAT,GTC,GTD,GTT,HID,IBKRATS,ICE,IMB,IOC,LIT,LMT,LOC,MIDPX,MIT,MKT,MOC,MTL,NGCOMB,NODARK,NONALGO,OCA,OPG,OPGREROUT,PEGBENCH,POSTONLY,PREOPGRTH,REL,RPI,RTH,SCALE,SCALEODD,SCALERST,SMARTSTG,SNAPMID,SNAPMKT,SNAPREL,STP,STPLMT,SWEEP,TRAIL,TRAILLIT,TRAILLMT,TRAILMIT,WHATIF-SMART,AMEX,NYSE,CBOE,PHLX,ISE,CHX,ARCA,ISLAND,DRCTEDGE,BEX,BATS,EDGEA,CSFBALGO,JEFFALGO,BYX,IEX,EDGX,FOXRIVER,TPLUS1,NYSENAT,PSX-1-0-MICRON TECHNOLOGY INC-NASDAQ--Technology-Semiconductors-Electronic Compo-Semicon-EST (Eastern Standard Time)-20200214:0400-20200214:2000;20200215:CLOSED;20200216:CLOSED;20200217:CLOSED;20200218:0400-20200218:2000;20200219:0400-20200219:2000;20200220:0400-20200220:2000;20200221:0400-20200221:2000;20200222:CLOSED;20200223:CLOSED;20200224:0400-20200224:2000;20200225:0400-20200225:2000;20200226:0400-20200226:2000;20200227:0400-20200227:2000;20200228:0400-20200228:2000;20200229:CLOSED;20200301:CLOSED;20200302:0400-20200302:2000;20200303:0400-20200303:2000;20200304:0400-20200304:2000;20200305:0400-20200305:2000;20200306:0400-20200306:2000;20200307:CLOSED;20200308:CLOSED;20200309:0400-20200309:2000;20200310:0400-20200310:2000;20200311:0400-20200311:2000;20200312:0400-20200312:2000;20200313:0400-20200313:2000;20200314:CLOSED;20200315:CLOSED;20200316:0400-20200316:2000;20200317:0400-20200317:2000;20200318:0400-20200318:2000;20200319:0400-20200319:2000-20200214:0930-20200214:1600;20200215:CLOSED;20200216:CLOSED;20200217:CLOSED;20200218:0930-20200218:1600;20200219:0930-20200219:1600;20200220:0930-20200220:1600;20200221:0930-20200221:1600;20200222:CLOSED;20200223:CLOSED;20200224:0930-20200224:1600;20200225:0930-20200225:1600;20200226:0930-20200226:1600;20200227:0930-20200227:1600;20200228:0930-20200228:1600;20200229:CLOSED;20200301:CLOSED;20200302:0930-20200302:1600;20200303:0930-20200303:1600;20200304:0930-20200304:1600;20200305:0930-20200305:1600;20200306:0930-20200306:1600;20200307:CLOSED;20200308:CLOSED;20200309:0930-20200309:1600;20200310:0930-20200310:1600;20200311:0930-20200311:1600;20200312:0930-20200312:1600;20200313:0930-20200313:1600;20200314:CLOSED;20200315:CLOSED;20200316:0930-20200316:1600;20200317:0930-20200317:1600;20200318:0930-20200318:1600;20200319:0930-20200319:1600---0-

14:01:12:151 -> 52-1-16777220-

As could be seen, an additional warning message (code 2106) was received before the contract details reply. However the code in 'reqContractDetails' doesn't check the expected message type and counts the warning (2106) message as yet another contract details message - resulting in "multiple answers" conclusion.

After restarting the strategy few times, the error was reproduced at least three more times today - so it is not as rare and it may seem at first.

Will try to fix it by checking the expected message type and upload the PR.

BTW it seems there are quite a few PRs from multiple contributors already pending and nobody are reviewing/integrating them - would be interested to know what is the policy here, as I'm quite new to the game.

Thanks
Vlad

@Yury-Tsar said in Multiple strategies on single data:

@vladisld
question was how to run them simultaneously, not using iteration 1 after another.

This wasn't clear from the original question:

@Yury-Tsar said in Multiple strategies on single data:

how to implement different strategies on single data feed

In case you would like to backtest the same strategy with a different parameters, where each run of the strategy will have its own position, and all of them will be run simultaneously, but still having the same "big OHLC" data loaded just once - this is exactly were the cerebro.optstrategy comes to rescue. Cerebro engine could be setup to preload the data feed (so that it will be read just once) and each strategy instance will run in its own process - all of them running simultaneously as you wished.

Am I still getting your question wrong ?

If your strategies are completely different you may just add all of them to cerebro - each one will be given the same data feed upon start, also each strategy's next method will be called once for each update in the data feed.
Alternatively you may use a more advanced technique described here: (https://www.backtrader.com/blog/posts/2016-10-29-strategy-selection/strategy-selection/) and here: (https://www.backtrader.com/blog/posts/2017-05-16-stsel-revisited/stsel-revisited/)

In case the only difference between your strategies is in a single parameter (% stop loss) you may set it as a parameter of the same strategy like:

class MyStrategy(bt.Strategy):
    params = (
        ('stop_loss_percent', 0.96),
    )

and use the cerebro.optstrategy instead of using cerebro.addstrategy. The 'optstrategy' method accepts a range for each strategy's parameter and will instantiate the same strategy for each value of the parameter in the above range. See more info here: (https://www.backtrader.com/docu/cerebro/)

found the answer here: ERROR: len() should return >= 0 on liveFeed

TWS API defines the following warning message codes that application (Backtrader in this case) may usually safely ignore:

* 2105	A historical data farm is disconnected.	

Indicates a connectivity problem to an IB server. Outside of the nightly IB server reset, this typically indicates an underlying ISP connectivity issue.

* 2106	A historical data farm is connected. 	

A notification that connection to the market data server is ok. This is a notification and not a true error condition, and is expected on first establishing connection. 

However in some circumstances the 2105 error may cause all data subscriptions (realtimeBar subscriptions in particular) to stop working. One such scenario is described in the following TWSAPI group post (https://groups.io/g/twsapi/topic/6374066#38957), and it happens in my case as well.

Upon running the Backtrader strategy in live mode using IB paper account, the above error notification may be sent by IB in case another connection to the corresponding IB live account is established (probably from another machine or even mobile phone). In this case all the data subscriptions stop working for the paper account session even if the live session was ended. Usually the 2105 error is followed by 2106 error message right away:

<error id=-1, errorCode=2105, errorMsg=HMDS data farm connection is broken:ushmds>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>

*** no more data notifications are received from this point on

Currently IBStore implementation in Backtrader just ignores such error messages. So I wonder whether the above scenario is severe enough to add a support in IBStore/IBData ? (I'll upload the PR if there is a consensus)

Thanks a lot @Zachary-Harris

I've got the same error during live trading through IB (see my post here: https://community.backtrader.com/topic/642/iqfeed-store-and-feed-help-needed/12 ) and your fix seems to be working great.

This seems to be a pretty serious issue that should be fixed in the official source.

I just wonder why there are so little reports about it - it should have happen pretty often to everyone using backtrader for live trading.

Please check the following similar posts (probably you may have the same problem):

https://community.backtrader.com/topic/407/indexerror-array-assignment-index-out-of-range
https://community.backtrader.com/topic/1797/getting-indexerror-array-assignment-index-out-of-range-in-cerebro-run

Just to bring one more point regarding the throttling mechanism. Starting from v.974 of TWS API there is a '+PACEAPI' connection option that is effectively pacing the incoming messages. Quoting (https://www.interactivebrokers.com/en/index.php?f=5061):

"""
API messages sent at a higher rate than 50/second can now be paced by TWS at the 50/second rate instead of potentially causing a disconnection. This is now done automatically by the RTD Server API and can be done with other API technologies by invoking SetConnectOptions("+PACEAPI") prior to eConnect.
"""

Unfortunately I'm not sure IbPy library is updated to include the SetConnectOptions method. AFAIU the IbPy is just generating its source code from the official TWS Java API (by translating it to Python), but it hasn't been updated to the latest version yet. This is probably one of the reasons to directly use the official TWS Python API instead of IbPy.

Hi Ed,Daniel

I know this thread is pretty old, but probably you may remember how you've overcome the above problem.

Exactly the same error appears right now (the same stack as well) while live trading using IB with multiple datas using RTVolume prices (rtbar=False, timeframe=Seconds, compression=5, no resampling, no replay for all datas):

  File "/home/vladisld_trade/run_multi_ib.py", line 144, in run
    cerebro.run()
  File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1630, in _runnext
    strat._next()
  File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 347, in _next
    super(Strategy, self)._next()
  File "/home/vladisld_trade/backtrader/backtrader/lineiterator.py", line 260, in _next
    clock_len = self._clk_update()
  File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 327, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 327, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "/home/vladisld_trade/backtrader/backtrader/lineseries.py", line 464, in __len__
    return len(self.lines)
  File "/home/vladisld_trade/backtrader/backtrader/lineseries.py", line 220, in __len__
    return len(self.lines[0])
  ValueError: __len__() should return >= 0

The following events were preceding the crash:

<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
<error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefeu>
<error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:usfarm>
<error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:usfarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
<marketDataType reqId=16777217, marketDataType=1>
<tickString tickerId=16777217, tickType=45, value=1581108818>
<tickPrice tickerId=16777217, field=4, price=6.64, canAutoExecute=0>
<tickSize tickerId=16777217, field=5, size=1>
<tickSize tickerId=16777217, field=5, size=1>
<tickSize tickerId=16777217, field=8, size=18642>
<tickPrice tickerId=16777217, field=6, price=6.66, canAutoExecute=0>
<tickPrice tickerId=16777217, field=7, price=6.6, canAutoExecute=0>
<tickPrice tickerId=16777217, field=9, price=6.64, canAutoExecute=0>
<tickPrice tickerId=16777217, field=14, price=6.64, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=6.64, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=316>
<tickPrice tickerId=16777217, field=2, price=6.65, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=491>
<tickSize tickerId=16777217, field=0, size=316>
<tickSize tickerId=16777217, field=3, size=491>
<tickString tickerId=16777217, tickType=32, value=AIXBQTMHJ>
<tickString tickerId=16777217, tickType=33, value=ACIXBQTMHJ>
<marketDataType reqId=16777219, marketDataType=1>
<tickPrice tickerId=16777219, field=1, price=4.37, canAutoExecute=1>
<tickSize tickerId=16777219, field=0, size=0>
<tickString tickerId=16777219, tickType=45, value=1581108819>
<tickPrice tickerId=16777219, field=4, price=4.39, canAutoExecute=0>
<tickSize tickerId=16777219, field=5, size=1>
<tickSize tickerId=16777219, field=5, size=1>
<tickSize tickerId=16777219, field=8, size=978454>
<tickPrice tickerId=16777219, field=6, price=4.42, canAutoExecute=0>
<tickPrice tickerId=16777219, field=7, price=4.26, canAutoExecute=0>
<tickPrice tickerId=16777219, field=9, price=4.2, canAutoExecute=0>
<tickPrice tickerId=16777219, field=14, price=4.27, canAutoExecute=0>
<tickPrice tickerId=16777219, field=1, price=4.39, canAutoExecute=1>
<tickSize tickerId=16777219, field=0, size=6611>
<tickPrice tickerId=16777219, field=2, price=4.4, canAutoExecute=1>
<tickSize tickerId=16777219, field=3, size=8356>
<tickSize tickerId=16777219, field=0, size=6611>
<tickSize tickerId=16777219, field=3, size=8356>
<tickString tickerId=16777219, tickType=32, value=ACIXBQTMHJ>
<tickString tickerId=16777219, tickType=33, value=ACIXBQTMHJ>
<marketDataType reqId=16777221, marketDataType=1>
<tickString tickerId=16777221, tickType=45, value=1581108819>
<tickPrice tickerId=16777221, field=4, price=56.93, canAutoExecute=0>
<tickSize tickerId=16777221, field=5, size=1>
<tickSize tickerId=16777221, field=5, size=1>
<tickSize tickerId=16777221, field=8, size=179048>
<tickPrice tickerId=16777221, field=6, price=58.0, canAutoExecute=0>
<tickPrice tickerId=16777221, field=7, price=56.77, canAutoExecute=0>
<tickPrice tickerId=16777221, field=9, price=58.68, canAutoExecute=0>
<tickPrice tickerId=16777221, field=14, price=57.97, canAutoExecute=0>
<tickPrice tickerId=16777221, field=1, price=56.93, canAutoExecute=1>
<tickSize tickerId=16777221, field=0, size=6>
<tickPrice tickerId=16777221, field=2, price=56.94, canAutoExecute=1>
<tickSize tickerId=16777221, field=3, size=43>
<tickSize tickerId=16777221, field=0, size=6>
<tickSize tickerId=16777221, field=3, size=43>
<tickString tickerId=16777221, tickType=32, value=E>
<tickString tickerId=16777221, tickType=33, value=IBQWTEJ>
<marketDataType reqId=16777223, marketDataType=1>
<tickString tickerId=16777223, tickType=45, value=1581108818>
<tickPrice tickerId=16777223, field=4, price=12.15, canAutoExecute=0>
<tickSize tickerId=16777223, field=5, size=12>
<tickSize tickerId=16777223, field=5, size=12>
<tickSize tickerId=16777223, field=8, size=97877>
<tickPrice tickerId=16777223, field=6, price=12.17, canAutoExecute=0>
<tickPrice tickerId=16777223, field=7, price=11.98, canAutoExecute=0>
<tickPrice tickerId=16777223, field=9, price=12.12, canAutoExecute=0>
<tickPrice tickerId=16777223, field=14, price=12.14, canAutoExecute=0>
<tickPrice tickerId=16777223, field=1, price=12.14, canAutoExecute=1>
<tickSize tickerId=16777223, field=0, size=378>
<tickPrice tickerId=16777223, field=2, price=12.15, canAutoExecute=1>
<tickSize tickerId=16777223, field=3, size=249>
<tickSize tickerId=16777223, field=0, size=378>
<tickSize tickerId=16777223, field=3, size=249>
<tickString tickerId=16777223, tickType=32, value=ACIXBQTMHJ>
<tickString tickerId=16777223, tickType=33, value=AIXQTMHJ>
<marketDataType reqId=16777225, marketDataType=1>
<tickString tickerId=16777225, tickType=45, value=1581108819>
<tickPrice tickerId=16777225, field=4, price=49.78, canAutoExecute=0>
<tickSize tickerId=16777225, field=5, size=2>
<tickSize tickerId=16777225, field=5, size=2>
<tickSize tickerId=16777225, field=8, size=497234>
<tickPrice tickerId=16777225, field=6, price=50.67, canAutoExecute=0>
<tickPrice tickerId=16777225, field=7, price=48.63, canAutoExecute=0>
<tickPrice tickerId=16777225, field=9, price=49.32, canAutoExecute=0>
<tickPrice tickerId=16777225, field=14, price=48.93, canAutoExecute=0>
<tickPrice tickerId=16777225, field=1, price=49.78, canAutoExecute=1>
<tickSize tickerId=16777225, field=0, size=38>
<tickPrice tickerId=16777225, field=2, price=49.79, canAutoExecute=1>
<tickSize tickerId=16777225, field=3, size=98>
<tickSize tickerId=16777225, field=0, size=38>
<tickSize tickerId=16777225, field=3, size=98>
<tickString tickerId=16777225, tickType=32, value=QWT>
<tickString tickerId=16777225, tickType=33, value=ACIBQWTMEJ>
<marketDataType reqId=16777227, marketDataType=1>
<tickPrice tickerId=16777227, field=2, price=13.1, canAutoExecute=1>
<tickSize tickerId=16777227, field=3, size=0>
<tickString tickerId=16777227, tickType=45, value=1581108817>
<tickPrice tickerId=16777227, field=4, price=13.15, canAutoExecute=0>
<tickSize tickerId=16777227, field=5, size=1>
<tickSize tickerId=16777227, field=5, size=1>
<tickPrice tickerId=16777227, field=14, price=13.79, canAutoExecute=0>
<tickSize tickerId=16777227, field=8, size=3442>
<tickPrice tickerId=16777227, field=6, price=13.79, canAutoExecute=0>
<tickPrice tickerId=16777227, field=7, price=13.02, canAutoExecute=0>
<tickPrice tickerId=16777227, field=9, price=13.79, canAutoExecute=0>
<tickPrice tickerId=16777225, field=4, price=49.79, canAutoExecute=0>
<tickSize tickerId=16777225, field=5, size=2>
<tickSize tickerId=16777225, field=8, size=497236>
<tickPrice tickerId=16777227, field=1, price=13.13, canAutoExecute=1>
<tickSize tickerId=16777227, field=0, size=5>
<tickPrice tickerId=16777227, field=2, price=13.17, canAutoExecute=1>
<tickSize tickerId=16777227, field=3, size=6>
<tickSize tickerId=16777227, field=0, size=5>
<tickSize tickerId=16777227, field=3, size=6>
<tickString tickerId=16777227, tickType=32, value=XB>
<tickString tickerId=16777227, tickType=33, value=XM>
<tickString tickerId=16777221, tickType=48, value=56.93;4;1581108819203;179276;57.21285544;false>
<tickString tickerId=16777221, tickType=48, value=;0;1581108819206;179276;57.21285544;true>
<tickString tickerId=16777221, tickType=48, value=;0;1581108819207;179276;57.21285544;true>
<tickPrice tickerId=16777221, field=1, price=56.92, canAutoExecute=1>
<tickSize tickerId=16777221, field=0, size=69>
<tickPrice tickerId=16777221, field=2, price=56.93, canAutoExecute=1>
<tickSize tickerId=16777221, field=3, size=41>
<tickSize tickerId=16777221, field=0, size=69>
<tickSize tickerId=16777221, field=3, size=41>
<tickString tickerId=16777221, tickType=32, value=IQWTE>
<tickString tickerId=16777221, tickType=33, value=TE>
<tickString tickerId=16777221, tickType=48, value=56.93;2;1581108819400;179278;57.21285224;false>
<tickString tickerId=16777221, tickType=48, value=;0;1581108819401;179278;57.21285224;true>
<tickString tickerId=16777225, tickType=48, value=49.78;37;1581108819401;497722;50.08132026;false>
<tickString tickerId=16777225, tickType=48, value=;0;1581108819401;497722;50.08132026;true>
<tickString tickerId=16777225, tickType=48, value=;0;1581108819403;497722;50.08132026;true>
<tickSize tickerId=16777221, field=8, size=179049>
<tickPrice tickerId=16777221, field=4, price=56.92, canAutoExecute=0>
<tickSize tickerId=16777221, field=5, size=1>
<tickSize tickerId=16777227, field=8, size=3446>
<tickPrice tickerId=16777225, field=1, price=49.77, canAutoExecute=1>
<tickSize tickerId=16777225, field=0, size=129>
<tickPrice tickerId=16777225, field=2, price=49.78, canAutoExecute=1>
<tickSize tickerId=16777225, field=3, size=84>
<tickSize tickerId=16777225, field=0, size=129>
<tickSize tickerId=16777225, field=3, size=84>
<tickString tickerId=16777225, tickType=32, value=AIQWTE>
<tickString tickerId=16777225, tickType=33, value=TE>
<tickString tickerId=16777219, tickType=48, value=4.39;187;1581108819649;978667;4.35446026;false>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819650;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819650;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819652;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819652;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819653;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819653;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819654;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819655;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819656;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819656;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819659;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819661;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819661;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819669;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819670;978667;4.35446026;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819670;978667;4.35446026;true>
<tickString tickerId=16777225, tickType=48, value=49.78;1;1581108819670;497723;50.08131965;true>
<tickString tickerId=16777219, tickType=45, value=1581108820>
<tickSize tickerId=16777219, field=5, size=39>
<tickSize tickerId=16777219, field=8, size=978493>
<tickString tickerId=16777225, tickType=45, value=1581108820>
<tickPrice tickerId=16777225, field=4, price=49.78, canAutoExecute=0>
<tickSize tickerId=16777225, field=5, size=1>
<tickSize tickerId=16777225, field=5, size=1>
<tickSize tickerId=16777225, field=8, size=497237>
<tickSize tickerId=16777223, field=3, size=250>
<tickString tickerId=16777219, tickType=48, value=4.39;24;1581108819900;978691;4.35446113;false>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819900;978691;4.35446113;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819900;978691;4.35446113;true>
<tickString tickerId=16777219, tickType=48, value=;0;1581108819901;978691;4.35446113;true>
<tickSize tickerId=16777219, field=0, size=6408>
<tickSize tickerId=16777219, field=3, size=8511>
<tickSize tickerId=16777221, field=0, size=77>
<tickSize tickerId=16777221, field=3, size=10>
<tickString tickerId=16777221, tickType=32, value=CIQWTMEJ>
<tickString tickerId=16777221, tickType=33, value=WT>
<tickSize tickerId=16777217, field=3, size=481>
Traceback (most recent call last):
...
<the stacktrace you saw above goes here>
...

Thanks
Vlad

@bigdavediode said in Interactive Brokers Multiple reqMktData:

My strategy subscribes to about 37 symbols

It still hard to explain the "Max rate of messages per second has been exceeded" (error 100) without looking at the TWS API logs. AFAIK, TWS API has 50 msg/sec limitation averaged over 5 sec (i.e 250 / 5sec ) in addition to 60 historical data requests within 10 min and some other more obscure limitations (https://interactivebrokers.github.io/tws-api/historical_limitations.html)

So although I agree that throttling mechanism should be in place ( I'd argue if it needs to be implemented in backtrader or in the lower component - ibpy, like the one implemented ib_insync library), I would still first make sure that there is no excessive requests generated by the current backtrader code by looking at the TWS API logs.

@bigdavediode said in Interactive Brokers Multiple reqMktData:

the combination of my historical data requests combined with live requests still jointly caused a pacing error

That sounds strange, given that you just subscribe for approx 10 symbols. It would be interesting to take a look at the TWS API logs to see what is really happening.

AFAIU, upon resampling new bars are built aggregating the information from the originals bars of the feed with a smaller timeframe. Those new bars have no knowledge about the non-standard attributes of the originals bars ( only datetime, high, low, open, close, volume and openinterest fields are processed), nor they know anything about how to aggregate such fields. This may explain the behavior you saw.

Thanks
Vlad

Please see my post in "TWS API Users Group": https://groups.io/g/twsapi/topic/69943076

Up to the TWS API v972, IB realtimeBar subscriptions could stop working during the trading session and no error notification was sent to the application. This happens because of what's called 'busts' events.

Quoting the response in "TWS API Users Group" question:

"The busts occur when the datastream is interrupted or modified in the middle of a bar and the subscription is cancelled by the backend. They should be relatively uncommon but unfortunately no can't be avoided completely"

In TWS API v978,a special error code was added to notify the application about the bust event so that the subscription could be renewed. Below is the example of such error message (from TWS Gateway API logs):

15:24:00:896 -> 4-2-16777221-10225-Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately.-

Currently backtrader doesn't process the bust event error code (10225).

I've submitted PR with a fix: https://github.com/mementum/backtrader/pull/396

(bwt this PR also fixes the issue with multiple strategies and data feeds: https://community.backtrader.com/topic/2122/live-trading-multiple-strategies-each-one-on-a-separate-data-feed, although it could be separated into its own PR if at all)

Thanks
Vlad

After further testing, the above 'fix' appears not to be good enough. The problem is with connection reset events:

<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>

In case of connection resets, the state of the data feeds seems not to be reset and stays in _ST_LIVE state.

Here the code from ibdata.py IBData class:

    def _load(self):
            # (vladisld) ... code ommitted for clarity
            while True:
                if self._state == self._ST_LIVE:
                    # (vladisld) ... code ommitted for clarity
                    elif msg == -1100:  # conn broken
                        # Tell to wait for a message to do a backfill
                        # self._state = self._ST_DISCONN
                        self._statelivereconn = self.p.backfill
                        continue
    

Original code (without fix) would still reconnect the data feeds in that state. However with my 'fix' the reconnect will not work anymore.

Will update if a better fix is found. Any suggestion is appreciated of cause.

Thanks
Vlad

Are you expecting multiple time frames for the same symbol to be in the same CSV file ? ( in that case how do you know which time frame each line is associated with ?

I didn't try it myself, but probably the following may work:

class MyGenericCSVData(bt.GenericCSVData):
    def start(self):
        super(MyGenericCSVData, self).start()
        if self.p.headers:
            self.f.readline()  # skip the second header (the first one was skipped in GenericCSVData start method)

class MyGenericCSV(bt.GenericCSV):
    DataCls = MyGenericCSVData

Thanks
Vlad

It seems the 'fix' worked pretty well on real time (paper) trading.

2020-01-17 15:45:00:HMI-STK-SMART-USD:track:notify_order: Market Buy Partial, price: 14.63, size: 400, cost: 5852.0, comm: 2.0, ref: 7
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits: p1: 0, p2: 3
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 0, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 1, dt: 2020-01-17 15:45:06, bit size:200, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 2, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits done.
...
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:notify_order: Market Buy Partial, price: 14.63, size: 500, cost: 7315.0, comm: 2.5, ref: 7
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits: p1: 3, p2: 4
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 0, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 1, dt: 2020-01-17 15:45:06, bit size:200, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 2, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 3, dt: 2020-01-17 15:45:08, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 4, dt: 2020-01-17 15:45:09, bit size:182, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits done.
...
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:notify_order: Market Buy Completed, price: 14.629999999999999, size: 682, cost: 9977.66, comm: 3.41, ref: 7
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits: p1: 4, p2: 5
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 0, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 1, dt: 2020-01-17 15:45:06, bit size:200, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 2, dt: 2020-01-17 15:45:06, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 3, dt: 2020-01-17 15:45:08, bit size:100, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:idx: 4, dt: 2020-01-17 15:45:09, bit size:182, bit price:14.63
2020-01-17 15:45:00:HMI-STK-SMART-USD:track:order bits done.

Now I wonder how such 'fix' would have been tested in normal scenario ( in dev environment). Usually this requires 'component level testing' of IBStore/IBBroker/IBFeed components. However this will in turn require some mockups for interfaces below (celebro) and above the IB implementation (IBPy2) that are not existing today.

How are you usually testing the components outside the core ?

I couldn't find such tests in the package, but given that this is already a second issue I've seen in IBStore/IBBroker/IBFeed component it is a good idea to add some tests (at least to cover the issues already found) - volunteering to help with it if accepted by maintainer.

Thanks
Vlad

A simple fix maybe to call self.markpending() before cloning the OrderData:

def clone(self):
        self.markpending()  # <-- here
        obj = copy(self)
        return obj

Any better idea?

IIRC if optstrategy is used, the 'run' method will return a list of lists. Quoting the docs ( 'Returning the results' section of the cerebro section https://www.backtrader.com/docu/cerebro/):

Returning the results

cerebro returns the instances of the strategies it created during backtesting. This allows to analyze what they did, because all elements in the strategies are accessible:

result = cerebro.run(**kwargs)

The format of result returned by run will vary depending on whether optimization is used (a strategy was added with optstrategy):

    **All strategies added with addstrategy**

    result will be a list of the instances run during the backtesting

    **1 or more strategies were added with optstrategy**

    result will be a list of list. Each internal list will contain the strategies after each optimization run
``