Backtrader Community
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Live trading multiple strategies, each one on a separate data feed

    General Code/Help
    9
    15
    2384
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • vladisld
      vladisld last edited by

      I'm currently testing my strategy on a IB paper account. The scenario is that multiple instances of the same strategy should be run, each one running on its own data feed (instrument) using instrument specific parameters.

      Testing a single strategy on a single data feed works flawlessly - but once multiple strategies are added - strange things start to happen.

      The suspicious error I'm getting from IB broker:

      <error id=16777228, errorCode=420, errorMsg=Invalid Real-time Query:Historical data request pacing violation>

      Also no 'next' methods are called on any of the added strategies.

      Searching through the forum I found several example using multiple strategies on a single data feed or a single strategy on multiple data feeds. However it seems that my case a little bit different.

      Here my sample setup code (which doesn't work):

      for cfg in cfgs:
        datakwargs = dict(
            timeframe=cfg['timeframe'],
            compression=cfg['compression'],
            historical=False,
            fromdate=None,
            rtbar=cfg['rtbar'],
            qcheck=args.qcheck,
            what=None,  # use the default for the STK
            backfill_start=False,
            backfill=False,
            latethrough=False,
            tz=args.timezone
        )

        data = IBDataFactory(dataname=cfg['dataname'], **datakwargs)
        cerebro.adddata(data)

        # Add a strategy
        strategy_kwargs = dict(
            data_idx = data._id,
            use_lotdb=True,
            lotdb_path='some value here',
            start_price=cfg['start_price'],
            percentage=cfg['percentage'],
            lot_value=cfg['lot_value'],
            oor_gap=cfg['trailing_percentage']
        )
        cerebro.addstrategy(TestStrategy, **strategy_kwargs)

    cerebro.run(exactbars=args.exactbars)

      Here the 'cfgs' are just read from the persistent storage and specify which instrument to trade (equity) and with what parameters.

      Each strategy gets the data index as a parameter (actually the index is data._id -1 to be exact)

      Is such a scenario supported? Anything I'm doing wrong ?

      Thanks.
      Vlad

      B 1 Reply Last reply Reply Quote 1
      • A
        ab_trader last edited by

        IB has limitations on amount of data user is able to get, and based on the error it seems that you are trying to get more data than you are allowed.

        Link - Pacing Violations for Small Bars

        • If my answer helped, hit reputation up arrow at lower right corner of the post.
        • Python Debugging With Pdb
        • New to python and bt - check this out
        1 Reply Last reply Reply Quote 2
        • vladisld
          vladisld last edited by

          Thanks for the reply.

          As you said, the error definitely says about historical data pacing violation, however upon adding data I deliberately specifying 'historical=False' attribute (see the code above) - only live data is needed, no other indicators are added - so no waiting period. In addition the test is running using only 4 equities (which should not cause the violation AFAIU).

          Still the question remains, why no 'next' method is called on any of the strategies, even when running during the market session hours given that:

          • there is a proper data subscription with IB
          • using the _debug option for IBStore/IBBroker the real time quotes are logged correctly (the following sample log is outside the market session but the similar log is seen during the session as well)
          <realtimeBar reqId=16777229, time=1572339465, open=48.69, high=48.69, low=48.69, close=48.69, volume=0, wap=48.69, count=0>
<realtimeBar reqId=16777235, time=1572339465, open=8.19, high=8.19, low=8.19, close=8.19, volume=0, wap=8.19, count=0>
<realtimeBar reqId=16777217, time=1572339470, open=33.8, high=33.8, low=33.8, close=33.8, volume=0, wap=33.8, count=0>
<realtimeBar reqId=16777219, time=1572339470, open=33.8, high=33.8, low=33.8, close=33.8, volume=0, wap=33.8, count=0>
<realtimeBar reqId=16777222, time=1572339470, open=33.8, high=33.8, low=33.8, close=33.8, volume=0, wap=33.8, count=0>
<realtimeBar reqId=16777226, time=1572339470, open=33.8, high=33.8, low=33.8, close=33.8, volume=0, wap=33.8, count=0>
<realtimeBar reqId=16777231, time=1572339470, open=33.8, high=33.8, low=33.8, close=33.8, volume=0, wap=33.8, count=0>
<realtimeBar reqId=16777229, time=1572339470, open=48.69, high=48.69, low=48.69, close=48.69, volume=0, wap=48.69, count=0>
<realtimeBar reqId=16777224, time=1572339470, open=3.49, high=3.49, low=3.49, close=3.49, volume=0, wap=3.49, count=0>
<realtimeBar reqId=16777233, time=1572339470, open=3.49, high=3.49, low=3.49, close=3.49, volume=0, wap=3.49, count=0>
<realtimeBar reqId=16777235, time=1572339470, open=8.19, high=8.19, low=8.19, close=8.19, volume=0, wap=8.19, count=0>

          Will try to debug it further of cause.

          Thanks for help
          Vlad

          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @vladisld last edited by

            @vladisld said in Live trading multiple strategies, each one on a separate data feed:

            The suspicious error I'm getting from IB broker:
            <error id=16777228, errorCode=420, errorMsg=Invalid Real-time Query:Historical data request pacing violation>

            Also no 'next' methods are called on any of the added strategies.

            Because the error breaks data continuity?

            @vladisld said in Live trading multiple strategies, each one on a separate data feed:

            till the question remains, why no 'next' method is called on any of the strategies, even when running during the market session hours given that:

            • What is the warm-up period?
            • How can anyone know why your code isn't being called when no code is shown?
            • Why don't you scale down to 1 data feed, have no warm-up period and see what happens?
            1 Reply Last reply Reply Quote 0
            • vladisld
              vladisld last edited by

              Thanks backtrader for a reply,

              First some answers to your follow up questions:

              @backtrader said in Live trading multiple strategies, each one on a separate data feed:

              What is the warm-up period?

              There is no warm-up period ( it least it should not be) - the setup code above just requests real-time bars for each data.

              @backtrader said in Live trading multiple strategies, each one on a separate data feed:

              How can anyone know why your code isn't being called when no code is shown?

              The strategy setup code is shown above - the strategy code itself if mostly standard (lets assume it just prints log)

              @backtrader said in Live trading multiple strategies, each one on a separate data feed:

              Why don't you scale down to 1 data feed, have no warm-up period and see what happens?

              As I've said at the very beginning - running just a single data feed with a single strategy works as expected.

              Now, debugging it little bit further I wonder if the following code in ibstore.py is working correctly in regard of pacing violation prevention:

                  def startdatas(self):
        # kickstrat datas, not returning until all of them have been done
        ts = list()
        for data in self.datas:
            t = threading.Thread(target=data.reqdata)
            t.start()
            ts.append(t)

        for t in ts:
            t.join()

              given multiple data feeds and multiple strategies - it is called multiple times, resulting in reqRealTimeBars request sent to IB for the same contract in very short amount of time. The following stack shows the call sequence:

              startdatas, ibstore.py:402
reconnect, ibstore.py:367
_st_start, ibdata.py:635
start, ibdata.py:402
_start, feed.py:203
runstrategies, cerebro.py:1210
run, cerebro.py:1127

              Thanks,
              Vlad

              1 Reply Last reply Reply Quote 0
              • vladisld
                vladisld last edited by

                It seems that even with a single strategy with multiple datas - the problem is still present. I've re-written my code to have a single strategy that wraps multiple strategies internally:

                class TestStrategyWrapper(bt.Strategy):
    params = (
        ('lotdb_path', ''),  # lotdb files root path
        ('cfgs', []),  # configurations to load
    )

    def __init__(self):
        self.strategies = []
        for cfg in self.p.cfgs:
            lotdb_path = os.path.join(self.p.lotdb_path, '{}.lot.csv'.format(cfg['dataname'].replace('-', '_')))
            # Add a strategy
            strategy_args = [cfg['data']]
            strategy_kwargs = dict(
                use_lotdb=True,
                lotdb_path=lotdb_path,
                start_price=cfg['start_price'],
                percentage=cfg['percentage'],
                lot_value=cfg['lot_value'],
                oor_gap=cfg['trailing_percentage']
            )
            self.strategies.append(TestStrategy(*strategy_args, **strategy_kwargs))

    def start(self):
        for strat in self.strategies:
            strat.start()

    def notify_store(self, msg, *args, **kwargs):
        self.logger.log(msg)

    def notify_order(self, order):
        for strat in self.strategies:
            if order.data is strat.data_feed:
                strat.notify_order(order)

    def notify_trade(self, trade):
        for strat in self.strategies:
            strat.notify_trade(trade)

    def next(self):
        for strat in self.strategies:
            strat.next()

                The configuration is stored in an external csv file and, just for example, contains the startup info for 3 strategies:

                dataname,timeframe,compression,rtbar,start_price,percentage,lot_value,trailing_percentage
FIT-STK-SMART-USD,3,5,1,0.0,0.95,10000,0.90
NOK-STK-SMART-USD,3,5,1,0.0,0.95,10000,0.90
MU-STK-SMART-USD,3,5,1,0.0,0.95,10000,0.90

                Running the above setup still produces 3 sets of REQ_REAL_TIME_BARS requests (code 50) in the TWS logs:

                0:50:30:449 -> 15-1-DU1461635-
20:50:30:449 -> 9-1-8-
20:50:30:449 -> 4-2--1-2107-HMDS data farm connection is inactive but should be available upon demand.ushmds-
20:50:30:449 <- 62-
20:50:30:449 <- 49-1-
20:50:30:449 -> 49-1-1572807030-
20:50:30:449 <- 6-2-1-DU1461635-
20:50:30:465 -> 6-2-AccountCode-DU1461635--DU1461635-
20:50:30:465 -> 6-2-AccountOrGroup-DU1461635-BASE-DU1461635-
20:50:30:465 -> 6-2-AccountOrGroup-DU1461635-USD-DU1461635-
20:50:30:465 -> 6-2-AccountReady-true--DU1461635-
20:50:30:465 -> 6-2-AccountType-INDIVIDUAL--DU1461635-
20:50:30:465 -> 6-2-AccruedCash-1142.53-BASE-DU1461635-
20:50:30:465 -> 6-2-AccruedCash-1142.53-USD-DU1461635-
20:50:30:465 -> 6-2-AccruedCash-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-AccruedCash-S-1142.53-USD-DU1461635-
20:50:30:465 -> 6-2-AccruedDividend-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-AccruedDividend-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-AccruedDividend-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-AvailableFunds-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-AvailableFunds-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-AvailableFunds-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-Billable-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Billable-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Billable-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-BuyingPower-4010595.98-USD-DU1461635-
20:50:30:465 -> 6-2-CashBalance-987870.78-BASE-DU1461635-
20:50:30:465 -> 6-2-CashBalance-987870.78-USD-DU1461635-
20:50:30:465 -> 6-2-CorporateBondValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-CorporateBondValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Currency-BASE-BASE-DU1461635-
20:50:30:465 -> 6-2-Currency-USD-USD-DU1461635-
20:50:30:465 -> 6-2-Cushion-0.991783--DU1461635-
20:50:30:465 -> 6-2-DayTradesRemaining--1--DU1461635-
20:50:30:465 -> 6-2-DayTradesRemainingT+1--1--DU1461635-
20:50:30:465 -> 6-2-DayTradesRemainingT+2--1--DU1461635-
20:50:30:465 -> 6-2-DayTradesRemainingT+3--1--DU1461635-
20:50:30:465 -> 6-2-DayTradesRemainingT+4--1--DU1461635-
20:50:30:465 -> 6-2-EquityWithLoanValue-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-EquityWithLoanValue-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-EquityWithLoanValue-S-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-ExcessLiquidity-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-ExcessLiquidity-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-ExcessLiquidity-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-ExchangeRate-1.00-BASE-DU1461635-
20:50:30:465 -> 6-2-ExchangeRate-1.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullAvailableFunds-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullAvailableFunds-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullAvailableFunds-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullExcessLiquidity-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullExcessLiquidity-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullExcessLiquidity-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullInitMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-FullInitMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullInitMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-FullMaintMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-FullMaintMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FullMaintMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-FundValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-FundValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FutureOptionValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-FutureOptionValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FuturesPNL-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-FuturesPNL-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-FxCashBalance-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-FxCashBalance-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-GrossPositionValue-21942.76-USD-DU1461635-
20:50:30:465 -> 6-2-GrossPositionValue-S-21942.76-USD-DU1461635-
20:50:30:465 -> 6-2-Guarantee-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Guarantee-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Guarantee-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-IndianStockHaircut-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-IndianStockHaircut-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-IndianStockHaircut-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-InitMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-InitMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-InitMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-IssuerOptionValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-IssuerOptionValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-Leverage-S-0.02--DU1461635-
20:50:30:465 -> 6-2-LookAheadAvailableFunds-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadAvailableFunds-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadAvailableFunds-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadExcessLiquidity-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadExcessLiquidity-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadExcessLiquidity-S-1002649.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadInitMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadInitMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadInitMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadMaintMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadMaintMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadMaintMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-LookAheadNextChange-0--DU1461635-
20:50:30:465 -> 6-2-MaintMarginReq-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-MaintMarginReq-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-MaintMarginReq-S-8307.08-USD-DU1461635-
20:50:30:465 -> 6-2-MoneyMarketFundValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-MoneyMarketFundValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-MutualFundValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-MutualFundValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-NLVAndMarginInReview-false--DU1461635-
20:50:30:465 -> 6-2-NetDividend-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-NetDividend-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-NetLiquidation-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-NetLiquidation-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-NetLiquidation-S-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-NetLiquidationByCurrency-1010956.0701-BASE-DU1461635-
20:50:30:465 -> 6-2-NetLiquidationByCurrency-1010956.0701-USD-DU1461635-
20:50:30:465 -> 6-2-NetLiquidationUncertainty-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-OptionMarketValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-OptionMarketValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PASharesValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PASharesValue-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PASharesValue-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationExcess-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationExcess-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationExcess-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationMargin-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationMargin-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-PostExpirationMargin-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-RealCurrency-BASE-BASE-DU1461635-
20:50:30:465 -> 6-2-RealCurrency-USD-USD-DU1461635-
20:50:30:465 -> 6-2-RealizedPnL-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-RealizedPnL-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-RegTEquity-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-RegTEquity-S-1010956.07-USD-DU1461635-
20:50:30:465 -> 6-2-RegTMargin-10971.38-USD-DU1461635-
20:50:30:465 -> 6-2-RegTMargin-S-10971.38-USD-DU1461635-
20:50:30:465 -> 6-2-SMA-999984.69-USD-DU1461635-
20:50:30:465 -> 6-2-SMA-S-999984.69-USD-DU1461635-
20:50:30:465 -> 6-2-SegmentTitle-C-US Commodities--DU1461635-
20:50:30:465 -> 6-2-SegmentTitle-S-US Securities--DU1461635-
20:50:30:465 -> 6-2-StockMarketValue-21942.76-BASE-DU1461635-
20:50:30:465 -> 6-2-StockMarketValue-21942.76-USD-DU1461635-
20:50:30:465 -> 6-2-TBillValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-TBillValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TBondValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-TBondValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TotalCashBalance-987870.78-BASE-DU1461635-
20:50:30:465 -> 6-2-TotalCashBalance-987870.78-USD-DU1461635-
20:50:30:465 -> 6-2-TotalCashValue-987870.78-USD-DU1461635-
20:50:30:465 -> 6-2-TotalCashValue-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TotalCashValue-S-987870.78-USD-DU1461635-
20:50:30:465 -> 6-2-TotalDebitCardPendingCharges-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TotalDebitCardPendingCharges-C-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TotalDebitCardPendingCharges-S-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-TradingType-S-STKNOPT--DU1461635-
20:50:30:465 -> 6-2-UnrealizedPnL-1912.05-BASE-DU1461635-
20:50:30:465 -> 6-2-UnrealizedPnL-1912.05-USD-DU1461635-
20:50:30:465 -> 6-2-WarrantValue-0.00-BASE-DU1461635-
20:50:30:465 -> 6-2-WarrantValue-0.00-USD-DU1461635-
20:50:30:465 -> 6-2-WhatIfPMEnabled-true--DU1461635-
20:50:30:465 -> 7-8-197066570-FIT-STK--0-0--NYSE-USD-FIT-FIT-1694-7.0999999-12027.4-5.915-2007.39-0.0-DU1461635-
20:50:30:465 -> 8-1-12:36-
20:50:30:465 -> 7-8-661513-NOK-STK--0-0--NYSE-USD-NOK-NOK-2724-3.6400001-9915.36-3.675--95.34-0.0-DU1461635-
20:50:30:465 -> 8-1-12:36-
20:50:30:465 -> 8-1-20:48-
20:50:30:465 -> 54-1-DU1461635-
20:50:30:512 <- 9-7-16777216-0-FIT-STK--0---SMART-USD---0---
20:50:30:607 -> 10-8-16777216-FIT-STK--0--SMART-USD-FIT-FIT-FIT-197066570-0.01--ACTIVETIM,ADJUST,ALERT,ALGO,ALLOC,AVGCOST,BASKET,BENCHPX,COND,CONDORDER,DARKONLY,DARKPOLL,DAY,DEACT,DEACTDIS,DEACTEOD,DIS,GAT,GTC,GTD,GTT,HID,IBKRATS,ICE,IMB,IOC,LIT,LMT,LOC,MIT,MKT,MOC,MTL,NGCOMB,NODARK,NONALGO,OCA,OPG,OPGREROUT,PEGBENCH,POSTONLY,PREOPGRTH,REL,RPI,RTH,RTHIGNOPG,SCALE,SCALEODD,SCALERST,SMARTSTG,SNAPMID,SNAPMKT,SNAPREL,STP,STPLMT,SWEEP,TRAIL,TRAILLIT,TRAILLMT,TRAILMIT,WHATIF-SMART,AMEX,NYSE,CBOE,PHLX,ISE,CHX,ARCA,ISLAND,DRCTEDGE,BEX,BATS,EDGEA,CSFBALGO,JEFFALGO,BYX,IEX,EDGX,FOXRIVER,TPLUS1,NYSENAT,PSX-1-0-FITBIT INC - A-NYSE--Industrial-Electronics-Electronic Measur Instr-EST5EDT-20191103:CLOSED;20191104:0400-20191104:2000;20191105:0400-20191105:2000;20191106:0400-20191106:2000;20191107:0400-20191107:2000;20191108:0400-20191108:2000;20191109:CLOSED;20191110:CLOSED;20191111:0400-20191111:2000;20191112:0400-20191112:2000;20191113:0400-20191113:2000;20191114:0400-20191114:2000;20191115:0400-20191115:2000;20191116:CLOSED;20191117:CLOSED;20191118:0400-20191118:2000;20191119:0400-20191119:2000;20191120:0400-20191120:2000;20191121:0400-20191121:2000;20191122:0400-20191122:2000;20191123:CLOSED;20191124:CLOSED;20191125:0400-20191125:2000;20191126:0400-20191126:2000;20191127:0400-20191127:2000;20191128:0400-20191128:2000;20191129:0400-20191129:2000;20191130:CLOSED;20191201:CLOSED;20191202:0400-20191202:2000;20191203:0400-20191203:2000;20191204:0400-20191204:2000;20191205:0400-20191205:2000;20191206:0400-20191206:2000;20191207:CLOSED-20191103:CLOSED;20191104:0930-20191104:1600;20191105:0930-20191105:1600;20191106:0930-20191106:1600;20191107:0930-20191107:1600;20191108:0930-20191108:1600;20191109:CLOSED;20191110:CLOSED;20191111:0930-20191111:1600;20191112:0930-20191112:1600;20191113:0930-20191113:1600;20191114:0930-20191114:1600;20191115:0930-20191115:1600;20191116:CLOSED;20191117:CLOSED;20191118:0930-20191118:1600;20191119:0930-20191119:1600;20191120:0930-20191120:1600;20191121:0930-20191121:1600;20191122:0930-20191122:1600;20191123:CLOSED;20191124:CLOSED;20191125:0930-20191125:1600;20191126:0930-20191126:1600;20191127:0930-20191127:1600;20191128:0930-20191128:1600;20191129:0930-20191129:1600;20191130:CLOSED;20191201:CLOSED;20191202:0930-20191202:1600;20191203:0930-20191203:1600;20191204:0930-20191204:1600;20191205:0930-20191205:1600;20191206:0930-20191206:1600;20191207:CLOSED---0-
20:50:30:607 -> 52-1-16777216-
20:50:30:716 <- 50-2-16777217-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:30:732 <- 9-7-16777218-0-NOK-STK--0---SMART-USD---0---
20:50:30:872 -> 10-8-16777218-NOK-STK--0--SMART-USD-NOK-NOK-NOK-661513-0.01--ACTIVETIM,ADJUST,ALERT,ALGO,ALLOC,AVGCOST,BASKET,BENCHPX,COND,CONDORDER,DARKONLY,DARKPOLL,DAY,DEACT,DEACTDIS,DEACTEOD,DIS,GAT,GTC,GTD,GTT,HID,IBKRATS,ICE,IMB,IOC,LIT,LMT,LOC,MIT,MKT,MOC,MTL,NGCOMB,NODARK,NONALGO,OCA,OPG,OPGREROUT,PEGBENCH,POSTONLY,PREOPGRTH,REL,RPI,RTH,RTHIGNOPG,SCALE,SCALEODD,SCALERST,SMARTSTG,SNAPMID,SNAPMKT,SNAPREL,STP,STPLMT,SWEEP,TRAIL,TRAILLIT,TRAILLMT,TRAILMIT,WHATIF-SMART,AMEX,NYSE,CBOE,PHLX,ISE,CHX,ARCA,ISLAND,DRCTEDGE,BEX,BATS,EDGEA,CSFBALGO,JEFFALGO,BYX,IEX,EDGX,FOXRIVER,NYSENAT,PSX-1-0-NOKIA CORP-SPON ADR-NYSE--Communications-Telecommunications-Wireless Equipment-EST5EDT-20191103:CLOSED;20191104:0400-20191104:2000;20191105:0400-20191105:2000;20191106:0400-20191106:2000;20191107:0400-20191107:2000;20191108:0400-20191108:2000;20191109:CLOSED;20191110:CLOSED;20191111:0400-20191111:2000;20191112:0400-20191112:2000;20191113:0400-20191113:2000;20191114:0400-20191114:2000;20191115:0400-20191115:2000;20191116:CLOSED;20191117:CLOSED;20191118:0400-20191118:2000;20191119:0400-20191119:2000;20191120:0400-20191120:2000;20191121:0400-20191121:2000;20191122:0400-20191122:2000;20191123:CLOSED;20191124:CLOSED;20191125:0400-20191125:2000;20191126:0400-20191126:2000;20191127:0400-20191127:2000;20191128:0400-20191128:2000;20191129:0400-20191129:2000;20191130:CLOSED;20191201:CLOSED;20191202:0400-20191202:2000;20191203:0400-20191203:2000;20191204:0400-20191204:2000;20191205:0400-20191205:2000;20191206:0400-20191206:2000;20191207:CLOSED-20191103:CLOSED;20191104:0930-20191104:1600;20191105:0930-20191105:1600;20191106:0930-20191106:1600;20191107:0930-20191107:1600;20191108:0930-20191108:1600;20191109:CLOSED;20191110:CLOSED;20191111:0930-20191111:1600;20191112:0930-20191112:1600;20191113:0930-20191113:1600;20191114:0930-20191114:1600;20191115:0930-20191115:1600;20191116:CLOSED;20191117:CLOSED;20191118:0930-20191118:1600;20191119:0930-20191119:1600;20191120:0930-20191120:1600;20191121:0930-20191121:1600;20191122:0930-20191122:1600;20191123:CLOSED;20191124:CLOSED;20191125:0930-20191125:1600;20191126:0930-20191126:1600;20191127:0930-20191127:1600;20191128:0930-20191128:1600;20191129:0930-20191129:1600;20191130:CLOSED;20191201:CLOSED;20191202:0930-20191202:1600;20191203:0930-20191203:1600;20191204:0930-20191204:1600;20191205:0930-20191205:1600;20191206:0930-20191206:1600;20191207:CLOSED---0-
20:50:30:872 -> 52-1-16777218-
20:50:30:888 <- 50-2-16777219-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:30:888 <- 50-2-16777220-661513-NOK-STK--0.0---SMART-NYSE-USD-NOK-NOK-5-TRADES-0-
20:50:30:888 <- 9-7-16777221-0-MU-STK--0---SMART-USD---0---
20:50:31:107 -> 10-8-16777221-MU-STK--0--SMART-USD-MU-NMS-NMS-9939-0.01--ACTIVETIM,ADJUST,ALERT,ALGO,ALLOC,AVGCOST,BASKET,BENCHPX,COND,CONDORDER,DARKONLY,DARKPOLL,DAY,DEACT,DEACTDIS,DEACTEOD,DIS,GAT,GTC,GTD,GTT,HID,IBKRATS,ICE,IMB,IOC,LIT,LMT,LOC,MIT,MKT,MOC,MTL,NGCOMB,NODARK,NONALGO,OCA,OPG,OPGREROUT,PEGBENCH,POSTONLY,PREOPGRTH,REL,RPI,RTH,SCALE,SCALEODD,SCALERST,SMARTSTG,SNAPMID,SNAPMKT,SNAPREL,STP,STPLMT,SWEEP,TRAIL,TRAILLIT,TRAILLMT,TRAILMIT,WHATIF-SMART,AMEX,NYSE,CBOE,PHLX,ISE,CHX,ARCA,ISLAND,DRCTEDGE,BEX,BATS,EDGEA,CSFBALGO,JEFFALGO,BYX,IEX,EDGX,FOXRIVER,TPLUS1,NYSENAT,PSX-1-0-MICRON TECHNOLOGY INC-NASDAQ--Technology-Semiconductors-Electronic Compo-Semicon-EST5EDT-20191103:CLOSED;20191104:0400-20191104:2000;20191105:0400-20191105:2000;20191106:0400-20191106:2000;20191107:0400-20191107:2000;20191108:0400-20191108:2000;20191109:CLOSED;20191110:CLOSED;20191111:0400-20191111:2000;20191112:0400-20191112:2000;20191113:0400-20191113:2000;20191114:0400-20191114:2000;20191115:0400-20191115:2000;20191116:CLOSED;20191117:CLOSED;20191118:0400-20191118:2000;20191119:0400-20191119:2000;20191120:0400-20191120:2000;20191121:0400-20191121:2000;20191122:0400-20191122:2000;20191123:CLOSED;20191124:CLOSED;20191125:0400-20191125:2000;20191126:0400-20191126:2000;20191127:0400-20191127:2000;20191128:0400-20191128:2000;20191129:0400-20191129:2000;20191130:CLOSED;20191201:CLOSED;20191202:0400-20191202:2000;20191203:0400-20191203:2000;20191204:0400-20191204:2000;20191205:0400-20191205:2000;20191206:0400-20191206:2000;20191207:CLOSED-20191103:CLOSED;20191104:0930-20191104:1600;20191105:0930-20191105:1600;20191106:0930-20191106:1600;20191107:0930-20191107:1600;20191108:0930-20191108:1600;20191109:CLOSED;20191110:CLOSED;20191111:0930-20191111:1600;20191112:0930-20191112:1600;20191113:0930-20191113:1600;20191114:0930-20191114:1600;20191115:0930-20191115:1600;20191116:CLOSED;20191117:CLOSED;20191118:0930-20191118:1600;20191119:0930-20191119:1600;20191120:0930-20191120:1600;20191121:0930-20191121:1600;20191122:0930-20191122:1600;20191123:CLOSED;20191124:CLOSED;20191125:0930-20191125:1600;20191126:0930-20191126:1600;20191127:0930-20191127:1600;20191128:0930-20191128:1600;20191129:0930-20191129:1600;20191130:CLOSED;20191201:CLOSED;20191202:0930-20191202:1600;20191203:0930-20191203:1600;20191204:0930-20191204:1600;20191205:0930-20191205:1600;20191206:0930-20191206:1600;20191207:CLOSED---0-
20:50:31:107 -> 52-1-16777221-
20:50:31:122 <- 50-2-16777222-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:31:122 <- 50-2-16777223-661513-NOK-STK--0.0---SMART-NYSE-USD-NOK-NOK-5-TRADES-0-
20:50:31:122 <- 50-2-16777224-9939-MU-STK--0.0---SMART-NASDAQ-USD-MU-NMS-5-TRADES-0-
20:50:31:876 -> 4-2--1-2106-HMDS data farm connection is OK:ushmds-
20:50:32:189 -> 4-2-16777219-420-Invalid Real-time Query:Historical data request pacing violation-
20:50:32:189 -> 4-2-16777222-420-Invalid Real-time Query:Historical data request pacing violation-
20:50:32:189 -> 4-2-16777223-420-Invalid Real-time Query:Historical data request pacing violation-

                You may clearly see this by searching for the request 50. Here the relevant log records from the above:

                20:50:30:716 <- 50-2-16777217-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:30:888 <- 50-2-16777219-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:30:888 <- 50-2-16777220-661513-NOK-STK--0.0---SMART-NYSE-USD-NOK-NOK-5-TRADES-0-
20:50:31:122 <- 50-2-16777222-197066570-FIT-STK--0.0---SMART-NYSE-USD-FIT-FIT-5-TRADES-0-
20:50:31:122 <- 50-2-16777223-661513-NOK-STK--0.0---SMART-NYSE-USD-NOK-NOK-5-TRADES-0-
20:50:31:122 <- 50-2-16777224-9939-MU-STK--0.0---SMART-NASDAQ-USD-MU-NMS-5-TRADES-0-
20:50:32:189 -> 4-2-16777219-420-Invalid Real-time Query:Historical data request pacing violation-
20:50:32:189 -> 4-2-16777222-420-Invalid Real-time Query:Historical data request pacing violation-
20:50:32:189 -> 4-2-16777223-420-Invalid Real-time Query:Historical data request pacing violation-

                The violation is clear, since the request for the real-time bars for the same symbol is sent within 15 seconds.

                AFAIS, the problem is that the IBStore::startdatas (which is responsible for sending the reqRealTimeBars for all the registered datas), is called once for each separate IBData object- resulting in multiple requests for each symbol to be sent through TWS API.

                So if we are requesting data for 3 symbols (FIT,NOK,MU for example) there will be 3 calls for startdatas:

                • First one for FIT - the request will only be sent for FIT symbol obviously
                • Second one for NOK - the requests will be sent both for FIT and NOK, since datas are registered now
                • Third one for MU - the requests will be sent for all of them once again (FIT,NOK,MU).

                Which is what we see in the logs.

                1 Reply Last reply Reply Quote 0
                • vladisld
                  vladisld last edited by

                  Thinking about fixing the way the data subscriptions logic is implemented, I see the following options:

                  1. Each time the data subscription request is sent, issue the cancel request first (even if there were no previous subscription request sent )

                  Pros:

                  • pretty easy to implement, right in the IBData::reqdata probably

                  Cons:

                  • not sure if cancelling and re-subscribe for the same contract will not still result in pacing violation
                  • doesn't address the underlying issue that allowed the multiple subscription requests to be sent for the same data in the first place
                  • multiple cancellation/re-subscribing request will still be sent under current design.
                  1. Maintaining the state in IBData class, indicating that subscription request was already sent. Reference this state inside the IBStore::startdatas, and only issue the subscription request for IBData objects with the above state cleared. Reset a subscription state once the connection error was reported ( or any error for that matter )

                  Pros:

                  • Ensures that only a single subscription request will be sent for each data, thus eliminating the cases for pacing violation

                  Cons:

                  • more complicated solution, involving maintaining an additional state inside the IBData objects
                  • not clear whether a separate state need to be maintained for different kinds of subscriptions (historical, real time bars, delayed market data )

                  Any other suggestions/comments? Will appreciate any feedback...

                  1 Reply Last reply Reply Quote 0
                  • vladisld
                    vladisld last edited by

                    Ended up with just a small change that 'fixes' the issue for me (probably not the best fix of cause):

                    diff --git a/backtrader/feeds/ibdata.py b/backtrader/feeds/ibdata.py
index ab3c184..84cff62 100644
--- a/backtrader/feeds/ibdata.py
+++ b/backtrader/feeds/ibdata.py
@@ -408,7 +408,7 @@ class IBData(with_metaclass(MetaIBData, DataBase)):

     def reqdata(self):
         '''request real-time data. checks cash vs non-cash) and param useRT'''
-        if self.contract is None:
+        if self.contract is None or self._state == self._ST_LIVE:
             return

         if self._usertvol:

                    Thanks

                    1 Reply Last reply Reply Quote 0
                    • vladisld
                      vladisld last edited by

                      After further testing, the above 'fix' appears not to be good enough. The problem is with connection reset events:

                      <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>

                      In case of connection resets, the state of the data feeds seems not to be reset and stays in _ST_LIVE state.

                      Here the code from ibdata.py IBData class:

                          def _load(self):
            # (vladisld) ... code ommitted for clarity
            while True:
                if self._state == self._ST_LIVE:
                    # (vladisld) ... code ommitted for clarity
                    elif msg == -1100:  # conn broken
                        # Tell to wait for a message to do a backfill
                        # self._state = self._ST_DISCONN
                        self._statelivereconn = self.p.backfill
                        continue

                      Original code (without fix) would still reconnect the data feeds in that state. However with my 'fix' the reconnect will not work anymore.

                      Will update if a better fix is found. Any suggestion is appreciated of cause.

                      Thanks
                      Vlad

                      S 1 Reply Last reply Reply Quote 0
                      • B
                        bigdavediode last edited by

                        Hi Vlad,

                        Just a thought: when it runs for the first time under live data self.qlive is never set and self_storedmsg is an empty dict so in ibdata:

                            def haslivedata(self):
        return bool(self._storedmsg or self.qlive)

                        Returns False. And if it doesn't have live data (which is what it would be reporting) a calling routine wouldn't bother reconnecting.

                        Perhaps you could adapt my throttle delay routine below to remove the atrocious sleep line and instead return a boolean -- delay data requests and return out of req_data IF _ST_LIVE is true and throt_delay == True. This way the reqdata will test for reconnects but not until the queue delay score is low enough to flush the queue to avoid overloading the IB request queue:

                        Something like (untested)

                        reqcount = 0            #throttle limits
lastreqtime = time.time()
reqlmt = 50


@staticmethod
def _throt_delay():
    global reqcount, lastreqtime
    throttime = 1 / reqlmt  # throttle delay in fraction of seconds
    reqcount += 1  # add one throt period
    reqcount -= int((time.time() - lastreqtime) / throttime)  # Decrement count by time passed:
    reqcount = max(reqcount, 0)  # keep it above zero
    lastreqtime = time.time()
    # do not req data for this period
    return  (reqcount >= 4 * reqlmt - 1)     #return true if time to pause requests when live

                        and change reqdata to:

                        if self.contract is None or (self._state == self._ST_LIVE and _throt_delay())

                        By combining the two ideas you get rid of the sleep and the reconnect issues with never req'ing data (reconnects) when live.Note that you would also have to alter the historical data request routine. I haven't tested this.

                        Good luck,
                        B.

                        Atul Patel 1 Reply Last reply Reply Quote 0
                        • Atul Patel
                          Atul Patel @bigdavediode last edited by

                          @bigdavediode @vladisld I find the IB data limits kind of archane in today's world. I understand why they do it, but wish the thresholds were greater.

                          I'm planning to get data from IEX(coding the feed today) and compare its quality to the IB data. My trading is mostly in 1m time frames so cost effective.

                          Maor Kima M 2 Replies Last reply Reply Quote 0
                          • Maor Kima
                            Maor Kima @Atul Patel last edited by

                            @Atul-Patel i'm on the same boat with you, using 1 minute data - sick and tired from IB data performance. searching for another vendor.
                            how does IEX Cloud works for you till now? if you happy with it - would you be willing to share the live data feed for it?

                            thanks,
                            Maor.

                            1 Reply Last reply Reply Quote 0
                            • M
                              mv @Atul Patel last edited by

                              @Atul-Patel were you able to get this done for IEX data? How did it compare to IB data feed? Did you find any issues with BT resampling for indicators as values are off like mentioned in my others posts.

                              Any updates on alternate and reliable data feed for Backtrader would be very much appreciated

                              Thanks

                              1 Reply Last reply Reply Quote 1
                              • S
                                shspersonalwork @vladisld last edited by

                                @vladisld Were you able to resolve this issue? I have one strategy and two stocks. I don't see any hang or anything, but next() only runs for one of the stocks.

                                hghhgghdf dfdf 1 Reply Last reply Reply Quote 0
                                • hghhgghdf dfdf
                                  hghhgghdf dfdf @shspersonalwork last edited by

                                  @shspersonalwork

                                  order the strategy to create orders for each data in self.datas.

                                  1 Reply Last reply Reply Quote 0
                                  • 1 / 1
                                  • First post
                                    Last post
                                  Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors