3. IB Tick deliver latency
For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

IB Tick deliver latency


  • I have been running the code in the link below to see if there is any remarkable difference between system clock and IB ticker data's datetime. Indeed there is a remarkable difference and I am trying to figure out what might be causing such a delay in IB ticker's delivery to the system.

    Code:
    https://gist.github.com/eduedix/7c985bf2d0c71f159b521733f6dc503b

    Logs:

    https://gist.github.com/eduedix/009246c3a8a91a1619a5adac4e5a967b

    Best

    1
  • B
    administrators

    Would you care to elaborate what the problem actually is?

    0

  • Sure, the datetime of IB data ticks are not in sync with system clock.

    If the system is way behind the data delivery time, then it will be late to make new orders.

    I thought self.datas[i].datetime.datetime() would refer to the datetime of the delivery for a given data ( in this case tick data added via adddata ). But as seen in the logs, this is not the case, there is always a few seconds of diff between system and data datetimes. I will also paste the logs here with more instruments for you to realize the difference in system and bt's data clocks.

    log with more instruments here:
    https://gist.github.com/eduedix/b3b42f2e6f3c0d6654d48c3b05694448

    0
  • B
    administrators

    As long as you keep on posting logs and don't actually let us know where the problem is ...

    0 1 Reply

  • Let's check this bit from the log above:

    2018-08-28 12:13:34.922744
AUD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.201050 0.95024 ***
AUD.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.331496 1.09546 ***
AUD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.527748 0.73544 ***
CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 ---
CAD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.669510 85.936 ***
CHF.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.735663 113.805 ***
EUR.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.291379 1.59258 ***
EUR.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.347569 1.51333 ***
EUR.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.311837 1.14277 ***
EUR.GBP-CASH-IDEALPRO_tick 2018-08-28 12:13:09.305364 0.90687 ***
EUR.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.375999 130.054 ***
EUR.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.292429 1.17127 ***
GBP.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.155154 1.75601 ***
GBP.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.286686 1.66871 ***
GBP.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.647251 1.26006 ***
GBP.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.609080 143.406 ***
GBP.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.719570 1.92374 ***
GBP.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.343485 1.29151 ***
NZD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.085550 0.86737 ***
NZD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.648840 0.67132 ***
USD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.348243 1.29209 ***
USD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.272795 0.97567 ***
USD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:07.832831 111.039 ***

    system clock: 2018-08-28 12:13:34.922744, latest delivered data's datetime: CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 ---.

    The data arrived at 12:13:09, but before 12:13:34 the system will not be able to make an order which depends on the data from 12:13:09 . In other terms, the system is way too late to make a new order.

    1
  • B
    administrators

    utcnow uses your system clock. The timestamps from IB are from IB, which obviously uses the system clock of IB (no, no the system clock of the system on which the gateway runs, but from the server)

    0 1 Reply

  • @backtrader in that case I would expect the difference to be between 2 to be almost constant throughout the runtime. This is definitely not the case.

    0
  • B
    administrators

    The differences in your log are sometimes smaller and sometimes larger.

    But you seem to suggest that it always grow, which isn't the case. Some ticks seem to catch up.

    0 1 Reply

  • @backtrader indeed. the difference is not constant. It ranges roughly between 2 and 20 seconds. And I am trying to figure out why there is such a lag anyway.

    I am sure this is not because of the internet connection, because I have been running the same test with ib_insync ( no framework upon) and am experiencing a lag between 0.1 and 0.6 seconds.

    1

  • @backtrader So, is there anything else I can do to resolve this issue?

    What mostly happens is, the time difference increases up to some level (20-21 seconds) and then just like a spike the system collects a big chunk of data, which leads to reduced time difference suddenly again. This is happening in a cycle.

    Is this the expected behaviour when backtrader is used with IB? I was hoping to get at most 1 second of latency.

    Best

    0

  • @backtrader

    I have done many tests to figure out what might be causing such delays. backtrader itself can handle multiple contracts well for backtesting, so the issue was not backtrader itself.

    I have tried reqMktData on multiple contracts (20-30) simultaneously with ib_insync and IbPy2 and it turned out that IbPy2 was lagging way behind and causing the delays. (lots of more than 5 seconds delays for IbPy2 and way less amount of 1-2 seconds and less delays for ib_insync).

    I will post here the test codes for you guys if you want to.

    Best

    0

  • In addition, when a network problem or similar occurs, with stalled IbPy2 and multiple contracts, it will not be able to notice the network problem right away like ib_insync.

    0

  • contacted IB live chat and it was suggested to use ib_insync instead of ibapi or IBpy to avoid lags.

    1