IB Tick deliver latency
-
I have been running the code in the link below to see if there is any remarkable difference between system clock and IB ticker data's datetime. Indeed there is a remarkable difference and I am trying to figure out what might be causing such a delay in IB ticker's delivery to the system.
Code:
https://gist.github.com/eduedix/7c985bf2d0c71f159b521733f6dc503bLogs:
https://gist.github.com/eduedix/009246c3a8a91a1619a5adac4e5a967b
Best
-
Would you care to elaborate what the problem actually is?
-
Sure, the datetime of IB data ticks are not in sync with system clock.
If the system is way behind the data delivery time, then it will be late to make new orders.
I thought
self.datas[i].datetime.datetime()
would refer to the datetime of the delivery for a given data ( in this case tick data added viaadddata
). But as seen in the logs, this is not the case, there is always a few seconds of diff between system and data datetimes. I will also paste the logs here with more instruments for you to realize the difference in system and bt's data clocks.log with more instruments here:
https://gist.github.com/eduedix/b3b42f2e6f3c0d6654d48c3b05694448 -
As long as you keep on posting logs and don't actually let us know where the problem is ...
-
Let's check this bit from the log above:
2018-08-28 12:13:34.922744 AUD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.201050 0.95024 *** AUD.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.331496 1.09546 *** AUD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.527748 0.73544 *** CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 --- CAD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.669510 85.936 *** CHF.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.735663 113.805 *** EUR.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.291379 1.59258 *** EUR.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.347569 1.51333 *** EUR.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.311837 1.14277 *** EUR.GBP-CASH-IDEALPRO_tick 2018-08-28 12:13:09.305364 0.90687 *** EUR.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.375999 130.054 *** EUR.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.292429 1.17127 *** GBP.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.155154 1.75601 *** GBP.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.286686 1.66871 *** GBP.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.647251 1.26006 *** GBP.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.609080 143.406 *** GBP.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.719570 1.92374 *** GBP.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.343485 1.29151 *** NZD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.085550 0.86737 *** NZD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.648840 0.67132 *** USD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.348243 1.29209 *** USD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.272795 0.97567 *** USD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:07.832831 111.039 ***
system clock:
2018-08-28 12:13:34.922744
, latest delivered data's datetime:CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 ---
.The data arrived at 12:13:09, but before 12:13:34 the system will not be able to make an order which depends on the data from 12:13:09 . In other terms, the system is way too late to make a new order.
-
utcnow
uses your system clock. The timestamps from IB are from IB, which obviously uses the system clock of IB (no, no the system clock of the system on which the gateway runs, but from the server) -
@backtrader in that case I would expect the difference to be between 2 to be almost constant throughout the runtime. This is definitely not the case.
-
The differences in your log are sometimes smaller and sometimes larger.
But you seem to suggest that it always grow, which isn't the case. Some ticks seem to catch up.
-
@backtrader indeed. the difference is not constant. It ranges roughly between 2 and 20 seconds. And I am trying to figure out why there is such a lag anyway.
I am sure this is not because of the internet connection, because I have been running the same test with
ib_insync
( no framework upon) and am experiencing a lag between 0.1 and 0.6 seconds. -
@backtrader So, is there anything else I can do to resolve this issue?
What mostly happens is, the time difference increases up to some level (20-21 seconds) and then just like a spike the system collects a big chunk of data, which leads to reduced time difference suddenly again. This is happening in a cycle.
Is this the expected behaviour when backtrader is used with IB? I was hoping to get at most 1 second of latency.
Best
-
I have done many tests to figure out what might be causing such delays. backtrader itself can handle multiple contracts well for backtesting, so the issue was not backtrader itself.
I have tried
reqMktData
on multiple contracts (20-30) simultaneously with ib_insync and IbPy2 and it turned out that IbPy2 was lagging way behind and causing the delays. (lots of more than 5 seconds delays for IbPy2 and way less amount of 1-2 seconds and less delays for ib_insync).I will post here the test codes for you guys if you want to.
Best
-
In addition, when a network problem or similar occurs, with stalled IbPy2 and multiple contracts, it will not be able to notice the network problem right away like ib_insync.
-
contacted IB live chat and it was suggested to use ib_insync instead of ibapi or IBpy to avoid lags.