contacted IB live chat and it was suggested to use ib_insync instead of ibapi or IBpy to avoid lags.

@eduedix self.datas[0].datetime.datetime(-1)

I have been running the code in the link below to see if there is any remarkable difference between system clock and IB ticker data's datetime. Indeed there is a remarkable difference and I am trying to figure out what might be causing such a delay in IB ticker's delivery to the system.

Code:
https://gist.github.com/eduedix/7c985bf2d0c71f159b521733f6dc503b

Logs:

https://gist.github.com/eduedix/009246c3a8a91a1619a5adac4e5a967b

Best

Let's check this bit from the log above:

2018-08-28 12:13:34.922744
AUD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.201050 0.95024 ***
AUD.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.331496 1.09546 ***
AUD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.527748 0.73544 ***
CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 ---
CAD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.669510 85.936 ***
CHF.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.735663 113.805 ***
EUR.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.291379 1.59258 ***
EUR.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.347569 1.51333 ***
EUR.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.311837 1.14277 ***
EUR.GBP-CASH-IDEALPRO_tick 2018-08-28 12:13:09.305364 0.90687 ***
EUR.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.375999 130.054 ***
EUR.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.292429 1.17127 ***
GBP.AUD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.155154 1.75601 ***
GBP.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.286686 1.66871 ***
GBP.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:08.647251 1.26006 ***
GBP.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:08.609080 143.406 ***
GBP.NZD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.719570 1.92374 ***
GBP.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.343485 1.29151 ***
NZD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.085550 0.86737 ***
NZD.USD-CASH-IDEALPRO_tick 2018-08-28 12:13:08.648840 0.67132 ***
USD.CAD-CASH-IDEALPRO_tick 2018-08-28 12:13:09.348243 1.29209 ***
USD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.272795 0.97567 ***
USD.JPY-CASH-IDEALPRO_tick 2018-08-28 12:13:07.832831 111.039 ***

system clock: 2018-08-28 12:13:34.922744, latest delivered data's datetime: CAD.CHF-CASH-IDEALPRO_tick 2018-08-28 12:13:09.359146 0.7551 ---.

The data arrived at 12:13:09, but before 12:13:34 the system will not be able to make an order which depends on the data from 12:13:09 . In other terms, the system is way too late to make a new order.

@backtrader indeed. the difference is not constant. It ranges roughly between 2 and 20 seconds. And I am trying to figure out why there is such a lag anyway.

I am sure this is not because of the internet connection, because I have been running the same test with ib_insync ( no framework upon) and am experiencing a lag between 0.1 and 0.6 seconds.

@cwse in that case i would write my own data feed of the custom source for backtrader and use ib_insync to make orders. In my setup I am using ib_insync for paper/live trading and backtrader for backtesting only. I found backtrader not flexible enough for a live trading solution.

hey @vytautas-a , thanks for the contribution, i will test it and share my results here.

@backtrader I am already doing so, will also share here when I am finished. Though I am really wondering what your setup is like. ibtest was not able to solve connection issues for as it is.

this problem has been discussed many times here, people including me claim that such a problem exists while the author claims against. bt's ib integration is not in ideal condition for continuous automation.

Hello,

I am paper trading with IB and have trade history enabled. I want to find out in which case notify_trade doesn't obtain a new Trade instance after a closed one. Assuming I have a closed trade for a given contract, if I make a new order and not close the trade, should notify_trade obtain a new trade with new history or append to history?

Every once in a while I notice that trade.history[0] is does not belong to my new open order but the previous opening order of the closed price. An example of trade.history you can see below:

[TradeHistory([('status', AutoOrderedDict([('status', 1), ('dt', 736969.4541666667), ('barlen', 0), ('size', -107500), ('price', 1.15164), ('value', 123801.3), ('pnl', 0.0), ('pnlcomm', 0.0), ('tz', <UTC>)])), ('event', AutoOrderedDict([('order', <backtrader.brokers.ibbroker.IBOrder object at 0x113c50cc0>), ('size', -107500), ('price', 1.15164), ('commission', 2.476026)]))]),
TradeHistory([('status', AutoOrderedDict([('status', 2), ('dt', 736969.4736689815), ('barlen', 337), ('size', 0), ('price', 1.15164), ('value', 0.0), ('pnl', -95.67500000000618), ('pnlcomm', -100.62896600000617), ('tz', <UTC>)])), ('event', AutoOrderedDict([('order', <backtrader.brokers.ibbroker.IBOrder object at 0x114dbe710>), ('size', 107500), ('price', 1.15253), ('commission', 2.47794)]))]),
TradeHistory([('status', AutoOrderedDict([('status', 1), ('dt', 736969.4907986111), ('barlen', 0), ('size', 107500), ('price', 1.15262), ('value', 123906.65), ('pnl', -95.67500000000618), ('pnlcomm', -100.62896600000617), ('tz', <UTC>)])), ('event', AutoOrderedDict([('order', <backtrader.brokers.ibbroker.IBOrder object at 0x114c9cbe0>), ('size', 107500), ('price', 1.15262), ('commission', 2.4781330000000006)]))])]

@backtrader thank you for the suggestion. Though would that still work If I need to replay multiple timeframes?

I was thinking of resampling my low tf data dataframe with pandas to higher timeframe and use it for backfilling source. (if I got you right above).

If I have multiple replays, then I will do the pandas resampling multiple times, but I think concatenating and backfilling wouldn't be the right approach, because they have different compressions on bt PandasData level.

What would you suggest for multiple replays?

Hello,

I want to reduce the time required for backfilling.

I was wondering whether it is ever possible to backfill replayed data. Currently backfilling is done via executing the strategy on low time frame data and higher time framed replayed datas, which takes a bit of time. I just need to do backfilling to initialize higher time frame datas before executing my strategy live.

It would be easier for me if I was directly allowed to provide higher timeframe data myself instead of executing the strategy for backfilling.

Best

contacted IB live chat and it was suggested to use ib_insync instead of ibapi or IBpy to avoid lags.

In addition, when a network problem or similar occurs, with stalled IbPy2 and multiple contracts, it will not be able to notice the network problem right away like ib_insync.

@backtrader

I have done many tests to figure out what might be causing such delays. backtrader itself can handle multiple contracts well for backtesting, so the issue was not backtrader itself.

I have tried reqMktData on multiple contracts (20-30) simultaneously with ib_insync and IbPy2 and it turned out that IbPy2 was lagging way behind and causing the delays. (lots of more than 5 seconds delays for IbPy2 and way less amount of 1-2 seconds and less delays for ib_insync).

I will post here the test codes for you guys if you want to.

Best