Delayed never turn live with futures

Hello,
I'm using demo IB Account with Backtrader, connected to a live account with market data subscription.

When i call ibtest for some stock it works fine, but not for futures.

python ibtest.py --port 4002 --qcheck 2 --reconnect 3 --data0 AAPL-STK-SMART-USD --timeframe Seconds --compression 10

Output:

Server Version: 76
TWS Time at connection:20220119 20:01:04 CET
.........
***** DATA NOTIF: DELAYED
Data0, 0001, 738174.6673611111, 2022-01-19T11:01:00.000000, 169.09, 169.15, 169.06, 169.09, 805, 0, nan
........
Data0, 1086, 738174.7924383796, 2022-01-19T14:01:06.675997, 168.58, 168.58, 168.58, 168.58, 3, 0, 168.568
***** DATA NOTIF: LIVE
Data0, 1087, 738174.7924413426, 2022-01-19T14:01:06.932001, 168.6, 168.6, 168.6, 168.6, 21, 0, 168.574

For futures (I've tested also VIX, GOLD...). It never turn live!

python ibtest.py --port 4002 --qcheck 2 --reconnect 3 --data0 HG-FUT-NYMEX-USD-20220127 --timeframe Seconds --compression 10

Output:

Server Version: 76
TWS Time at connection:20220119 20:06:28 CET
Strategy Created
Timezone from ContractDetails: US/Eastern
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** STORE NOTIF: <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefil>
***** DATA NOTIF: DELAYED
***** STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ushmds>

Thank you ;)
@backtrader

EMR

@dotto95 I can confirm the issue. What is strange is that with an old version of TWS, it used to work but not anymore.
I am working on it to find a solution, don't know how much time it would take me.
I tried backtrader-insync but have other issue with it (price at -1).

EMR

@dotto95 " for some stock it works fine" could you please tell which one ? Tried with AAPL-STK-SMART-USD and this reproduce the issue and I would like to analyse which stock have the issue and which one haven't got it.

Anyway, I think this may be due to share fractionning (https://www.businesswire.com/news/home/20191125005453/en/Interactive-Brokers-Offering-Fractional-Share-Trading) : it seems that in IB (now ?) size and volume is sent in decimal format. This cause a value error at line 885 of ibstore, in the function tickstring, which receive RTVolume market data ticks. Forex have no size, so I guess that's why forex do not have this issue.

When I modify the class RTVolume in ibstore.py, and change the data type of some items in _fields : size and volume from int to decimal.Decimal, I am able to live stream future data. But there may be some side effects (what about orders, position ?) so more analysis & testing have to be done.

@emr said in Delayed never turn live with futures:

decimal.Decimal

I've tried to modify the class as you suggest.
The problem is the same. Today it's even worse. Not even AAPL can recover, it doesn't just connect to the history farm:

In the gateway log there are online prices:

***** STORE NOTIF: <contractDetails reqId=16777216, contractDetails=<ib.ext.ContractDetails.ContractDetails object at 0x7f62ac950b20>>
***** STORE NOTIF: <contractDetailsEnd reqId=16777216>
***** DATA NOTIF: DELAYED
***** STORE NOTIF: <marketDataType reqId=16777217, marketDataType=1>
***** STORE NOTIF: <tickPrice tickerId=16777217, field=1, price=-1.0, canAutoExecute=1>
***** STORE NOTIF: <tickSize tickerId=16777217, field=0, size=0>
***** STORE NOTIF: <tickPrice tickerId=16777217, field=2, price=-1.0, canAutoExecute=1>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=0>
***** STORE NOTIF: <tickPrice tickerId=16777217, field=9, price=164.51, canAutoExecute=0>
***** STORE NOTIF: <currentTime time=1642750562>

My question is: Why backtrader / store don't call historical Data Farm for futures and stocks?
If I call ibtest.py with --historical parameter it works like a charm.

EMR

@dotto95 Strange.
This is what I have :

Backtrader2 last version
TWS version 10.12.2i Jan 19
TWS settings : UTC/Universal, Europe, TWS API settings : "encode API messages" ASCII7, "send market data in lots for us stocks for dual" is ticked
Modified ibstore.py. lines 63 and 65 :

    _fields = [             
        ('price', float),   
        ('size', decimal.Dec
        ('datetime', _ts2dt)
        ('volume', decimal.D
        ('vwap', float),    
        ('single', bool)    
    ]

When I run ibtest.py from /backtrader/samples/ibtest, with :

py ibtest.py --port 7497 --data0 AAPL-STK-SMART-USD --timeframe Seconds --compression 10 --broker --historical
I have :

[...]
Data0, 0719, 738176.3983796296, 2022-01-21T04:33:40.000000, 163.71, 163.71, 163.71, 163.71, 0, 0, 163.73600000000002
Data0, 0720, 738176.3984953704, 2022-01-21T04:33:50.000000, 163.69, 163.69, 163.69, 163.69, 1, 0, 163.71200000000002
Data0, 0721, 738176.3986111111, 2022-01-21T04:34:00.000000, 163.69, 163.69, 163.69, 163.69, 0, 0, 163.702
***** DATA NOTIF: DISCONNECTED

Tried without historical, un 1 second (to be sure to avoid rtbar) :

py ibtest.py --port 7497 --data0 AAPL-STK-SMART-USD --timeframe Seconds --compression 1

[...]
Data0, 1799, 738176.4025115741, 2022-01-21T04:39:37.000000, 163.32, 163.32, 163.32, 163.32, 0, 0, 163.32
Data0, 1800, 738176.4025231481, 2022-01-21T04:39:38.000000, 163.32, 163.32, 163.32, 163.32, 0, 0, 163.32
***** DATA NOTIF: LIVE
Data0, 1801, 738176.4025377893, 2022-01-21T04:39:39.264996, 163.46, 163.46, 163.46, 163.46, 218.0000000000000000, 0, 163.348
Data0, 1802, 738176.4025563195, 2022-01-21T04:39:40.866003, 163.32, 163.32, 163.32, 163.32, 1.0000000000000000, 0, 163.348
Data0, 1803, 738176.4030939584, 2022-01-21T04:40:27.318004, 163.37, 163.37, 163.37, 163.37, 1.0000000000000000, 0, 163.358

With

py ibtest.py --port 7497 --data0 EUR.USD-CASH-IDEALPRO --timeframe Seconds --compression 10

Data0, 1080, 738176.4041666667, 2022-01-21T04:42:00.000000, 1.13412, 1.13415, 1.13411, 1.13415, -1, 0, 1.134154
Data0, 1081, 738176.4042824074, 2022-01-21T04:42:10.000000, 1.13415, 1.13421, 1.13415, 1.13421, -1, 0, 1.134162
***** DATA NOTIF: LIVE
Data0, 1082, 738176.4043882311, 2022-01-21T04:42:19.143167, 1.13421, 1.13421, 1.13421, 1.13421, 0, 0, 1.134172
Data0, 1083, 738176.4044105696, 2022-01-21T04:42:21.073211, 1.13419, 1.13419, 1.13419, 1.13419, 0, 0, 1.134176

The same with --historical :

Data0, 0718, 738176.4049768519, 2022-01-21T04:43:10.000000, 1.13402, 1.1341, 1.13398, 1.13408, -1, 0, 1.1340999999999999
Data0, 0719, 738176.4050925926, 2022-01-21T04:43:20.000000, 1.13408, 1.1341, 1.13404, 1.13407, -1, 0, 1.134066
Data0, 0720, 738176.4052083333, 2022-01-21T04:43:30.000000, 1.13407, 1.13407, 1.13393, 1.13393, -1, 0, 1.134018
Data0, 0721, 738176.4053240741, 2022-01-21T04:43:40.000000, 1.13393, 1.13393, 1.13387, 1.1339, -1, 0, 1.134

Tried
py ibtest.py --port 7497 --data0 NQ-202203-GLOBEX-USD-20 --timeframe Seconds --compression 10

Data0, 1080, 738176.4061342593, 2022-01-21T03:44:50.000000, 14765.75, 14766.75, 14765.75, 14766.5, 12, 0, 14765.95
Data0, 1081, 738176.40625, 2022-01-21T03:45:00.000000, 14766.25, 14766.75, 14766.25, 14766.5, 4, 0, 14765.75
***** DATA NOTIF: LIVE
Data0, 1082, 738176.4063021065, 2022-01-21T03:45:04.501997, 14767.75, 14767.75, 14767.75, 14767.75, 1.0000000000000000, 0, 14765.9
Data0, 1083, 738176.4063219444, 2022-01-21T03:45:06.215999, 14769.0, 14769.0, 14769.0, 14769.0, 2.0000000000000000, 0, 14766.85
Data0, 1084, 738176.4063230093, 2022-01-21T03:45:06.308002, 14769.5, 14769.5, 14769.5, 14769.5, 2.0000000000000000, 0, 14767.85

The same with historical :

Data0, 0719, 738176.4069444444, 2022-01-21T03:46:00.000000, 14763.5, 14764.25, 14761.25, 14762.0, 32, 0, 14765.05
Data0, 0720, 738176.4070601852, 2022-01-21T03:46:10.000000, 14762.25, 14763.5, 14761.75, 14762.5, 39, 0, 14764.0
Data0, 0721, 738176.4071759259, 2022-01-21T03:46:20.000000, 14764.0, 14764.75, 14764.0, 14764.75, 3, 0, 14763.9
***** DATA NOTIF: DISCONNECTED

Could you check if you have versions / parameters different from mine, and a behavior different with the same ibtest.py parameters ?

EMR

@emr

To be precise, as this was truncated in my previous post, modification in ibstore is :

_fields = [                     
    ('price', float),           
    ('size', decimal.Decimal),  
    ('datetime', _ts2dt),       
    ('volume', decimal.Decimal),
    ('vwap', float),            
    ('single', bool)            
]

I also added in the import section :

import decimal

@emr oh! I found it without change any line of code.

If I request the current in front contract the software didn't retrieve the historical data.
Using next contract it works like a charm.

Maybe, IB API Bug?

EMR

@dotto95 Good news ! I should have tested the same contract that you tested.
I am not sure to understand precisely "current" a "front", what I understand is that you tried HGF2 (january) : delayed daya. maybe also HGG2 (february) : delayed data. but that when you tried HGH2 (march) you had live data.

I think that in order to get live data, BT need to receive at least one tick. As Jan & Feb are less traded, my guess is that you received no tick outside of RTH, but received one with the more traded March contract. You can check if some trading activity occurred here : https://www.cmegroup.com/markets/metals/base/copper.quotes.html

And you can try in a few minutes with jan & feb contract. Could you please confirm that this was the reason of the problème you faced ?