Gaps in historical minute data

jstin

I'm seeing what appears to be gaps when requesting historical data. Notice how it jumps from 2017-04-11 T23:59:00 to 2017-04-12 T18:00:00. I'm getting this across multiple weekdays. Any help with what I am missing would be appreciated.

python ibtest.py --port 4002 --clientId 555 --data0 CL-201705-NYMEX --timeframe Minutes --compression 1 --fromdate 2017-04-10 --historical

Data0, 0717, 736431.163889, 2017-04-11T23:56:00.000000, 53.56, 53.56, 53.55, 53.55, 12, 0, 53.552
Data0, 0718, 736431.164583, 2017-04-11T23:57:00.000000, 53.55, 53.55, 53.54, 53.54, 30, 0, 53.55
Data0, 0719, 736431.165278, 2017-04-11T23:58:00.000000, 53.54, 53.55, 53.54, 53.55, 28, 0, 53.548
Data0, 0720, 736431.165972, 2017-04-11T23:59:00.000000, 53.55, 53.56, 53.55, 53.56, 4, 0, 53.55
Data0, 0721, 736431.916667, 2017-04-12T18:00:00.000000, 52.85, 52.9, 52.82, 52.87, 255, 0, 53.414
Data0, 0722, 736431.917361, 2017-04-12T18:01:00.000000, 52.87, 52.89, 52.87, 52.89, 35, 0, 53.282
Data0, 0723, 736431.918056, 2017-04-12T18:02:00.000000, 52.89, 52.91, 52.88, 52.91, 70, 0, 53.156
Data0, 0724, 736431.91875, 2017-04-12T18:03:00.000000, 52.9, 52.91, 52.9, 52.9, 79, 0, 53.026
Data0, 0725, 736431.919444, 2017-04-12T18:04:00.000000, 52.89, 52.9, 52.88, 52.89, 34, 0, 52.892

backtrader

CL seems to be the ticker for the Crude Oil futures which after some perusal seem to operate over the midnight barrier.

From here: http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contractSpecs_futures.html

During development a behavior in the Interactive Brokers API reqHistoricalData was observed, where times seem to play an invisible role in filtering out some bars (they are not received by the client, you can check the lack of those messages with --debug)

Given the date of the message (2017-04-19) and the fromdate request (2017-04-10), the calculated duration to pass to the API is in Days (the IB API has not a regular mechanism to say "from this date" to "this other date", but works with a backwards calculation from a given time).

That means that because you have not specified a time in --fromdate 2017-04-10, the request to the API goes with a time of 00:00:00 which is midnight from today and trying to download "x Days backwards. This 00:00:00 may be creating the invisible filter mentioned above.

You may try the following line where the start time of the session is given.

python ibtest.py --port 4002 --clientId 555 --data0 CL-201705-NYMEX --timeframe Minutes --compression 1 --fromdate 2017-04-10T18:00:00 --historical

jstin

Thanks so much!