Gaps in historical minute data



  • I'm seeing what appears to be gaps when requesting historical data. Notice how it jumps from 2017-04-11 T23:59:00 to 2017-04-12 T18:00:00. I'm getting this across multiple weekdays. Any help with what I am missing would be appreciated.

    python ibtest.py --port 4002 --clientId 555 --data0 CL-201705-NYMEX --timeframe Minutes --compression 1 --fromdate 2017-04-10 --historical

    Data0, 0717, 736431.163889, 2017-04-11T23:56:00.000000, 53.56, 53.56, 53.55, 53.55, 12, 0, 53.552
    Data0, 0718, 736431.164583, 2017-04-11T23:57:00.000000, 53.55, 53.55, 53.54, 53.54, 30, 0, 53.55
    Data0, 0719, 736431.165278, 2017-04-11T23:58:00.000000, 53.54, 53.55, 53.54, 53.55, 28, 0, 53.548
    Data0, 0720, 736431.165972, 2017-04-11T23:59:00.000000, 53.55, 53.56, 53.55, 53.56, 4, 0, 53.55
    Data0, 0721, 736431.916667, 2017-04-12T18:00:00.000000, 52.85, 52.9, 52.82, 52.87, 255, 0, 53.414
    Data0, 0722, 736431.917361, 2017-04-12T18:01:00.000000, 52.87, 52.89, 52.87, 52.89, 35, 0, 53.282
    Data0, 0723, 736431.918056, 2017-04-12T18:02:00.000000, 52.89, 52.91, 52.88, 52.91, 70, 0, 53.156
    Data0, 0724, 736431.91875, 2017-04-12T18:03:00.000000, 52.9, 52.91, 52.9, 52.9, 79, 0, 53.026
    Data0, 0725, 736431.919444, 2017-04-12T18:04:00.000000, 52.89, 52.9, 52.88, 52.89, 34, 0, 52.892
    

  • administrators

    CL seems to be the ticker for the Crude Oil futures which after some perusal seem to operate over the midnight barrier.

    From here: http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contractSpecs_futures.html

    During development a behavior in the Interactive Brokers API reqHistoricalData was observed, where times seem to play an invisible role in filtering out some bars (they are not received by the client, you can check the lack of those messages with --debug)

    Given the date of the message (2017-04-19) and the fromdate request (2017-04-10), the calculated duration to pass to the API is in Days (the IB API has not a regular mechanism to say "from this date" to "this other date", but works with a backwards calculation from a given time).

    That means that because you have not specified a time in --fromdate 2017-04-10, the request to the API goes with a time of 00:00:00 which is midnight from today and trying to download "x Days backwards. This 00:00:00 may be creating the invisible filter mentioned above.

    You may try the following line where the start time of the session is given.

    python ibtest.py --port 4002 --clientId 555 --data0 CL-201705-NYMEX --timeframe Minutes --compression 1 --fromdate 2017-04-10T18:00:00 --historical
    


  • Thanks so much!


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.