IB backfilling gap...

skytrading54

I am running the ibtest.py and it does historic data download for backfilling, however there are always gaps due to time zone issues.
backtrader sends endtime in GMT to reqHistoricaData and assuming the exchange as DAX (CET) it doesn't download all the data till the current time and there remains a gap between the last bar and the current prices....
below is the command and last few lines of the data showing difference where historic data ends (actually its hours behind) and current prices.

python ibtest.py --port 8099 --clientId 1 --data0 IBUS500-CFD-SMART --timeframe Minutes --compression 1 --what BID --broker --rtbar
I have also used --timezone Asia/Singapore and --no-timeoffset still the same issue.

Any thoughts... I am running this from Singapore, I standalone programs using ibpy downloads correctly for similar settings to reqHistoricalData.

Data0, 0051, 736338.701389, 2017-01-09T16:50:00.000000, 2278.12, 2278.13, 2277.87, 2277.87, -1, 0, 2278.07
Data0, 0052, 736338.702083, 2017-01-09T16:51:00.000000, 2277.87, 2277.88, 2277.37, 2277.37, -1, 0, 2277.92
Data0, 0053, 736338.702778, 2017-01-09T16:52:00.000000, 2277.37, 2277.38, 2277.12, 2277.12, -1, 0, 2277.72
Data0, 0054, 736338.703472, 2017-01-09T16:53:00.000000, 2277.12, 2277.13, 2276.87, 2277.12, -1, 0, 2277.52
Data0, 0055, 736338.704167, 2017-01-09T16:54:00.000000, 2277.12, 2277.38, 2277.12, 2277.12, -1, 0, 2277.32
Data0, 0056, 736338.704861, 2017-01-09T16:55:00.000000, 2277.12, 2277.13, 2276.87, 2276.87, -1, 0, 2277.12
Data0, 0057, 736338.705556, 2017-01-09T16:56:00.000000, 2276.87, 2276.88, 2276.87, 2276.87, -1, 0, 2277.02
***** DATA NOTIF: LIVE
Data0, 0058, 736338.705845, 2017-01-09T16:56:25.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2276.182
Data0, 0059, 736338.705903, 2017-01-09T16:56:30.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2275.344
Data0, 0060, 736338.705961, 2017-01-09T16:56:35.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2274.506
Data0, 0061, 736338.706019, 2017-01-09T16:56:40.000000, 2272.94, 2272.94, 2272.68, 2272.69, -1, 0, 2273.67

backtrader

Some things in the post may need a bit of clarification, to try to discern where the problem may be (should there be)

Execution line

python ibtest.py --port 8099 --clientId 1 --data0 IBUS500-CFD-SMART --timeframe Minutes --compression 1 --what BID --broker --rtbar

This is for sure running a modified version of backtrader, because as discussed in another thread, the CFD notation is not supported. But for the sake of a fruitful post, let's assume the modification wer properly done and only for the contract parsing part.

Exchanges

sends endtime in GMT to reqHistoricaData and assuming the exchange as DAX (CET)

The command above is not querying the DAX, but something else. If IBUS500-CFD-SMART also uses a CET timezone is unknown but unlikely, because this CFD seems to track the S&P500. An EST timezone would be more probable.

Time gap between historical and live data

... last few lines of the data showing difference where historic data ends (actually its hours behind) and current prices.

...
Data0, 0056, 736338.704861, 2017-01-09T16:55:00.000000, 2277.12, 2277.13, 2276.87, 2276.87, -1, 0, 2277.12
Data0, 0057, 736338.705556, 2017-01-09T16:56:00.000000, 2276.87, 2276.88, 2276.87, 2276.87, -1, 0, 2277.02
***** DATA NOTIF: LIVE
Data0, 0058, 736338.705845, 2017-01-09T16:56:25.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2276.182
Data0, 0059, 736338.705903, 2017-01-09T16:56:30.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2275.344
...

The output above shows only a gap between 2017-01-09T16:56:00 (last 1-minute bar which comes from the historical download) and 2017-01-09T16:56:25. It doesn't seem to be hours behind. The obvious difference is that:

The timeframe and compression have been set to Minutes and 1 and this has been used to download the historical data.
The sample has not been requested to further resample (with --resample) incoming data and delivers the 5-seconds bars it receives

skytrading54

the modification is only to treat CFD inline with CASH so that it requests BID instead of TRADES
one correction though the backtrader code doesn't put timezone while querying for reqHistoricalData, it just puts time entry.

here is sample for Australia which shows considerable difference between end of historical data and live ones...

python ibtest.py --port 8099 --clientId 1 --data0 IBAU200-CFD-SMART --timeframe Seconds --compression 5 --what BID --broker

Data0, 1435, 736338.541319, 2017-01-09T12:59:30.000000, 5807.08, 5807.09, 5807.08, 5807.09, -1, 0, 5807.682
Data0, 1436, 736338.541377, 2017-01-09T12:59:35.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.484
Data0, 1437, 736338.541435, 2017-01-09T12:59:40.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.286
Data0, 1438, 736338.541493, 2017-01-09T12:59:45.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.088
Data0, 1439, 736338.541551, 2017-01-09T12:59:50.000000, 5807.09, 5807.09, 5807.08, 5807.08, -1, 0, 5807.088
Data0, 1440, 736338.541609, 2017-01-09T12:59:55.000000, 5807.08, 5807.09, 5807.08, 5807.08, -1, 0, 5807.086
***** DATA NOTIF: LIVE
Data0, 1441, 736339.076198, 2017-01-10T01:49:43.530001, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5796.708
Data0, 1442, 736339.076276, 2017-01-10T01:49:50.216004, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5786.53
Data0, 1443, 736339.076612, 2017-01-10T01:50:19.282000, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5776.152
Data0, 1444, 736339.076617, 2017-01-10T01:50:19.750998, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5765.976
Data0, 1445, 736339.07663, 2017-01-10T01:50:20.796999, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5755.6
Data0, 1446, 736339.076635, 2017-01-10T01:50:21.296996, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5755.8

backtrader

Following your statement one can conclude that corrections to the platform go beyond contract parsing and affect time input to Interactive Brokers

May be you don't mind sharing what you have actually modified? (git can provide unified diff output)

How the platform works with regards to time management and timezones is detailed here:

https://www.backtrader.com/docu/timemgmt.html

The --tz parameter in the `ibtest.py`` sample (which is just a toy sample to try the widest possible amount of things) has nothing to do with which timezone will be used for requests. It is exclusively for the output. From the sample code:

    parser.add_argument('--timezone', default=None,
                        required=False, action='store',
                        help='timezone to get time output into (pytz names)')

Furthermore the sample outputs the actual timezone read in from IB which is used to transform the time to a non-timezone dependent time internally. From the sample code

            print('Timezone from ContractDetails: {}'.format(
                  self.data0.contractdetails.m_timeZoneId))

But none of the output samples above reveals which is the incoming timezone.

backtrader

The Interactive Brokers CFDs operate in the GB timezone. This is not UTC as mentioned above, only that the GB timezone matches now, during the winter, the times of UTC.

Output from the sample when querying one of the aforementioned CFDs:

$ ./ibtest.py --port 7497 --resample --timeframe Minutes --compression 1 --data0 IBAU200-CFD-SMART
Server Version: 76
TWS Time at connection:20170110 11:37:26 CET
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: GB
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>

The time management part of the documentation also states:

Note*
Input from the user like for example the parameters fromdate or sessionstart is expected to be in sync with the actual tz, be it automatically calculated by the data source, supplied by the user or left as default (None, which means direct input-output of datetime)

The time management part in backtrader will not request the historical data in your desired timezone and the reason as explained in the Time Management part of the documentation is to avoid datetime overlap errors due to time shifting (the example shows what happens between Europe and the USA during 2 weeks every year)

Internally the core will not request the historical data in your desired timezone, because of that. It tries to work in UTC (or UTC-like as explained in the docs) format and only give you the desired timezone when you query the datetime from the data (for example with self.data.datetime.datetime()

In any case it seems wiser to always work with the target timezone of the asset and not with the local timezone.

skytrading54

Thanks for details backtrader and appreciate your responses.

I have gone through the Time management docs and as per that we don't need to derive specific feed since IB does provide timezone in contractdetails.

I have now revered all my changes and taken the backtrader code as is and trying on ES (e-mini futures), the issue however persists although it shows the correct timings the data doesn't match..

python ibtest.py --port 8099 --clientId 1 --data0 ES-201703-GLOBEX-USD --timeframe Minutes --compression 1 --broker --resample --stopafter 1

Server Version: 76
TWS Time at connection:20170111 00:10:21 SGT
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
......
......
Data0, 0547, 736339.670833, 2017-01-10T10:06:00.000000, 2265.75, 2265.75, 2265.5, 2265.75, 453.0, 0.0, 2265.1
Data0, 0548, 736339.671528, 2017-01-10T10:07:00.000000, 2265.5, 2265.75, 2265.25, 2265.25, 750.0, 0.0, 2265.25
Data0, 0549, 736339.672222, 2017-01-10T10:08:00.000000, 2265.25, 2265.5, 2265.25, 2265.25, 360.0, 0.0, 2265.45
Data0, 0550, 736339.672917, 2017-01-10T10:09:00.000000, 2265.25, 2265.75, 2265.25, 2265.5, 278.0, 0.0, 2265.5
Data0, 0551, 736339.673611, 2017-01-10T10:10:00.000000, 2265.5, 2265.75, 2265.5, 2265.5, 113.0, 0.0, 2265.45
***** DATA NOTIF: LIVE
Data0, 0552, 736339.674306, 2017-01-10T10:11:00.000000, 2269.5, 2269.5, 2269.25, 2269.25, 24451.0, 0.0, 2266.15

this is backtrader as is code and not using standard products (not CFD)
This is what I guess is happening after looking into the code....
backtrader queries for realtime data and takes the timestamp from the first row and then queries historical data using that as endtime to backfill with max possible duration, the IB server however in absence of timezone in the request treats it as local time and returns data accordingly...

backtrader

The problem is actually somewhere else to start with. The historical backfilling datetime strings are currently being returned in localtime format and there is where you may have seen the gap (due to discrepancies in the price because some hours have elapsed). The code doesn't know it is in localtime and that's why times perfectly align later.

Looking at the output of the original testing, documented in the blog, there were no gaps in the prices.

Switching to using unix timestamps is not straightforward, because the the most up to date API documentation from IB (the one at GitHub) states that the 1 day bars can only request strings for the datetime timestamps.

backtrader

It was actually a lot easier than it seemed. Regardless of why (and/or when) the problem has shown up, the request is now tried with an expected seconds timestamp. TWS will still return a datetime string in some cases (barsize dependent), but the code keeps track of the expectation and parses accordingly a string or a UTC timestamp and the subsequent request knows where to request the enddate for the backfilling.

The changes after some testing have been pushed to both the master and development branches. You may pull them and give it a try.

skytrading54

@ backtrader
I have taken this copy and see it working fine with respect to backfilling, appreciate the quick fix. Thanks.

backtrader

An extra fix has been pushed to the development branch, because @randyt also checked it out and found a typo in the part which services historical data downloads between 2 dates (the part initially tested, downloads as much as possible and not between two given dates, which may happend during a disconnection/reconnection cycle)