IB backfilling gap...
-
I am running the ibtest.py and it does historic data download for backfilling, however there are always gaps due to time zone issues.
backtrader sends endtime in GMT to reqHistoricaData and assuming the exchange as DAX (CET) it doesn't download all the data till the current time and there remains a gap between the last bar and the current prices....
below is the command and last few lines of the data showing difference where historic data ends (actually its hours behind) and current prices.python ibtest.py --port 8099 --clientId 1 --data0 IBUS500-CFD-SMART --timeframe Minutes --compression 1 --what BID --broker --rtbar
I have also used --timezone Asia/Singapore and --no-timeoffset still the same issue.Any thoughts... I am running this from Singapore, I standalone programs using ibpy downloads correctly for similar settings to reqHistoricalData.
Data0, 0051, 736338.701389, 2017-01-09T16:50:00.000000, 2278.12, 2278.13, 2277.87, 2277.87, -1, 0, 2278.07
Data0, 0052, 736338.702083, 2017-01-09T16:51:00.000000, 2277.87, 2277.88, 2277.37, 2277.37, -1, 0, 2277.92
Data0, 0053, 736338.702778, 2017-01-09T16:52:00.000000, 2277.37, 2277.38, 2277.12, 2277.12, -1, 0, 2277.72
Data0, 0054, 736338.703472, 2017-01-09T16:53:00.000000, 2277.12, 2277.13, 2276.87, 2277.12, -1, 0, 2277.52
Data0, 0055, 736338.704167, 2017-01-09T16:54:00.000000, 2277.12, 2277.38, 2277.12, 2277.12, -1, 0, 2277.32
Data0, 0056, 736338.704861, 2017-01-09T16:55:00.000000, 2277.12, 2277.13, 2276.87, 2276.87, -1, 0, 2277.12
Data0, 0057, 736338.705556, 2017-01-09T16:56:00.000000, 2276.87, 2276.88, 2276.87, 2276.87, -1, 0, 2277.02
***** DATA NOTIF: LIVE
Data0, 0058, 736338.705845, 2017-01-09T16:56:25.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2276.182
Data0, 0059, 736338.705903, 2017-01-09T16:56:30.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2275.344
Data0, 0060, 736338.705961, 2017-01-09T16:56:35.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2274.506
Data0, 0061, 736338.706019, 2017-01-09T16:56:40.000000, 2272.94, 2272.94, 2272.68, 2272.69, -1, 0, 2273.67 -
Some things in the post may need a bit of clarification, to try to discern where the problem may be (should there be)
Execution line
python ibtest.py --port 8099 --clientId 1 --data0 IBUS500-CFD-SMART --timeframe Minutes --compression 1 --what BID --broker --rtbar
This is for sure running a modified version of backtrader, because as discussed in another thread, the
CFD
notation is not supported. But for the sake of a fruitful post, let's assume the modification wer properly done and only for the contract parsing part.Exchanges
sends endtime in GMT to reqHistoricaData and assuming the exchange as DAX (CET)
The command above is not querying the
DAX
, but something else. IfIBUS500-CFD-SMART
also uses aCET
timezone is unknown but unlikely, because thisCFD
seems to track theS&P500
. AnEST
timezone would be more probable.Time gap between historical and live data
... last few lines of the data showing difference where historic data ends (actually its hours behind) and current prices.
... Data0, 0056, 736338.704861, 2017-01-09T16:55:00.000000, 2277.12, 2277.13, 2276.87, 2276.87, -1, 0, 2277.12 Data0, 0057, 736338.705556, 2017-01-09T16:56:00.000000, 2276.87, 2276.88, 2276.87, 2276.87, -1, 0, 2277.02 ***** DATA NOTIF: LIVE Data0, 0058, 736338.705845, 2017-01-09T16:56:25.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2276.182 Data0, 0059, 736338.705903, 2017-01-09T16:56:30.000000, 2272.93, 2272.93, 2272.93, 2272.93, -1, 0, 2275.344 ...
The output above shows only a gap between
2017-01-09T16:56:00
(last1-minute
bar which comes from the historical download) and2017-01-09T16:56:25
. It doesn't seem to be hours behind. The obvious difference is that:- The timeframe and compression have been set to
Minutes
and1
and this has been used to download the historical data. - The sample has not been requested to further resample (with
--resample
) incoming data and delivers the5-seconds
bars it receives
- The timeframe and compression have been set to
-
the modification is only to treat CFD inline with CASH so that it requests BID instead of TRADES
one correction though the backtrader code doesn't put timezone while querying for reqHistoricalData, it just puts time entry.here is sample for Australia which shows considerable difference between end of historical data and live ones...
python ibtest.py --port 8099 --clientId 1 --data0 IBAU200-CFD-SMART --timeframe Seconds --compression 5 --what BID --broker
Data0, 1435, 736338.541319, 2017-01-09T12:59:30.000000, 5807.08, 5807.09, 5807.08, 5807.09, -1, 0, 5807.682
Data0, 1436, 736338.541377, 2017-01-09T12:59:35.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.484
Data0, 1437, 736338.541435, 2017-01-09T12:59:40.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.286
Data0, 1438, 736338.541493, 2017-01-09T12:59:45.000000, 5807.09, 5807.09, 5807.08, 5807.09, -1, 0, 5807.088
Data0, 1439, 736338.541551, 2017-01-09T12:59:50.000000, 5807.09, 5807.09, 5807.08, 5807.08, -1, 0, 5807.088
Data0, 1440, 736338.541609, 2017-01-09T12:59:55.000000, 5807.08, 5807.09, 5807.08, 5807.08, -1, 0, 5807.086
***** DATA NOTIF: LIVE
Data0, 1441, 736339.076198, 2017-01-10T01:49:43.530001, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5796.708
Data0, 1442, 736339.076276, 2017-01-10T01:49:50.216004, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5786.53
Data0, 1443, 736339.076612, 2017-01-10T01:50:19.282000, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5776.152
Data0, 1444, 736339.076617, 2017-01-10T01:50:19.750998, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5765.976
Data0, 1445, 736339.07663, 2017-01-10T01:50:20.796999, 5755.2, 5755.2, 5755.2, 5755.2, 0, 0, 5755.6
Data0, 1446, 736339.076635, 2017-01-10T01:50:21.296996, 5756.2, 5756.2, 5756.2, 5756.2, 0, 0, 5755.8 -
Following your statement one can conclude that corrections to the platform go beyond contract parsing and affect time input to Interactive Brokers
- May be you don't mind sharing what you have actually modified? (git can provide unified diff output)
How the platform works with regards to time management and timezones is detailed here:
The
--tz
parameter in the `ibtest.py`` sample (which is just a toy sample to try the widest possible amount of things) has nothing to do with which timezone will be used for requests. It is exclusively for the output. From the sample code:parser.add_argument('--timezone', default=None, required=False, action='store', help='timezone to get time output into (pytz names)')
Furthermore the sample outputs the actual timezone read in from
IB
which is used to transform the time to a non-timezone dependent time internally. From the sample codeprint('Timezone from ContractDetails: {}'.format( self.data0.contractdetails.m_timeZoneId))
But none of the output samples above reveals which is the incoming timezone.
-
The Interactive Brokers
CFD
s operate in theGB
timezone. This is notUTC
as mentioned above, only that theGB
timezone matches now, during the winter, the times ofUTC
.Output from the sample when querying one of the aforementioned
CFD
s:$ ./ibtest.py --port 7497 --resample --timeframe Minutes --compression 1 --data0 IBAU200-CFD-SMART Server Version: 76 TWS Time at connection:20170110 11:37:26 CET -------------------------------------------------- Strategy Created -------------------------------------------------- Timezone from ContractDetails: GB Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture> ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm> ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm> ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
The time management part of the documentation also states:
Note*
Input from the user like for example the parametersfromdate
orsessionstart
is expected to be in sync with the actualtz
, be it automatically calculated by the data source, supplied by the user or left as default (None
, which means direct input-output of datetime)The time management part in backtrader will not request the historical data in your desired timezone and the reason as explained in the Time Management part of the documentation is to avoid datetime overlap errors due to time shifting (the example shows what happens between Europe and the USA during 2 weeks every year)
Internally the core will not request the historical data in your desired timezone, because of that. It tries to work in
UTC
(orUTC-like
as explained in the docs) format and only give you the desired timezone when you query the datetime from the data (for example withself.data.datetime.datetime()
In any case it seems wiser to always work with the target timezone of the asset and not with the local timezone.
-
Thanks for details backtrader and appreciate your responses.
I have gone through the Time management docs and as per that we don't need to derive specific feed since IB does provide timezone in contractdetails.
I have now revered all my changes and taken the backtrader code as is and trying on ES (e-mini futures), the issue however persists although it shows the correct timings the data doesn't match..
python ibtest.py --port 8099 --clientId 1 --data0 ES-201703-GLOBEX-USD --timeframe Minutes --compression 1 --broker --resample --stopafter 1 Server Version: 76 TWS Time at connection:20170111 00:10:21 SGT -------------------------------------------------- Strategy Created -------------------------------------------------- Timezone from ContractDetails: America/Belize Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA ...... ...... Data0, 0547, 736339.670833, 2017-01-10T10:06:00.000000, 2265.75, 2265.75, 2265.5, 2265.75, 453.0, 0.0, 2265.1 Data0, 0548, 736339.671528, 2017-01-10T10:07:00.000000, 2265.5, 2265.75, 2265.25, 2265.25, 750.0, 0.0, 2265.25 Data0, 0549, 736339.672222, 2017-01-10T10:08:00.000000, 2265.25, 2265.5, 2265.25, 2265.25, 360.0, 0.0, 2265.45 Data0, 0550, 736339.672917, 2017-01-10T10:09:00.000000, 2265.25, 2265.75, 2265.25, 2265.5, 278.0, 0.0, 2265.5 Data0, 0551, 736339.673611, 2017-01-10T10:10:00.000000, 2265.5, 2265.75, 2265.5, 2265.5, 113.0, 0.0, 2265.45 ***** DATA NOTIF: LIVE Data0, 0552, 736339.674306, 2017-01-10T10:11:00.000000, 2269.5, 2269.5, 2269.25, 2269.25, 24451.0, 0.0, 2266.15
this is backtrader as is code and not using standard products (not CFD)
This is what I guess is happening after looking into the code....
backtrader queries for realtime data and takes the timestamp from the first row and then queries historical data using that as endtime to backfill with max possible duration, the IB server however in absence of timezone in the request treats it as local time and returns data accordingly... -
The problem is actually somewhere else to start with. The historical backfilling datetime strings are currently being returned in localtime format and there is where you may have seen the gap (due to discrepancies in the price because some hours have elapsed). The code doesn't know it is in localtime and that's why times perfectly align later.
Looking at the output of the original testing, documented in the blog, there were no gaps in the prices.
Switching to using unix timestamps is not straightforward, because the the most up to date API documentation from
IB
(the one at GitHub) states that the1 day
bars can only request strings for the datetime timestamps. -
It was actually a lot easier than it seemed. Regardless of why (and/or when) the problem has shown up, the request is now tried with an expected seconds timestamp. TWS will still return a datetime string in some cases (barsize dependent), but the code keeps track of the expectation and parses accordingly a string or a UTC timestamp and the subsequent request knows where to request the enddate for the backfilling.
The changes after some testing have been pushed to both the
master
anddevelopment
branches. You may pull them and give it a try. -
@ backtrader
I have taken this copy and see it working fine with respect to backfilling, appreciate the quick fix. Thanks. -
An extra fix has been pushed to the
development
branch, because @randyt also checked it out and found a typo in the part which services historical data downloads between 2 dates (the part initially tested, downloads as much as possible and not between two given dates, which may happend during a disconnection/reconnection cycle)