Replay data and backfilling on real time
-
Trying to run the ibtest script from the source code with this command
./ibtest.py --replay --timeframe Days --compression 1 --data0 EUR.USD-CASH-IDEALPRO
and the log for the first data received onto the backfilling is:
***** DATA NOTIF: DELAYED Data0, 0001, 736088.1666666665, 2016-05-03T23:59:59.999986, 1.1531, 1.1616, 1.1495, 1.1496, -1.0, 0, nan Data0, 0001, 736089.1666666665, 2016-05-04T23:59:59.999986, 1.1531, 1.1616, 1.1466, 1.1486, -2.0, 0, nan Data0, 0002, 736090.1666666665, 2016-05-05T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.14045, -1.0, 0, nan Data0, 0002, 736091.1666666665, 2016-05-06T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.1406, -2.0, 0, nan Data0, 0003, 736094.1666666665, 2016-05-09T23:59:59.999986, 1.13945, 1.14195, 1.1375, 1.1383, -1.0, 0, nan Data0, 0003, 736095.1666666665, 2016-05-10T23:59:59.999986, 1.13945, 1.14195, 1.13585, 1.13705, -2.0, 0, nan Data0, 0004, 736096.1666666665, 2016-05-11T23:59:59.999986, 1.13735, 1.14465, 1.1369, 1.14255, -1.0, 0, nan
The first thing I see wrong is that the open and high seems to be consecutive the same every 2 days.
Also, at this point in time if I try to access the close of the data, this defers from what´s received on the IB feed:
So If I print last data0.close is:datas[0].close[0] = 1.1376 datas[0].close[-1] = 1.13705 datas[0].close[-2] = 1.1406 datas[0].close[-3] = 1.1486 datas[0].close[-4] = 1.1376 datas[0].close[-5] = 1.13705 datas[0].close[-6] = 1.1406 datas[0].close[-7] = 1.1486 datas[0].close[-8] = "Not available yet"`
How is the data being backfilled?
Also, when the data reach the "LIVE" status, so the feed starts consuming in real time I don´t have access to the whole data backfilled; only the last values.
I´ve tried also with other securities like SPY-STK-SMART-USD and the behaviour is the same.
Any ideas?
-
Seems that the access to old backhistory was related to running with exactbars=1.
When changed, removing this property, I have access to all the history.
But still with the data issue... -
I´ve tried to run the same script with "resample" instead of replay and the data is backfilled accordingly to what it´s received.
But the data received seems to be losing 1 of each 2 consecutive days... (I´ve set the --debug option and I´ve verified that IB data is being received correctly)
Am I doing something wrong or do I need to use some extra parameter?Data0, 0103, 736385.2083333333, 2017-02-25T00:00:00.000000, 236.27, 236.99, 235.21, 236.74, 1423468.0, 0, 234.906 Data0, 0104, 736389.2083333333, 2017-03-01T00:00:00.000000, 236.8, 237.31, 236.01, 236.93, 1340010.0, 0, 235.998 Data0, 0105, 736391.2083333333, 2017-03-03T00:00:00.000000, 237.42, 240.32, 237.42, 238.16, 1547475.0, 0, 236.62800000000001 Data0, 0106, 736393.2083333333, 2017-03-05T00:00:00.000000, 237.88, 238.61, 237.66, 238.35, 689651.0, 0, 237.27800000000002 Data0, 0107, 736397.2083333333, 2017-03-09T00:00:00.000000, 237.52, 238.12, 236.36, 236.73, 1712065.0, 0, 237.382 Data0, 0108, 736399.2083333333, 2017-03-11T00:00:00.000000, 236.79, 238.32, 235.74, 237.78, 1464157.0, 0, 237.59 Data0, 0109, 736403.1666666665, 2017-03-14T23:59:59.999986, 237.45, 237.89, 236.19, 237.08, 1009787.0, 0, 237.61999999999998 Data0, 0110, 736405.1666666665, 2017-03-16T23:59:59.999986, 237.32, 239.75, 237.29, 238.6, 1390938.0, 0, 237.708 Data0, 0111, 736407.1666666665, 2017-03-18T23:59:59.999986, 237.42, 238.23, 236.88, 236.98, 700959.0, 0, 237.43400000000003 Data0, 0112, 736411.1666666665, 2017-03-22T23:59:59.999986, 237.06, 237.61, 233.05, 234.51, 2358886.0, 0, 236.99 Data0, 0113, 736413.1666666665, 2017-03-24T23:59:59.999986, 234.25, 235.34, 232.96, 233.94, 1885646.0, 0, 236.22199999999998 Data0, 0114, 736417.1666666665, 2017-03-28T23:59:59.999986, 232.1, 235.81, 231.61, 235.12, 1505616.0, 0, 235.83 Data0, 0115, 736419.1666666665, 2017-03-30T23:59:59.999986, 235.26, 236.52, 234.72, 236.21, 990011.0, 0, 235.352 Data0, 0116, 736421.1666666665, 2017-04-01T23:59:59.999986, 235.76, 236.51, 235.49, 235.65, 618459.0, 0, 235.086 Data0, 0117, 736425.1666666665, 2017-04-05T23:59:59.999986, 236.13, 237.39, 233.91, 234.31, 2251815.0, 0, 235.046 Data0, 0118, 736427.1666666665, 2017-04-07T23:59:59.999986, 234.23, 236.04, 234.15, 235.19, 1347907.0, 0, 235.296 Data0, 0119, 736431.1666666665, 2017-04-11T23:59:59.999986, 235.26, 236.26, 233.34, 235.03, 1248691.0, 0, 235.27800000000002 Data0, 0120, 736433.1666666665, 2017-04-13T23:59:59.999986, 235.4, 235.48, 232.29, 232.42, 1478385.0, 0, 234.51999999999998 Data0, 0121, 736438.1666666665, 2017-04-18T23:59:59.999986, 232.5, 234.62, 232.4, 233.91, 1151612.0, 0, 234.17199999999997 Data0, 0122, 736440.1666666665, 2017-04-20T23:59:59.999986, 234.05, 235.85, 233.18, 235.32, 1370622.0, 0, 234.37399999999997 Data0, 0123, 736442.1666666665, 2017-04-22T23:59:59.999986, 235.45, 235.61, 234.0, 234.33, 733098.0, 0, 234.202 Data0, 0124, 736446.1666666665, 2017-04-26T23:59:59.999986, 237.06, 239.53, 234.59, 238.44, 2268990.0, 0, 234.88400000000001 Data0, 0125, 736448.1666666665, 2017-04-28T23:59:59.999986, 238.4, 238.96, 237.83, 237.93, 1057622.0, 0, 235.98600000000002
-
There was another thread which already addressed this problem, which is due to the rounding around midnight.
The sample might be in need of refreshing, due to the stricter usage of
sessionend
when for example doing resampling operations.ibtest
was created long before that was done to actually be safer in synchronizing data feeds.