Replay data and backfilling on real time

ealvarpe

Trying to run the ibtest script from the source code with this command

./ibtest.py --replay --timeframe Days --compression 1 --data0 EUR.USD-CASH-IDEALPRO

and the log for the first data received onto the backfilling is:

***** DATA NOTIF: DELAYED
Data0, 0001, 736088.1666666665, 2016-05-03T23:59:59.999986, 1.1531, 1.1616, 1.1495, 1.1496, -1.0, 0, nan
Data0, 0001, 736089.1666666665, 2016-05-04T23:59:59.999986, 1.1531, 1.1616, 1.1466, 1.1486, -2.0, 0, nan
Data0, 0002, 736090.1666666665, 2016-05-05T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.14045, -1.0, 0, nan
Data0, 0002, 736091.1666666665, 2016-05-06T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.1406, -2.0, 0, nan
Data0, 0003, 736094.1666666665, 2016-05-09T23:59:59.999986, 1.13945, 1.14195, 1.1375, 1.1383, -1.0, 0, nan
Data0, 0003, 736095.1666666665, 2016-05-10T23:59:59.999986, 1.13945, 1.14195, 1.13585, 1.13705, -2.0, 0, nan
Data0, 0004, 736096.1666666665, 2016-05-11T23:59:59.999986, 1.13735, 1.14465, 1.1369, 1.14255, -1.0, 0, nan

The first thing I see wrong is that the open and high seems to be consecutive the same every 2 days.

Also, at this point in time if I try to access the close of the data, this defers from what´s received on the IB feed:
So If I print last data0.close is:

datas[0].close[0] = 1.1376
datas[0].close[-1] = 1.13705
datas[0].close[-2] = 1.1406
datas[0].close[-3] = 1.1486
datas[0].close[-4] = 1.1376
datas[0].close[-5] = 1.13705
datas[0].close[-6] = 1.1406
datas[0].close[-7] = 1.1486
datas[0].close[-8] = "Not available yet"`

How is the data being backfilled?

Also, when the data reach the "LIVE" status, so the feed starts consuming in real time I don´t have access to the whole data backfilled; only the last values.

I´ve tried also with other securities like SPY-STK-SMART-USD and the behaviour is the same.

Any ideas?

ealvarpe

Seems that the access to old backhistory was related to running with exactbars=1.
When changed, removing this property, I have access to all the history.
But still with the data issue...

ealvarpe

I´ve tried to run the same script with "resample" instead of replay and the data is backfilled accordingly to what it´s received.
But the data received seems to be losing 1 of each 2 consecutive days... (I´ve set the --debug option and I´ve verified that IB data is being received correctly)
Am I doing something wrong or do I need to use some extra parameter?

Data0, 0103, 736385.2083333333, 2017-02-25T00:00:00.000000, 236.27, 236.99, 235.21, 236.74, 1423468.0, 0, 234.906
Data0, 0104, 736389.2083333333, 2017-03-01T00:00:00.000000, 236.8, 237.31, 236.01, 236.93, 1340010.0, 0, 235.998
Data0, 0105, 736391.2083333333, 2017-03-03T00:00:00.000000, 237.42, 240.32, 237.42, 238.16, 1547475.0, 0, 236.62800000000001
Data0, 0106, 736393.2083333333, 2017-03-05T00:00:00.000000, 237.88, 238.61, 237.66, 238.35, 689651.0, 0, 237.27800000000002
Data0, 0107, 736397.2083333333, 2017-03-09T00:00:00.000000, 237.52, 238.12, 236.36, 236.73, 1712065.0, 0, 237.382
Data0, 0108, 736399.2083333333, 2017-03-11T00:00:00.000000, 236.79, 238.32, 235.74, 237.78, 1464157.0, 0, 237.59
Data0, 0109, 736403.1666666665, 2017-03-14T23:59:59.999986, 237.45, 237.89, 236.19, 237.08, 1009787.0, 0, 237.61999999999998
Data0, 0110, 736405.1666666665, 2017-03-16T23:59:59.999986, 237.32, 239.75, 237.29, 238.6, 1390938.0, 0, 237.708
Data0, 0111, 736407.1666666665, 2017-03-18T23:59:59.999986, 237.42, 238.23, 236.88, 236.98, 700959.0, 0, 237.43400000000003
Data0, 0112, 736411.1666666665, 2017-03-22T23:59:59.999986, 237.06, 237.61, 233.05, 234.51, 2358886.0, 0, 236.99
Data0, 0113, 736413.1666666665, 2017-03-24T23:59:59.999986, 234.25, 235.34, 232.96, 233.94, 1885646.0, 0, 236.22199999999998
Data0, 0114, 736417.1666666665, 2017-03-28T23:59:59.999986, 232.1, 235.81, 231.61, 235.12, 1505616.0, 0, 235.83
Data0, 0115, 736419.1666666665, 2017-03-30T23:59:59.999986, 235.26, 236.52, 234.72, 236.21, 990011.0, 0, 235.352
Data0, 0116, 736421.1666666665, 2017-04-01T23:59:59.999986, 235.76, 236.51, 235.49, 235.65, 618459.0, 0, 235.086
Data0, 0117, 736425.1666666665, 2017-04-05T23:59:59.999986, 236.13, 237.39, 233.91, 234.31, 2251815.0, 0, 235.046
Data0, 0118, 736427.1666666665, 2017-04-07T23:59:59.999986, 234.23, 236.04, 234.15, 235.19, 1347907.0, 0, 235.296
Data0, 0119, 736431.1666666665, 2017-04-11T23:59:59.999986, 235.26, 236.26, 233.34, 235.03, 1248691.0, 0, 235.27800000000002
Data0, 0120, 736433.1666666665, 2017-04-13T23:59:59.999986, 235.4, 235.48, 232.29, 232.42, 1478385.0, 0, 234.51999999999998
Data0, 0121, 736438.1666666665, 2017-04-18T23:59:59.999986, 232.5, 234.62, 232.4, 233.91, 1151612.0, 0, 234.17199999999997
Data0, 0122, 736440.1666666665, 2017-04-20T23:59:59.999986, 234.05, 235.85, 233.18, 235.32, 1370622.0, 0, 234.37399999999997
Data0, 0123, 736442.1666666665, 2017-04-22T23:59:59.999986, 235.45, 235.61, 234.0, 234.33, 733098.0, 0, 234.202
Data0, 0124, 736446.1666666665, 2017-04-26T23:59:59.999986, 237.06, 239.53, 234.59, 238.44, 2268990.0, 0, 234.88400000000001
Data0, 0125, 736448.1666666665, 2017-04-28T23:59:59.999986, 238.4, 238.96, 237.83, 237.93, 1057622.0, 0, 235.98600000000002

backtrader

There was another thread which already addressed this problem, which is due to the rounding around midnight.

The sample might be in need of refreshing, due to the stricter usage of sessionend when for example doing resampling operations. ibtest was created long before that was done to actually be safer in synchronizing data feeds.