LIVE data resampling issue

sylvain

Hello all, I got ran into an issue with live data resampling from 1 min to 1 day. As you can see from some of my results below, the resampling works very well during backfilling, as well as resampling from 1min to 10min or 30min. However, it ran into this issue, the 1day timeframe data feeds did not get updated to the previous day's value.

My code for getdata() and resampling:

data = ibstore.getdata(dataname='TSLA-STK-SMART-USD', timeframe = bt.TimeFrame.Minutes, compression =1, rtbar=False,
                       qcheck=1.0, backfill_start=True, backfill=True, backfill_from = dataHist, useRTH=True, latethrough =False,
                       sessionstart=datetime.time(9,30,00), sessionend=datetime.time(16,00,00))


data.addfilter(bt.filters.SessionFilter(data))

cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1) ## Adding 1min bar （data0)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=10) ## Adding 10min bar (data1)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=30) ## Adding 30min bar (data2)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1) ## Adding 1day bar (data3)

datafeeds during backfilling:

.......
next run started @ 2019-02-15T09:01:17.282031
Data0, 24209, 2019-02-13T15:58:00.000000, 308.53, 308.55, 308.14, 308.34, 366.0,
Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0,
Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
Data3, 0062, 2019-02-12T16:00:00.000000, 316.2, 318.19, 309.62, 311.81, 41598.0,

next run started @ 2019-02-15T09:01:17.332751
Data0, 24210, 2019-02-13T15:59:00.000000, 308.36, 308.43, 308.0, 308.11, 273.0, nan
Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0, nan
Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
Data3, 0062, 2019-02-12T16:00:00.000000, 316.2, 318.19, 309.62, 311.81, 41598.0,

next run started @ 2019-02-15T09:01:17.386498
Data0, 24210, 2019-02-13T15:59:00.000000, 308.36, 308.43, 308.0, 308.11, 273.0, nan
Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0, nan
Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.

next run started @ 2019-02-15T09:01:17.435305
Data0, 24211, 2019-02-14T09:30:00.000000, 303.38, 304.62, 302.95, 304.1, 1294.0, nan
Data1, 2422, 2019-02-14T09:30:00.000000, 307.99, 308.75, 302.95, 304.1, 3439.0, -1.0
Data2, 0808, 2019-02-14T09:30:00.000000, 307.81, 308.75, 302.95, 304.1, 4910.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

next run started @ 2019-02-15T09:01:17.481209
Data0, 24212, 2019-02-14T09:31:00.000000, 304.11, 304.63, 303.21, 303.38, 387.0, nan
Data1, 2422, 2019-02-14T09:30:00.000000, 307.99, 308.75, 302.95, 304.1, 3439.0, -1.0
Data2, 0808, 2019-02-14T09:30:00.000000, 307.81, 308.75, 302.95, 304.1, 4910.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,
.......

Resampling while turn into live:

.......
next run started @ 2019-02-15T09:01:39.194076
Data0, 24600, 2019-02-14T15:59:00.000000, 304.0, 304.05, 303.35, 303.83, 635.0, nan
Data1, 2460, 2019-02-14T15:50:00.000000, 304.6, 304.86, 304.34, 304.42, 907.0, nan
Data2, 0820, 2019-02-14T15:30:00.000000, 304.37, 305.87, 304.2, 304.88, 1797.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

***** DATA NOTIF: DELAYED
***** DATA NOTIF: LIVE
next run started @ 2019-02-15T09:30:05.705076
Data0, 24601, 2019-02-15T09:30:00.000000, 304.4, 304.56, 304.15, 304.39, 681.0, nan
Data1, 2461, 2019-02-15T09:30:00.000000, 304.44, 304.56, 303.35, 304.39, 2784.0, nan
Data2, 0821, 2019-02-15T09:30:00.000000, 304.95, 305.09, 303.35, 304.39, 4159.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

next run started @ 2019-02-15T09:31:01.533144
Data0, 24602, 2019-02-15T09:31:00.000000, 304.31, 304.53, 304.09, 304.51, 204.0, nan
Data1, 2461, 2019-02-15T09:30:00.000000, 304.44, 304.56, 303.35, 304.39, 2784.0, nan
Data2, 0821, 2019-02-15T09:30:00.000000, 304.95, 305.09, 303.35, 304.39, 4159.0
Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,
.......

One can see the daily bar(Data3) is not getting update to the correct timestamp. For the information, the dataHist I am backfilling from is the RTH 1min bar stored locally, it will get update once a day after market close, so it alwasy up to yesterday's 15:59:00.

I really appreciate if you could provide any guidance.

sylvain

@sylvain I think one possible solution can be: change the timestamp of the historical data to 1 minute later. this would make sense since, IB 1 min hist bar starts at 9:30 ends at 15:59, and after the shift, it will starts at 9:31 ends at 16:00, this will line up with the resampled 1 min live bar. However still unclear about the issue mentioned above.
Also, it looks like the first 1min LIVE bar of the day is:

next run started @ 2019-02-15T09:30:05.705076
Data0, 24601, 2019-02-15T09:30:00.000000, 304.4, 304.56, 304.15, 304.39, 681.0, nan

should it be at 9:31:XX ?
Since the 1min bar would have enough info only after 1min of the mark opened, really not sure how to interpret this. Any help would be appreciated @backtrader

sylvain

@sylvain Problem solved after making the change of shift the timestamp of hist 1 min bar to 1 min later. Thank you !