3. LIVE data resampling issue
For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
2019-10-02: The community is currently in read-only mode

LIVE data resampling issue

  • S

    Hello all, I got ran into an issue with live data resampling from 1 min to 1 day. As you can see from some of my results below, the resampling works very well during backfilling, as well as resampling from 1min to 10min or 30min. However, it ran into this issue, the 1day timeframe data feeds did not get updated to the previous day's value.

    My code for getdata() and resampling:

    data = ibstore.getdata(dataname='TSLA-STK-SMART-USD', timeframe = bt.TimeFrame.Minutes, compression =1, rtbar=False,
                       qcheck=1.0, backfill_start=True, backfill=True, backfill_from = dataHist, useRTH=True, latethrough =False,
                       sessionstart=datetime.time(9,30,00), sessionend=datetime.time(16,00,00))


data.addfilter(bt.filters.SessionFilter(data))

cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1) ## Adding 1min bar （data0)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=10) ## Adding 10min bar (data1)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=30) ## Adding 30min bar (data2)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1) ## Adding 1day bar (data3)

    datafeeds during backfilling:

    .......
    next run started @ 2019-02-15T09:01:17.282031
    Data0, 24209, 2019-02-13T15:58:00.000000, 308.53, 308.55, 308.14, 308.34, 366.0,
    Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0,
    Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
    Data3, 0062, 2019-02-12T16:00:00.000000, 316.2, 318.19, 309.62, 311.81, 41598.0,

    next run started @ 2019-02-15T09:01:17.332751
    Data0, 24210, 2019-02-13T15:59:00.000000, 308.36, 308.43, 308.0, 308.11, 273.0, nan
    Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0, nan
    Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
    Data3, 0062, 2019-02-12T16:00:00.000000, 316.2, 318.19, 309.62, 311.81, 41598.0,

    next run started @ 2019-02-15T09:01:17.386498
    Data0, 24210, 2019-02-13T15:59:00.000000, 308.36, 308.43, 308.0, 308.11, 273.0, nan
    Data1, 2421, 2019-02-13T15:50:00.000000, 307.97, 307.97, 307.37, 307.92, 1027.0, nan
    Data2, 0807, 2019-02-13T15:30:00.000000, 309.31, 309.47, 307.79, 307.91, 1514.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.

    next run started @ 2019-02-15T09:01:17.435305
    Data0, 24211, 2019-02-14T09:30:00.000000, 303.38, 304.62, 302.95, 304.1, 1294.0, nan
    Data1, 2422, 2019-02-14T09:30:00.000000, 307.99, 308.75, 302.95, 304.1, 3439.0, -1.0
    Data2, 0808, 2019-02-14T09:30:00.000000, 307.81, 308.75, 302.95, 304.1, 4910.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

    next run started @ 2019-02-15T09:01:17.481209
    Data0, 24212, 2019-02-14T09:31:00.000000, 304.11, 304.63, 303.21, 303.38, 387.0, nan
    Data1, 2422, 2019-02-14T09:30:00.000000, 307.99, 308.75, 302.95, 304.1, 3439.0, -1.0
    Data2, 0808, 2019-02-14T09:30:00.000000, 307.81, 308.75, 302.95, 304.1, 4910.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,
    .......

    Resampling while turn into live:

    .......
    next run started @ 2019-02-15T09:01:39.194076
    Data0, 24600, 2019-02-14T15:59:00.000000, 304.0, 304.05, 303.35, 303.83, 635.0, nan
    Data1, 2460, 2019-02-14T15:50:00.000000, 304.6, 304.86, 304.34, 304.42, 907.0, nan
    Data2, 0820, 2019-02-14T15:30:00.000000, 304.37, 305.87, 304.2, 304.88, 1797.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

    ***** DATA NOTIF: DELAYED
    ***** DATA NOTIF: LIVE
    next run started @ 2019-02-15T09:30:05.705076
    Data0, 24601, 2019-02-15T09:30:00.000000, 304.4, 304.56, 304.15, 304.39, 681.0, nan
    Data1, 2461, 2019-02-15T09:30:00.000000, 304.44, 304.56, 303.35, 304.39, 2784.0, nan
    Data2, 0821, 2019-02-15T09:30:00.000000, 304.95, 305.09, 303.35, 304.39, 4159.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,

    next run started @ 2019-02-15T09:31:01.533144
    Data0, 24602, 2019-02-15T09:31:00.000000, 304.31, 304.53, 304.09, 304.51, 204.0, nan
    Data1, 2461, 2019-02-15T09:30:00.000000, 304.44, 304.56, 303.35, 304.39, 2784.0, nan
    Data2, 0821, 2019-02-15T09:30:00.000000, 304.95, 305.09, 303.35, 304.39, 4159.0
    Data3, 0063, 2019-02-13T16:00:00.000000, 312.35, 312.75, 305.57, 308.11, 38587.0,
    .......

    One can see the daily bar(Data3) is not getting update to the correct timestamp. For the information, the dataHist I am backfilling from is the RTH 1min bar stored locally, it will get update once a day after market close, so it alwasy up to yesterday's 15:59:00.

    I really appreciate if you could provide any guidance.

    0
    S
     1 Reply
  • S

    @sylvain I think one possible solution can be: change the timestamp of the historical data to 1 minute later. this would make sense since, IB 1 min hist bar starts at 9:30 ends at 15:59, and after the shift, it will starts at 9:31 ends at 16:00, this will line up with the resampled 1 min live bar. However still unclear about the issue mentioned above.
    Also, it looks like the first 1min LIVE bar of the day is:

    next run started @ 2019-02-15T09:30:05.705076
    Data0, 24601, 2019-02-15T09:30:00.000000, 304.4, 304.56, 304.15, 304.39, 681.0, nan

    should it be at 9:31:XX ?
    Since the 1min bar would have enough info only after 1min of the mark opened, really not sure how to interpret this. Any help would be appreciated @backtrader

    0
    S
     1 Reply
  • S

    @sylvain Problem solved after making the change of shift the timestamp of hist 1 min bar to 1 min later. Thank you !

    0
Log in to reply
 
});