Resample and fill gaps



  • When using resampled oanda data together with SessionFiller resampling does not seem to be working well, as soon as data goes live and starts returning tick data. Am i doing something wrong?

    Version: 1.9.57.122

    Sample code of data:

    datakwargs = dict(
        timeframe=bt.TimeFrame.Seconds, compression=5,
        qcheck=0.5,
        bidask=True,
        backfill_start=True,
        backfill=True,
        bar2edge=True,
        adjbartime=True,
        rightedge=True,
        filters=[btfilters.SessionFiller]
    )
    data = DataCls(dataname="EUR_USD", **datakwargs)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Seconds, compression=5)
    

    Output live data:

    Status 4 LIVE
    ---------- NEXT 1.16544 2017-07-25T17:20:25.000000 1
    ---------- NEXT 1.16544 2017-07-25T17:20:30.000000 2
    ---------- NEXT 1.16541 2017-07-25T17:20:30.832343 3
    ---------- NEXT 1.16541 2017-07-25T17:20:35.832346 4
    ---------- NEXT 1.16541 2017-07-25T17:20:40.832339 5
    ---------- NEXT 1.1654 2017-07-25T17:20:43.781207 6
    ---------- NEXT 1.1654 2017-07-25T17:20:45.000000 7
    ---------- NEXT 1.16533 2017-07-25T17:20:50.000000 8
    ---------- NEXT 1.16536 2017-07-25T17:20:55.000000 9
    ---------- NEXT 1.1654 2017-07-25T17:21:00.000000 10
    ---------- NEXT 1.16537 2017-07-25T17:21:05.000000 11
    ---------- NEXT 1.16536 2017-07-25T17:21:10.000000 12
    ---------- NEXT 1.16536 2017-07-25T17:21:10.311147 13
    ---------- NEXT 1.16536 2017-07-25T17:21:12.540737 14
    ---------- NEXT 1.16537 2017-07-25T17:21:20.000000 15
    ---------- NEXT 1.16537 2017-07-25T17:21:21.190720 16
    ---------- NEXT 1.16537 2017-07-25T17:21:22.798718 17
    ---------- NEXT 1.16538 2017-07-25T17:21:25.000000 18
    ---------- NEXT 1.16535 2017-07-25T17:21:30.000000 19
    ---------- NEXT 1.16535 2017-07-25T17:21:30.438075 20
    ---------- NEXT 1.16537 2017-07-25T17:21:30.705756 21
    


  • On the backfilled data, which has the right timeframe, the SessionFiller works as expected. Only when there is tick data to be resampled and some period is not changing it occurs.



  • some more output from session.py - _fillbar, in the output the is dtime, bar

    i have removed the return value from call, this at least won't output the data not yet resampled

    Status 4 LIVE
    ---------- NEXT 1.16282 2017-07-26T06:31:45.000000 1
    fillbar 2017-07-26 06:31:48.157822 [1.16282, 1.16282, 1.16282, 1.16282, nan, nan, 736536.27208516]
    ---------- NEXT 1.16281 2017-07-26T06:31:50.000000 2
    ---------- NEXT 1.16287 2017-07-26T06:31:55.000000 3
    ---------- NEXT 1.1629 2017-07-26T06:32:00.000000 4
    fillbar 2017-07-26 06:32:03.174930 [1.1629, 1.1629, 1.1629, 1.1629, nan, nan, 736536.2722589691]
    fillbar 2017-07-26 06:32:08.174930 [1.1629, 1.1629, 1.1629, 1.1629, nan, nan, 736536.2723168394]
    ---------- NEXT 1.1629 2017-07-26T06:32:05.000000 5
    ---------- NEXT 1.1629 2017-07-26T06:32:10.000000 6
    ---------- NEXT 1.16296 2017-07-26T06:32:15.000000 7
    ---------- NEXT 1.16304 2017-07-26T06:32:20.000000 8
    fillbar 2017-07-26 06:32:24.404175 [1.16304, 1.16304, 1.16304, 1.16304, nan, nan, 736536.272504678]
    ---------- NEXT 1.16304 2017-07-26T06:32:25.000000 9
    ---------- NEXT 1.16297 2017-07-26T06:32:30.000000 10
    ---------- NEXT 1.16295 2017-07-26T06:32:35.000000 11
    ---------- NEXT 1.16288 2017-07-26T06:32:40.000000 12
    ---------- NEXT 1.1628 2017-07-26T06:32:45.000000 13
    ---------- NEXT 1.16284 2017-07-26T06:32:50.000000 14
    ---------- NEXT 1.16284 2017-07-26T06:32:55.000000 15
    ---------- NEXT 1.16285 2017-07-26T06:33:00.000000 16
    ---------- NEXT 1.16284 2017-07-26T06:33:05.000000 17
    ---------- NEXT 1.16279 2017-07-26T06:33:10.000000 18
    fillbar 2017-07-26 06:33:12.650997 [1.16279, 1.16279, 1.16279, 1.16279, nan, nan, 736536.2730630903]
    ---------- NEXT 1.1628 2017-07-26T06:33:15.000000 19
    ---------- NEXT 1.16275 2017-07-26T06:33:20.000000 20
    


  • @backtrader maybe the rewriter could remove not yet rewritten data and collect it until the timeframe is due and add it then back to data?



  • i fixed it that way after looking into rewriter code:

    diff backtrader_orig/filters/session.py backtrader/filters/session.py
    140,141c140,142
    <         return ret
    < 
    ---
    >         if not data.resampling and not data.replaying:
    >             return ret
    >             
    183d183
    < 
    

  • administrators

    It seems you want to fill data in the resampled timeframe, as such:

    • Did you try adding the filter to the resampled data and not the original (which is tick based before going into the resampler) data?


  • I also tried to this the way below, but the filter is not working that way.

    datakwargs = dict(
        timeframe=bt.TimeFrame.Seconds, compression=5,
        qcheck=0.5,
        bidask=True,
        backfill_start=True,
        backfill=True,
        bar2edge=True,
        adjbartime=True,
        rightedge=True,
    )
    data = DataCls(dataname="EUR_USD", **datakwargs)
    data.resample(timeframe=bt.TimeFrame.Seconds, compression=5)
    data.addfilter(btfilters.SessionFiller)
    cerebro.adddata(data)
    

    i think this would work and is what you meant:

    datakwargs = dict(
        timeframe=bt.TimeFrame.Seconds, compression=5,
        qcheck=0.5,
        bidask=True,
        backfill_start=True,
        backfill=True,
        bar2edge=True,
        adjbartime=True,
        rightedge=True,
    )
    data = DataCls(dataname="EUR_USD", **datakwargs)
    data1 = cerebro.resampledata(data, timeframe=bt.TimeFrame.Seconds, compression=5)
    data1.addfilter(btfilters.SessionFiller)
    cerebro.adddata(data1)
    

    i will test this when there are gaps in the 5 seconds data.



  • When using 2 data sources, nothing happens with data1

    Status 4 LIVE
    ---------- NEXT 8.549 2017-07-26T10:34:55.000000 1
    ---------- NEXT DATA1 8.549 2017-07-26T10:34:55.000000 1
    ---------- NEXT 8.55 2017-07-26T10:37:10.000000 2
    ---------- NEXT DATA1 8.55 2017-07-26T10:37:10.000000 2
    ---------- NEXT 8.549 2017-07-26T10:37:35.000000 3
    ---------- NEXT DATA1 8.549 2017-07-26T10:37:35.000000 3
    ---------- NEXT 8.549 2017-07-26T10:37:45.000000 4
    ---------- NEXT DATA1 8.549 2017-07-26T10:37:45.000000 4
    ---------- NEXT 8.546 2017-07-26T10:37:55.000000 5
    ---------- NEXT DATA1 8.546 2017-07-26T10:37:55.000000 5
    ---------- NEXT 8.548 2017-07-26T10:38:05.000000 6
    ---------- NEXT DATA1 8.548 2017-07-26T10:38:05.000000 6
    


  • By doing it the way below, now data0 repeats data. while data1 seems fine.

    datakwargs = dict(
        timeframe=bt.TimeFrame.Seconds, compression=5,
        qcheck=0.5,
        bidask=True,
        backfill_start=True,
        backfill=True,
        bar2edge=True,
        adjbartime=True,
        rightedge=True,
    )
    data = DataCls(dataname="AUD_SGD", **datakwargs)
    data.resample(timeframe=bt.TimeFrame.Seconds, compression=5)
    data1 = data.clone(filters=[btfilters.SessionFiller],timeframe=bt.TimeFrame.Seconds, compression=5)
    cerebro.adddata(data)
    cerebro.adddata(data1)
    
    Status 4 LIVE
    ---------- NEXT DATA0: 1.07827 2017-07-26T10:55:40.000000 1
    ---------- NEXT DATA1: 1.07827 2017-07-26T10:55:40.000000 1
    ---------- NEXT DATA0: 1.07825 2017-07-26T10:55:45.000000 2
    ---------- NEXT DATA1: 1.07825 2017-07-26T10:55:45.000000 2
    ---------- NEXT DATA0: 1.0781 2017-07-26T10:55:50.000000 3
    ---------- NEXT DATA1: 1.0781 2017-07-26T10:55:50.000000 3
    ---------- NEXT DATA0: 1.07814 2017-07-26T10:55:55.000000 4
    ---------- NEXT DATA1: 1.07814 2017-07-26T10:55:55.000000 4
    ---------- NEXT DATA0: 1.07813 2017-07-26T10:56:00.000000 5
    ---------- NEXT DATA1: 1.07813 2017-07-26T10:56:00.000000 5
    ---------- NEXT DATA0: 1.07835 2017-07-26T10:56:05.000000 6
    ---------- NEXT DATA1: 1.07835 2017-07-26T10:56:05.000000 6
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 7
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:10.000000 7
    fillbar
    fillbar
    fillbar
    fillbar
    fillbar
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 8
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:15.000000 8
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 9
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:20.000000 9
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 10
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:25.000000 10
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 11
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:30.000000 11
    ---------- NEXT DATA0: 1.0783 2017-07-26T10:56:10.000000 12
    ---------- NEXT DATA1: 1.0783 2017-07-26T10:56:35.000000 12
    ---------- NEXT DATA0: 1.07832 2017-07-26T10:56:40.000000 13
    ---------- NEXT DATA1: 1.07832 2017-07-26T10:56:40.000000 13
    ---------- NEXT DATA0: 1.0784 2017-07-26T10:56:45.000000 14
    ---------- NEXT DATA1: 1.0784 2017-07-26T10:56:45.000000 14
    


  • @backtrader my intention is to get 5 seconds data without gaps from oanda.

    When just adding the cloned data, then there are errors. By just using resample the data may have gaps.

    When adding the SessionFiller to the tick data source being resampled, it will output the unsampled data waiting.

    With the hack in SessionFiller with not returning anything, the filled data is 5 seconds away from last tick, but not from last resampled bar.

    By adding a second data source like in the last post, then data0 has duplicate data in next while data1 seems fine.



  • So after playing with live data around, the only solution with oanda data, that seems to be working would be the way below.

    If anyone is trying to achieve the same, don't change the session.py as i wrote above since this will most probably generate wrong data.

    When doing this, you will need to use data[1] in your strategies.

    datakwargs = dict(
        timeframe=bt.TimeFrame.Seconds, compression=5,
        qcheck=0.5,
        bidask=True,
        backfill_start=True,
        backfill=True,
        bar2edge=True,
        adjbartime=True,
        rightedge=True,
    )
    data = store.getdata(dataname="EUR_USD", **datakwargs)
    data.resample(timeframe=bt.TimeFrame.Seconds, compression=5)
    data_filled = data.clone(filters=[btfilters.SessionFiller], timeframe=bt.TimeFrame.Seconds, compression=5)
    cerebro.adddata(data)
    cerebro.adddata(data_filled)
    

Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.