3. RollOver resampling issues
For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

RollOver resampling issues

  • M

    Hi,

    I've throughly checked and I could confirm that there is an issue with resampling when RollOver data is used. I'm using three datas,

    data0 bt.feeds.IBData for SPY Index
    data1 bt.feeds.RollOver for SPY OPT Calls
    data2 bt.feeds.RollOver SPY OPT Puts
    For simplicity I added just one strike to each RollOver

    I tested with and without RollOver for data1 and data2 and below are the outputs in each case. I'm printing in ibdata.py in def _load_rtbar so you can see date2num timestamps and match them in each case.

    Issues I observed are

    • Data is processed in batches for each data and not all three datas for every rtbar like in case of second output without RollOver. I checked IB logs and can see that data is sent for all three datas every 5 seconds in both cases. In second output you can see prints for data0, data1 and data2 for every rtbar (5seconds) In case of first ouput with RollOver you can see them in batches

    • Data is being reported a minute later sometime. Check line after 737699.800636574 in both cases. In case of without Rollover Data for 2020-10-02T00:43:00 is being reported correctly with Close as 337.02. Whereas in case of Rollover after the same date2num 737699.800636574 data is being reported for a minute old 2020-10-02T00:42:00 with close as 336.91

    Any help with this bt.feeds.RollOver resampling is much appreciated.

    @backtrader @vladisld Any inputs please

                                          With RollOver Data

2020-10-02T00:41:00, ************************* Data0: Close 337.10 *************************
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01
737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71
737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69
2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************
737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03
737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18

                                            Without RollOver Data

2020-10-02T00:41:00, ************************* Data0: Close 337.10 *************************
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 2.19, High: 2.19, Low: 2.19, Close 2.19
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 2.15, High: 2.17, Low: 2.15, Close 2.15
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99
737699.799537037 Time: 2020-10-02 00:41:20, Open: 2.11, High: 2.11, Low: 2.11, Close 2.11
737699.799537037 Time: 2020-10-02 00:41:20, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85
737699.799826389 Time: 2020-10-02 00:41:45, Open: 2.1, High: 2.1, Low: 2.1, Close 2.1
737699.799826389 Time: 2020-10-02 00:41:45, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 2.07, High: 2.09, Low: 2.07, Close 2.09
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************
737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8 Time: 2020-10-02 00:42:00, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08
737699.8 Time: 2020-10-02 00:42:00, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96
737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71
737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02
737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7
2020-10-02T00:43:00, ************************* Data0: Close 337.02 *************************
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 337.06, High: 337.09, Low: 337.06, Close 337.09
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 337.1, High: 337.1, Low: 337.07, Close 337.1
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 337.1, High: 337.1, Low: 337.03, Close 337.06
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69

    Screen Shot 2020-10-02 at 12.46.09 AM.png
    Screen Shot 2020-10-02 at 12.45.59 AM.png

    0
  • M

    @ab_trader Request your time and inputs.

    0
  • A

    @mv said in RollOver resampling issues:

    Request your time and inputs.

    Honestly, I hardly understand what you are trying to achieve and what is the issue. RollOver data feed is intended to return the single data feed consisting from the slices of number of sequential future contracts. Since you are using options and real time data feed, than it is possible that RollOver data feed is simply not suitable for you.

    0
  • M

    @ab_trader Yes, I'm using RollOver to create a single feed with multiple OPTs like mentioned in the document Rolling over Futures
     and Blog Post

    In my code below, I use data0 for SPY STK and data1 for SPY OPT (Calls) and data2 for SPY OPT (Puts). For testing purposes I just included one option in each RollOver. I use it for multiple options in each Rollover to create a single data feed. In case of Futures we RollOver based on expiration and here I use conditions to RollOver based on price or volume.

    data0 = IBDataFactory(dataname=args.data0, **datakwargs)

data1List = getOptionsList(args.data1)
d1temp = [IBDataFactory(dataname=data1List[i], **datakwargs) for i in range(0, len(data1List))]
data1 = bt.feeds.RollOver(*d1temp, name='Data1-Roll')

data2List = getOptionsList(args.data2)
d2temp = [IBDataFactory(dataname=data2List[i], **datakwargs) for i in range(0, len(data2List))]
data2 = bt.feeds.RollOver(*d2temp, name='Data2-Roll')

    This is working fine and I'm able to fetch and use data but I see resampling issue like I mentioned in my original post. For easier reading the debugs, data0 is in price range 333.xx, data1 is in price 2.xx and data2 is in price 1.xx

    Please let me know if you need further information. Thanks for your time.

    0
  • A

    @mv said in RollOver resampling issues:

    I'm printing in ibdata.py in def _load_rtbar so you can see date2num timestamps and match them in each case.

    If you use bt as expected i.e. print logs in the next(), but not somewhere deep in the bt scripts, will it returns the same issues?

    Also would be nice to have the full script without any excessive details. Resampling was mentioned, but there is nothing shows that it is there.

    0
  • M

    @ab_trader Yes, logs in next() led me to put debugs in bt to check for the issue and I was able to nail it to the case with RollOver included. I see that RollOver is hardly used by anyone especially for Options, so I changed the way of handling this and avoided RollOver in total. Without RollOver data looks fine.

    Appreciate your time and responses. Please continue your outstanding and support and help.

    Thanks

    0
Log in to reply
 
});