RollOver resampling issues
-
Hi,
I've throughly checked and I could confirm that there is an issue with resampling when RollOver data is used. I'm using three datas,
data0
bt.feeds.IBData
for SPY Index
data1bt.feeds.RollOver
for SPY OPT Calls
data2bt.feeds.RollOver
SPY OPT Puts
For simplicity I added just one strike to each RollOverI tested with and without RollOver for data1 and data2 and below are the outputs in each case. I'm printing in
ibdata.py
indef _load_rtbar
so you can seedate2num
timestamps and match them in each case.Issues I observed are
-
Data is processed in batches for each data and not all three datas for every rtbar like in case of second output without RollOver. I checked IB logs and can see that data is sent for all three datas every 5 seconds in both cases. In second output you can see prints for data0, data1 and data2 for every rtbar (5seconds) In case of first ouput with RollOver you can see them in batches
-
Data is being reported a minute later sometime. Check line after 737699.800636574 in both cases. In case of without Rollover Data for 2020-10-02T00:43:00 is being reported correctly with Close as 337.02. Whereas in case of Rollover after the same date2num 737699.800636574 data is being reported for a minute old 2020-10-02T00:42:00 with close as 336.91
Any help with this
bt.feeds.RollOver
resampling is much appreciated.@backtrader @vladisld Any inputs please
With RollOver Data 2020-10-02T00:41:00, ************************* Data0: Close 337.10 ************************* 737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08 737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06 737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03 737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01 737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99 737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99 737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99 737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96 737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96 737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85 737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89 737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71 737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69 2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************ 737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96 737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03 737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07 737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17 737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18 737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
Without RollOver Data 2020-10-02T00:41:00, ************************* Data0: Close 337.10 ************************* 737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08 737699.7993055555 Time: 2020-10-02 00:41:00, Open: 2.19, High: 2.19, Low: 2.19, Close 2.19 737699.7993055555 Time: 2020-10-02 00:41:00, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69 737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06 737699.7993634259 Time: 2020-10-02 00:41:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17 737699.7993634259 Time: 2020-10-02 00:41:05, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68 737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03 737699.7994212963 Time: 2020-10-02 00:41:10, Open: 2.15, High: 2.17, Low: 2.15, Close 2.15 737699.7994212963 Time: 2020-10-02 00:41:10, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68 737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01 737699.7994791666 Time: 2020-10-02 00:41:15, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15 737699.7994791666 Time: 2020-10-02 00:41:15, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68 737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99 737699.799537037 Time: 2020-10-02 00:41:20, Open: 2.11, High: 2.11, Low: 2.11, Close 2.11 737699.799537037 Time: 2020-10-02 00:41:20, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99 737699.7995949074 Time: 2020-10-02 00:41:25, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.7995949074 Time: 2020-10-02 00:41:25, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99 737699.7996527777 Time: 2020-10-02 00:41:30, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.7996527777 Time: 2020-10-02 00:41:30, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96 737699.7997106481 Time: 2020-10-02 00:41:35, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.7997106481 Time: 2020-10-02 00:41:35, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96 737699.7997685185 Time: 2020-10-02 00:41:40, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.7997685185 Time: 2020-10-02 00:41:40, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85 737699.799826389 Time: 2020-10-02 00:41:45, Open: 2.1, High: 2.1, Low: 2.1, Close 2.1 737699.799826389 Time: 2020-10-02 00:41:45, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72 737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89 737699.7998842592 Time: 2020-10-02 00:41:50, Open: 2.07, High: 2.09, Low: 2.07, Close 2.09 737699.7998842592 Time: 2020-10-02 00:41:50, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76 737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91 737699.7999421296 Time: 2020-10-02 00:41:55, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08 737699.7999421296 Time: 2020-10-02 00:41:55, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76 2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************ 737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91 737699.8 Time: 2020-10-02 00:42:00, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08 737699.8 Time: 2020-10-02 00:42:00, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09 737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12 737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96 737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13 737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71 737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02 737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15 737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7 2020-10-02T00:43:00, ************************* Data0: Close 337.02 ************************* 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15 737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69 737699.8007523149 Time: 2020-10-02 00:43:05, Open: 337.06, High: 337.09, Low: 337.06, Close 337.09 737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17 737699.8007523149 Time: 2020-10-02 00:43:05, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69 737699.8008101851 Time: 2020-10-02 00:43:10, Open: 337.1, High: 337.1, Low: 337.07, Close 337.1 737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18 737699.8008101851 Time: 2020-10-02 00:43:10, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69 737699.8008680556 Time: 2020-10-02 00:43:15, Open: 337.1, High: 337.1, Low: 337.03, Close 337.06 737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18 737699.8008680556 Time: 2020-10-02 00:43:15, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
-
-
@ab_trader Request your time and inputs.
-
@mv said in RollOver resampling issues:
Request your time and inputs.
Honestly, I hardly understand what you are trying to achieve and what is the issue.
RollOver
data feed is intended to return the single data feed consisting from the slices of number of sequential future contracts. Since you are using options and real time data feed, than it is possible thatRollOver
data feed is simply not suitable for you. -
@ab_trader Yes, I'm using RollOver to create a single feed with multiple OPTs like mentioned in the document Rolling over Futures
and Blog PostIn my code below, I use
data0
for SPY STK anddata1
for SPY OPT (Calls) anddata2
for SPY OPT (Puts). For testing purposes I just included one option in each RollOver. I use it for multiple options in each Rollover to create a single data feed. In case of Futures we RollOver based on expiration and here I use conditions to RollOver based on price or volume.data0 = IBDataFactory(dataname=args.data0, **datakwargs) data1List = getOptionsList(args.data1) d1temp = [IBDataFactory(dataname=data1List[i], **datakwargs) for i in range(0, len(data1List))] data1 = bt.feeds.RollOver(*d1temp, name='Data1-Roll') data2List = getOptionsList(args.data2) d2temp = [IBDataFactory(dataname=data2List[i], **datakwargs) for i in range(0, len(data2List))] data2 = bt.feeds.RollOver(*d2temp, name='Data2-Roll')
This is working fine and I'm able to fetch and use data but I see resampling issue like I mentioned in my original post. For easier reading the debugs,
data0
is in price range 333.xx,data1
is in price 2.xx anddata2
is in price 1.xxPlease let me know if you need further information. Thanks for your time.
-
@mv said in RollOver resampling issues:
I'm printing in ibdata.py in def _load_rtbar so you can see date2num timestamps and match them in each case.
If you use
bt
as expected i.e. print logs in thenext()
, but not somewhere deep in thebt
scripts, will it returns the same issues?Also would be nice to have the full script without any excessive details. Resampling was mentioned, but there is nothing shows that it is there.
-
@ab_trader Yes, logs in
next()
led me to put debugs inbt
to check for the issue and I was able to nail it to the case with RollOver included. I see that RollOver is hardly used by anyone especially for Options, so I changed the way of handling this and avoided RollOver in total. Without RollOver data looks fine.Appreciate your time and responses. Please continue your outstanding and support and help.
Thanks