@Atul-Patel were you able to get this done for IEX data? How did it compare to IB data feed? Did you find any issues with BT resampling for indicators as values are off like mentioned in my others posts.
Any updates on alternate and reliable data feed for Backtrader would be very much appreciated
Thanks
@ab_trader Yes, logs in next() led me to put debugs in bt to check for the issue and I was able to nail it to the case with RollOver included. I see that RollOver is hardly used by anyone especially for Options, so I changed the way of handling this and avoided RollOver in total. Without RollOver data looks fine.
Appreciate your time and responses. Please continue your outstanding and support and help.
Thanks
@ab_trader Yes, I'm using RollOver to create a single feed with multiple OPTs like mentioned in the document Rolling over Futures
and Blog Post
In my code below, I use data0 for SPY STK and data1 for SPY OPT (Calls) and data2 for SPY OPT (Puts). For testing purposes I just included one option in each RollOver. I use it for multiple options in each Rollover to create a single data feed. In case of Futures we RollOver based on expiration and here I use conditions to RollOver based on price or volume.
data0 = IBDataFactory(dataname=args.data0, **datakwargs)
data1List = getOptionsList(args.data1)
d1temp = [IBDataFactory(dataname=data1List[i], **datakwargs) for i in range(0, len(data1List))]
data1 = bt.feeds.RollOver(*d1temp, name='Data1-Roll')
data2List = getOptionsList(args.data2)
d2temp = [IBDataFactory(dataname=data2List[i], **datakwargs) for i in range(0, len(data2List))]
data2 = bt.feeds.RollOver(*d2temp, name='Data2-Roll')
This is working fine and I'm able to fetch and use data but I see resampling issue like I mentioned in my original post. For easier reading the debugs, data0 is in price range 333.xx, data1 is in price 2.xx and data2 is in price 1.xx
Please let me know if you need further information. Thanks for your time.
Hi,
I've throughly checked and I could confirm that there is an issue with resampling when RollOver data is used. I'm using three datas,
data0 bt.feeds.IBData for SPY Index
data1 bt.feeds.RollOver for SPY OPT Calls
data2 bt.feeds.RollOver SPY OPT Puts
For simplicity I added just one strike to each RollOver
I tested with and without RollOver for data1 and data2 and below are the outputs in each case. I'm printing in ibdata.py in def _load_rtbar so you can see date2num timestamps and match them in each case.
Issues I observed are
Data is processed in batches for each data and not all three datas for every rtbar like in case of second output without RollOver. I checked IB logs and can see that data is sent for all three datas every 5 seconds in both cases. In second output you can see prints for data0, data1 and data2 for every rtbar (5seconds) In case of first ouput with RollOver you can see them in batches
Data is being reported a minute later sometime. Check line after 737699.800636574 in both cases. In case of without Rollover Data for 2020-10-02T00:43:00 is being reported correctly with Close as 337.02. Whereas in case of Rollover after the same date2num 737699.800636574 data is being reported for a minute old 2020-10-02T00:42:00 with close as 336.91
Any help with this bt.feeds.RollOver resampling is much appreciated.
@backtrader @vladisld Any inputs please
With RollOver Data
2020-10-02T00:41:00, ************************* Data0: Close 337.10 *************************
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01
737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71
737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69
2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************
737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03
737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
Without RollOver Data
2020-10-02T00:41:00, ************************* Data0: Close 337.10 *************************
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 337.09, High: 337.09, Low: 337.07, Close 337.08
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 2.19, High: 2.19, Low: 2.19, Close 2.19
737699.7993055555 Time: 2020-10-02 00:41:00, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 337.07, High: 337.07, Low: 337.05, Close 337.06
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.7993634259 Time: 2020-10-02 00:41:05, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 337.06, High: 337.08, Low: 337.03, Close 337.03
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 2.15, High: 2.17, Low: 2.15, Close 2.15
737699.7994212963 Time: 2020-10-02 00:41:10, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 337.02, High: 337.02, Low: 337.0, Close 337.01
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.7994791666 Time: 2020-10-02 00:41:15, Open: 1.68, High: 1.68, Low: 1.68, Close 1.68
737699.799537037 Time: 2020-10-02 00:41:20, Open: 337.01, High: 337.01, Low: 336.97, Close 336.99
737699.799537037 Time: 2020-10-02 00:41:20, Open: 2.11, High: 2.11, Low: 2.11, Close 2.11
737699.799537037 Time: 2020-10-02 00:41:20, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 336.98, High: 336.99, Low: 336.97, Close 336.99
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.7995949074 Time: 2020-10-02 00:41:25, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 337.0, High: 337.01, Low: 336.99, Close 336.99
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.7996527777 Time: 2020-10-02 00:41:30, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 337.0, High: 337.0, Low: 336.94, Close 336.96
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.7997106481 Time: 2020-10-02 00:41:35, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.7997685185 Time: 2020-10-02 00:41:40, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.799826389 Time: 2020-10-02 00:41:45, Open: 336.95, High: 336.96, Low: 336.85, Close 336.85
737699.799826389 Time: 2020-10-02 00:41:45, Open: 2.1, High: 2.1, Low: 2.1, Close 2.1
737699.799826389 Time: 2020-10-02 00:41:45, Open: 1.72, High: 1.72, Low: 1.72, Close 1.72
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 336.86, High: 336.92, Low: 336.86, Close 336.89
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 2.07, High: 2.09, Low: 2.07, Close 2.09
737699.7998842592 Time: 2020-10-02 00:41:50, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 336.89, High: 336.91, Low: 336.88, Close 336.91
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08
737699.7999421296 Time: 2020-10-02 00:41:55, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
2020-10-02T00:42:00, ************************* Data0: Close 336.91 ************************
737699.8 Time: 2020-10-02 00:42:00, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8 Time: 2020-10-02 00:42:00, Open: 2.08, High: 2.08, Low: 2.08, Close 2.08
737699.8 Time: 2020-10-02 00:42:00, Open: 1.76, High: 1.76, Low: 1.76, Close 1.76
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 336.91, High: 336.92, Low: 336.9, Close 336.91
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 2.1, High: 2.1, Low: 2.09, Close 2.09
737699.8000578703 Time: 2020-10-02 00:42:05, Open: 1.77, High: 1.77, Low: 1.77, Close 1.77
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 336.9, High: 336.99, Low: 336.88, Close 336.98
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001157407 Time: 2020-10-02 00:42:10, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 336.98, High: 336.99, Low: 336.95, Close 336.97
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8001736111 Time: 2020-10-02 00:42:15, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 336.97, High: 336.98, Low: 336.96, Close 336.98
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8002314814 Time: 2020-10-02 00:42:20, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 336.98, High: 337.0, Low: 336.97, Close 336.99
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8002893518 Time: 2020-10-02 00:42:25, Open: 1.75, High: 1.75, Low: 1.75, Close 1.75
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 336.98, High: 336.99, Low: 336.95, Close 336.95
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8003472222 Time: 2020-10-02 00:42:30, Open: 1.73, High: 1.74, Low: 1.73, Close 1.74
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 336.94, High: 336.97, Low: 336.94, Close 336.96
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 2.12, High: 2.12, Low: 2.12, Close 2.12
737699.8004050925 Time: 2020-10-02 00:42:35, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.800462963 Time: 2020-10-02 00:42:40, Open: 336.96, High: 336.98, Low: 336.95, Close 336.96
737699.800462963 Time: 2020-10-02 00:42:40, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.800462963 Time: 2020-10-02 00:42:40, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 336.98, High: 336.99, Low: 336.97, Close 336.98
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005208333 Time: 2020-10-02 00:42:45, Open: 1.74, High: 1.74, Low: 1.74, Close 1.74
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 336.99, High: 337.03, Low: 336.99, Close 337.03
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 2.13, High: 2.13, Low: 2.13, Close 2.13
737699.8005787038 Time: 2020-10-02 00:42:50, Open: 1.71, High: 1.71, Low: 1.71, Close 1.71
737699.800636574 Time: 2020-10-02 00:42:55, Open: 337.03, High: 337.03, Low: 337.01, Close 337.02
737699.800636574 Time: 2020-10-02 00:42:55, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.800636574 Time: 2020-10-02 00:42:55, Open: 1.71, High: 1.71, Low: 1.7, Close 1.7
2020-10-02T00:43:00, ************************* Data0: Close 337.02 *************************
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 337.03, High: 337.07, Low: 337.03, Close 337.07
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 2.15, High: 2.15, Low: 2.15, Close 2.15
737699.8006944444 Time: 2020-10-02 00:43:00, Open: 1.7, High: 1.7, Low: 1.69, Close 1.69
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 337.06, High: 337.09, Low: 337.06, Close 337.09
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 2.17, High: 2.17, Low: 2.17, Close 2.17
737699.8007523149 Time: 2020-10-02 00:43:05, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 337.1, High: 337.1, Low: 337.07, Close 337.1
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008101851 Time: 2020-10-02 00:43:10, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 337.1, High: 337.1, Low: 337.03, Close 337.06
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 2.18, High: 2.18, Low: 2.18, Close 2.18
737699.8008680556 Time: 2020-10-02 00:43:15, Open: 1.69, High: 1.69, Low: 1.69, Close 1.69
@vladisld Even with this fix, I still see the issue. Worked once and failed twice.
@vladisld, Can you please help me with this issue. This is becoming a deal breaker as Indicator values are key to any automated trading systems. Do you use IB data or some other data source with backtrader? Did you notice any discrepancies with indicator values?
@vladisld In that case, I could have picked up wrong files. Anyways, good to hear that this is the solution for the issue. Thanks for your continued support. Appreciate!
Hi @vladisld,
I've added self.canceldata() before resubscribing and this fix seems to work. You might want to incorporate this in your patch after testing.
elif msg == -10225: # Bust event occurred, current subscription is deactivated.
self._subcription_valid = False
if not self._statelivereconn:
self._statelivereconn = self.p.backfill
self.canceldata() # Cancel before resubscribing
self.reqdata() # resubscribe
continue
Thanks @vladisld for your inputs.
I tried with the files you mentioned but it didn't help, still facing the issue now and then. Will look forward to your testing and patch. I can help you with testing too.
Thanks
@Atul-Patel were you able to get this done for IEX data? How did it compare to IB data feed? Did you find any issues with BT resampling for indicators as values are off like mentioned in my others posts.
Any updates on alternate and reliable data feed for Backtrader would be very much appreciated
Thanks