Problem with Datafeed, DataFiller, and Indicators
-
In my strategy I'm using a 1-second datafeed with multiple indicators attached to it. The datafeed works well during periods of high activity but goes for more than 10 seconds at a time without giving OHLC info during low activity periods. I attempted to apply the DataFiller to it but I'm still seeing the large gaps in my live output. Could someone please tell me what steps I'm missing or what mistake I'm making?
cerebro = backtrader.Cerebro() ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) live_data.addfilter(btfilters.DataFiller) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
Here's a sample of the collected live data. The main problem is that even though this was a 40 second window, my indicators (mostly moving averages) interpreted it as 14 seconds because it has 14 data points. I'm hoping that using the DataFiller correctly fixes this.
2021-03-03T15:57:05, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 2021-03-03T15:57:14, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 2021-03-03T15:57:16, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 2021-03-03T15:57:22, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 2021-03-03T15:57:24, Open: 3813.5000, High: 3813.7500, Low: 3813.5000, Close: 3813.7500 2021-03-03T15:57:26, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 2021-03-03T15:57:27, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 2021-03-03T15:57:29, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 2021-03-03T15:57:30, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 2021-03-03T15:57:31, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 2021-03-03T15:57:32, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 2021-03-03T15:57:42, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 2021-03-03T15:57:44, Open: 3813.0000, High: 3813.0000, Low: 3813.0000, Close: 3813.0000 2021-03-03T15:57:45, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500
-
Here are some of the things I've tried that didn't work:
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill, qcheck=0.25, ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.adddata(live_data_filled)
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill, qcheck=0.25, ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) live_data.addfilter(btfilters.DataFiller, timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill, qcheck=0.25, ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) live_data.addfilter(btfilters.SessionFiller) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill, qcheck=0.5, filters=[btfilters.SessionFiller] ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
I may have accidentally figured it out just now, and here is my current code:
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill, qcheck=0.5, filters=[btfilters.SessionFiller] ) cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
With some live output:
2021-03-10T17:13:02.136990, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 2021-03-10T17:13:03.136994, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 2021-03-10T17:13:04.136999, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 2021-03-10T17:13:05.136993, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 2021-03-10T17:13:05.744997, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:06.745001, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:07.744996, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:08.150003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:09.150007, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:10.150002, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:11.150007, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:12.150001, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:13.150006, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:14.150000, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:15.150005, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:15.564004, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:16.563998, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:17.564003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:17.667996, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:18.668000, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:18.772003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:19.771998, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 2021-03-10T17:13:20.375998, Open: 3906.5000, High: 3906.5000, Low: 3906.5000, Close: 3906.5000
I'll mess around with qcheck and resampling the feed again to try and clean up those time intervals a bit.
-
I was able to get the SessionFiller working kind of, but even though it's on a one-second timeframe it seems like it's still returning values in ticks. When I tried resampling it again the data wasn't in proper seconds. I've tried the following things and they haven't fixed this.
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, backfill_from=data_backfill filters=[btfilters.SessionFiller] ) live_data.resample(timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.adddata(live_data) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)
^This kept the data in a tick frequency but didn't cause any errors.
if args.live: ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496) cerebro.setbroker(ibstore.getbroker()) data_backfill = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1, historical=True, fromdate=now.date() ) live_data = ibstore.getdata( dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Ticks, backfill_from=data_backfill ) live_data.resample(timeframe=backtrader.TimeFrame.Seconds, compression=1) live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1) cerebro.adddata(live_data_filled) cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2) Traceback (most recent call last): File "./strategies/keltner_scalp.py", line 1588, in <module> live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1) File "/home/colin/Downloads/backtrader-master/backtrader/feed.py", line 311, in clone return DataClone(dataname=self, **kwargs) File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 89, in __call__ _obj, args, kwargs = cls.dopostinit(_obj, *args, **kwargs) File "/home/colin/Downloads/backtrader-master/backtrader/feed.py", line 113, in dopostinit fp = fp(_obj) File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "/home/colin/Downloads/backtrader-master/backtrader/filters/session.py", line 75, in __init__ self._tdframe = self._tdeltas[data._timeframe] KeyError: 1
^There's a thread on this forum where someone tried something like this and they said it worked. How do I fix this?
-
@colin AFAIK IB doesn't provide anything other than ticks or 5sec bars (rtbars) during the live trading:
https://community.backtrader.com/topic/3399/trouble-with-daily-replay/2
As for the usage of SessionFiller - still need to wait for other forum member to answer that one.