Navigation

    Backtrader Community
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Problem with Datafeed, DataFiller, and Indicators

    General Code/Help
    2
    4
    175
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      colin last edited by

      In my strategy I'm using a 1-second datafeed with multiple indicators attached to it. The datafeed works well during periods of high activity but goes for more than 10 seconds at a time without giving OHLC info during low activity periods. I attempted to apply the DataFiller to it but I'm still seeing the large gaps in my live output. Could someone please tell me what steps I'm missing or what mistake I'm making?

          cerebro = backtrader.Cerebro()

    ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
    cerebro.setbroker(ibstore.getbroker())
    data_backfill = ibstore.getdata(
        dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
        historical=True, fromdate=now.date()
    )
    
    live_data = ibstore.getdata(
        dataname=f'ES-{contract_date}-GLOBEX-USD',
        timeframe=backtrader.TimeFrame.Seconds,
        compression=1,
        backfill_from=data_backfill
    )
    
    cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
    live_data.addfilter(btfilters.DataFiller)
    cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

      Here's a sample of the collected live data. The main problem is that even though this was a 40 second window, my indicators (mostly moving averages) interpreted it as 14 seconds because it has 14 data points. I'm hoping that using the DataFiller correctly fixes this.

      2021-03-03T15:57:05, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 
2021-03-03T15:57:14, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 
2021-03-03T15:57:16, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 
2021-03-03T15:57:22, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 
2021-03-03T15:57:24, Open: 3813.5000, High: 3813.7500, Low: 3813.5000, Close: 3813.7500 
2021-03-03T15:57:26, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 
2021-03-03T15:57:27, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 
2021-03-03T15:57:29, Open: 3813.5000, High: 3813.5000, Low: 3813.5000, Close: 3813.5000 
2021-03-03T15:57:30, Open: 3813.7500, High: 3813.7500, Low: 3813.7500, Close: 3813.7500 
2021-03-03T15:57:31, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 
2021-03-03T15:57:32, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 
2021-03-03T15:57:42, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500 
2021-03-03T15:57:44, Open: 3813.0000, High: 3813.0000, Low: 3813.0000, Close: 3813.0000 
2021-03-03T15:57:45, Open: 3813.2500, High: 3813.2500, Low: 3813.2500, Close: 3813.2500
      1 Reply Last reply Reply Quote 0
      • C
        colin last edited by

        Here are some of the things I've tried that didn't work:

            if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill,
            qcheck=0.25,
        )
        cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
        live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.adddata(live_data_filled)

            if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill,
            qcheck=0.25,
        )
        cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
        live_data.addfilter(btfilters.DataFiller, timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

            if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill,
            qcheck=0.25,
        )
        cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
        live_data.addfilter(btfilters.SessionFiller)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

            if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill,
            qcheck=0.5,
            filters=[btfilters.SessionFiller]
        )
        cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

        I may have accidentally figured it out just now, and here is my current code:

            if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Minutes, compression=2,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill,
            qcheck=0.5,
            filters=[btfilters.SessionFiller]
        )

        cerebro.resampledata(live_data, name='ES_second', timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

        With some live output:

        2021-03-10T17:13:02.136990, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 
2021-03-10T17:13:03.136994, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 
2021-03-10T17:13:04.136999, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 
2021-03-10T17:13:05.136993, Open: 3906.0000, High: 3906.0000, Low: 3906.0000, Close: 3906.0000 
2021-03-10T17:13:05.744997, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:06.745001, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:07.744996, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:08.150003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:09.150007, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:10.150002, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:11.150007, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:12.150001, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:13.150006, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:14.150000, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:15.150005, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:15.564004, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:16.563998, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:17.564003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:17.667996, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:18.668000, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:18.772003, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:19.771998, Open: 3906.2500, High: 3906.2500, Low: 3906.2500, Close: 3906.2500 
2021-03-10T17:13:20.375998, Open: 3906.5000, High: 3906.5000, Low: 3906.5000, Close: 3906.5000

        I'll mess around with qcheck and resampling the feed again to try and clean up those time intervals a bit.

        1 Reply Last reply Reply Quote 0
        • C
          colin last edited by

          I was able to get the SessionFiller working kind of, but even though it's on a one-second timeframe it seems like it's still returning values in ticks. When I tried resampling it again the data wasn't in proper seconds. I've tried the following things and they haven't fixed this.

              if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Seconds,
            compression=1,
            backfill_from=data_backfill
            filters=[btfilters.SessionFiller]
        )

        live_data.resample(timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.adddata(live_data)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)

          ^This kept the data in a tick frequency but didn't cause any errors.

              if args.live:
        ibstore = backtrader.stores.IBStore(host='127.0.0.1', port=7497 if args.paper else 7496)
        cerebro.setbroker(ibstore.getbroker())
        data_backfill = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD', timeframe=backtrader.TimeFrame.Seconds, compression=1,
            historical=True, fromdate=now.date()
        )
        live_data = ibstore.getdata(
            dataname=f'ES-{contract_date}-GLOBEX-USD',
            timeframe=backtrader.TimeFrame.Ticks,
            backfill_from=data_backfill
        )
        live_data.resample(timeframe=backtrader.TimeFrame.Seconds, compression=1)
        live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1)
        cerebro.adddata(live_data_filled)
        cerebro.resampledata(live_data, name='ES_two_minute', timeframe=backtrader.TimeFrame.Minutes, compression=2)


Traceback (most recent call last):
  File "./strategies/keltner_scalp.py", line 1588, in <module>
    live_data_filled = live_data.clone(filters=[btfilters.SessionFiller], timeframe=backtrader.TimeFrame.Seconds, compression=1)
  File "/home/colin/Downloads/backtrader-master/backtrader/feed.py", line 311, in clone
    return DataClone(dataname=self, **kwargs)
  File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 89, in __call__
    _obj, args, kwargs = cls.dopostinit(_obj, *args, **kwargs)
  File "/home/colin/Downloads/backtrader-master/backtrader/feed.py", line 113, in dopostinit
    fp = fp(_obj)
  File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 88, in __call__
    _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
  File "/home/colin/Downloads/backtrader-master/backtrader/metabase.py", line 78, in doinit
    _obj.__init__(*args, **kwargs)
  File "/home/colin/Downloads/backtrader-master/backtrader/filters/session.py", line 75, in __init__
    self._tdframe = self._tdeltas[data._timeframe]
KeyError: 1

          ^There's a thread on this forum where someone tried something like this and they said it worked. How do I fix this?

          vladisld 1 Reply Last reply Reply Quote 0
          • vladisld
            vladisld @colin last edited by

            @colin AFAIK IB doesn't provide anything other than ticks or 5sec bars (rtbars) during the live trading:

            https://community.backtrader.com/topic/3399/trouble-with-daily-replay/2

            As for the usage of SessionFiller - still need to wait for other forum member to answer that one.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors