Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    (**NOT**) Bug: Indicator on higher timeframe is called at every cycle of lower timeframe

    General Code/Help
    2
    3
    464
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • pepelepew25
      pepelepew25 last edited by backtrader

      Hello,

      As reported previously [here] (https://community.backtrader.com/topic/441/psar-indicator-and-multiple-timeframes/7) there seem to be an issue with indicators on resampled data being called too often ?

      Example code below:
      1)load data on smaller timeframe (5min)
      2)resample data on daily
      3)call a strategy with an indicator on daily data
      4)the indicator "next" method is called at every cycle of 5min instead of every day.

      import backtrader as bt
      import logging
      from backtrader.feeds import PandasData
      import pandas as pd
      logging.basicConfig(level=logging.DEBUG, format='%(message)s',filemode='w')
      
      class MyIndicator(bt.Indicator):
          lines = ('et',)
          params = (("period", 100),)
      
          def __init__(self):
              self.prevdate = 0
      
          def next(self):
      
              #if self.data.datetime[0] == self.prevdate:
                  #print("returning as still the same date")
              #    return
              #self.prevdate = self.data.datetime[0]
              self.log("-----new indicator cycle-----------")
      
              #df = bt_to_pandas(self.data, self.p.period)
              #self.log("rows extracted:{}".format(len(df)))
      
          def log(self, txt, dt=None):
              dt = dt or self.data.datetime[0]
              if isinstance(dt, float):
                  dt = bt.num2date(dt)
              logging.info('%s, %s' % (dt.isoformat(), txt))
      
      class MyStrategy(bt.Strategy):
          def __init__(self):
              self.myindic = MyIndicator(self.data1, period=50)
      
          def next(self):
              self.log('strategy next pos {}'.format( self.position.size))
      
          def log(self, txt, dt=None):
              dt = dt or self.data.datetime[0]
              if isinstance(dt, float):
                  dt = bt.num2date(dt)
              logging.debug('%s, %s' % (dt.isoformat(), txt))
      
      def main():
          df = pd.read_pickle("bitmex_BTCUSD_5m_100days.pkl")
          datasmalltf = PandasData(dataname=df, timeframe=bt.TimeFrame.Minutes, compression=5)
      
          cerebro = bt.Cerebro()
      
          cerebro.adddata(datasmalltf)
          datadaily = cerebro.resampledata(datasmalltf, timeframe=bt.TimeFrame.Days, compression=1, name='daily')
      
          cerebro.addstrategy(MyStrategy)
          cerebro.run(runonce=True, stdstats=False)
      
      
      if __name__ == "__main__":
          print('starting main')
          main()
      
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:00:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:05:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:10:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:15:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:20:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:25:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:30:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:35:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:40:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:45:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:50:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T14:55:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:00:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:05:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:10:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:15:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:20:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:25:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:30:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:35:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:40:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      2019-01-02T15:45:00, strategy next pos 0
      2019-01-01T23:59:59.999989, -----new indicator cycle-----------
      

      I have a workaround for this by storing the date in a variable of the indicator and comparing the date at the beginning of the "next" method, but this is dirty

          def __init__(self):
              self.prevdate = 0
      
          def next(self):
              if self.data.datetime[0] == self.prevdate:
                  print("returning as still the same date")
                  return
              self.prevdate = self.data.datetime[0]
              self.log("-----new indicator cycle-----------")
      
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T13:50:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T13:55:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:00:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:05:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:10:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:15:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:20:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      2019-01-14T14:25:00, strategy next pos 0
      2019-01-13T23:59:59.999989, returning as still the same date
      
      B 1 Reply Last reply Reply Quote 0
      • pepelepew25
        pepelepew25 last edited by pepelepew25

        Now I have been using the previous workaround for a while, but found another issue that could be linked to my code but not sure.

        Now same code as above but extracting data at every cycle of the indicator for a period of 20 (last 20 candles)

        The data is extracted correctly for the first 20 cycles but after that the amount of data extracted diminish instead of staying at 20

        At the first cycle there is only one candle to extract, then 2, etc up to 20, but after this we should extract always 20 candles until the end. Instead we have 20,19,18 and so on

        This works correctly with the same code with only 1 timeframe (not using the resampled tf)

        import backtrader as bt
        import logging
        from backtrader.feeds import PandasData
        import pandas as pd
        logging.basicConfig(level=logging.DEBUG, format='%(message)s',filemode='w')
        
        
        class MyIndicator(bt.Indicator):
            lines = ('et',)
            params = (("period", 100),)
        
            def __init__(self):
                self.prevdate = 0
        
            def next(self):
                if self.data.datetime[0] == self.prevdate:
                    #self.log("returning as still the same date")
                    return
                self.prevdate = self.data.datetime[0]
                self.log("-----new indicator cycle-----------")
        
                df = bt_to_pandas(self.data, self.p.period)
                self.log("rows extracted:{}".format(len(df)))
        
            def log(self, txt, dt=None):
                dt = dt or self.data.datetime[0]
                if isinstance(dt, float):
                    dt = bt.num2date(dt)
                logging.info('%s, %s' % (dt.isoformat(), txt))
        
        
        class MyStrategy(bt.Strategy):
            def __init__(self):
                self.myindic = MyIndicator(self.data1, period=20)
        
            def next(self):
                pass
                #self.log('strategy next pos {}'.format( self.position.size))
        
            def log(self, txt, dt=None):
                dt = dt or self.data.datetime[0]
                if isinstance(dt, float):
                    dt = bt.num2date(dt)
                logging.debug('%s, %s' % (dt.isoformat(), txt))
        
        
        def bt_to_pandas(btdata, period):
            logging.debug("getting bt data for period {}".format(period))
            get = lambda mydata: mydata.get(ago=0, size=period)
        
            fields = {
                'Open': get(btdata.open),
                'High': get(btdata.high),
                'Low': get(btdata.low),
                'Close': get(btdata.close),
                'Volume': get(btdata.volume)
            }
            time = [btdata.num2date(x) for x in get(btdata.datetime)]
        
            df = pd.DataFrame(data=fields, index=time)
            logging.debug("data collected for period {}".format(len(df)))
            return df
        
        
        def main():
            df = pd.read_pickle("bitmex_BTCUSD_5m_100days.pkl")
            datasmalltf = PandasData(dataname=df, timeframe=bt.TimeFrame.Minutes, compression=5)
        
            cerebro = bt.Cerebro()
        
            cerebro.adddata(datasmalltf)
            datadaily = cerebro.resampledata(datasmalltf, timeframe=bt.TimeFrame.Days, compression=1, name='daily')
        
            cerebro.addstrategy(MyStrategy)
            cerebro.run(runonce=True, stdstats=False)
        
        
        if __name__ == "__main__":
            print('starting main')
            main()
        
        2019-01-01T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 1
        2019-01-01T23:59:59.999989, rows extracted:1
        2019-01-02T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 2
        2019-01-02T23:59:59.999989, rows extracted:2
        2019-01-03T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 3
        2019-01-03T23:59:59.999989, rows extracted:3
        2019-01-04T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 4
        2019-01-04T23:59:59.999989, rows extracted:4
        2019-01-05T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 5
        2019-01-05T23:59:59.999989, rows extracted:5
        2019-01-06T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 6
        2019-01-06T23:59:59.999989, rows extracted:6
        2019-01-07T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 7
        2019-01-07T23:59:59.999989, rows extracted:7
        2019-01-08T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 8
        2019-01-08T23:59:59.999989, rows extracted:8
        2019-01-09T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 9
        2019-01-09T23:59:59.999989, rows extracted:9
        2019-01-10T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 10
        2019-01-10T23:59:59.999989, rows extracted:10
        2019-01-11T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 9
        2019-01-11T23:59:59.999989, rows extracted:9
        2019-01-12T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 8
        2019-01-12T23:59:59.999989, rows extracted:8
        2019-01-13T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 7
        2019-01-13T23:59:59.999989, rows extracted:7
        2019-01-14T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 6
        2019-01-14T23:59:59.999989, rows extracted:6
        2019-01-15T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 5
        2019-01-15T23:59:59.999989, rows extracted:5
        2019-01-16T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 4
        2019-01-16T23:59:59.999989, rows extracted:4
        2019-01-17T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 3
        2019-01-17T23:59:59.999989, rows extracted:3
        2019-01-18T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 2
        2019-01-18T23:59:59.999989, rows extracted:2
        2019-01-19T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 1
        2019-01-19T23:59:59.999989, rows extracted:1
        2019-01-20T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-20T23:59:59.999989, rows extracted:20
        2019-01-21T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-21T23:59:59.999989, rows extracted:20
        2019-01-22T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-22T23:59:59.999989, rows extracted:20
        2019-01-23T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-23T23:59:59.999989, rows extracted:20
        2019-01-24T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-24T23:59:59.999989, rows extracted:20
        2019-01-25T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-25T23:59:59.999989, rows extracted:20
        2019-01-26T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        2019-01-26T23:59:59.999989, rows extracted:20
        2019-01-27T23:59:59.999989, -----new indicator cycle-----------
        getting bt data for period 20
        data collected for period 20
        
        
        1 Reply Last reply Reply Quote 0
        • B
          backtrader administrators @pepelepew25 last edited by

          @pepelepew25 said in Bug: Indicator on higher timeframe is called at every cycle of lower timeframe:

          4)the indicator "next" method is called at every cycle of 5min instead of every day.

          Not a bug. I suggest you read the documentation, you can start at Docs - Platform Concepts and read about len.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors