Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    .resampledata() skips data

    General Code/Help
    2
    4
    1498
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RandyT
      RandyT last edited by RandyT

      @backtrader seems we have discussed this before and not sure if this is an issue with the code, or an issue with my memory...

      I've mentioned I occasionally see the system skipping or not showing the most recent data when using backfill_from=.

      With the following example code, I see the following scenarios:

      1. Using .resampledata() to "register" the feed, with backfill_from= using the local static data, the output I see from the bar print in next() shows every other bar.

      2. Using .resampledata() without backfill_from= never gets to a LIVE data status with the IB connection.

      3. Using .adddata() shows the expected output.

      The code may be a bit more complicated than needed, but I have been using it to track down why I am not seeing valid indicator output when running live vs. in backtest (which may be related)

      #!/usr/bin/env python
      # -*- coding: utf-8; py-indent-offset:4 -*-
      from __future__ import (absolute_import, division, print_function,
                              unicode_literals)
      
      import datetime as dt
      import pytz
      
      # Import the backtrader platform
      import backtrader as bt
      import backtrader.feeds as btfeed
      import backtrader.indicators as btind
      
      EST = pytz.timezone('US/Eastern')
      
      
      class TestDVStrategy(bt.Strategy):
      
          def log(self, txt, dt=None):
              ''' Logging function for this strategy'''
              dt = dt or self.data_spy.datetime.datetime(tz=EST)
              lstr = 'TEST: {}: {}'.format(dt.isoformat(), txt)
              print(lstr)
      
          def __init__(self):
              # To keep track of pending orders and buy price/commission
              self.trend = None
              self.datastatus = False
              self.data_spy = self.getdatabyname('SPY')
      
              # Add a MovingAverageSimple indicator based from SPY
              self.sma = btind.SMA(self.data_spy.close, period=200)
      
          def notify_data(self, data, status, *args, **kwargs):
              print('*' * 3, '%s DATA NOTIF: %s' %
                    (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                     data._getstatusname(status)), *args)
      
              if status == data.LIVE:
                  self.datastatus = True
      
          def notify_store(self, msg, *args, **kwargs):
              print('*' * 3, '%s STORE NOTIF: %s' %
                    (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'), msg))
      
          def next(self):
              if len(self) < 4:
                  return
      
              if self.sma[-1] > self.sma[-2] > self.sma[-3] > self.sma[-4]:
                  self.trend = 'up'
              else:
                  self.trend = 'down'
      
              print('-- %004d' % len(self), str(self.datetime.datetime()))
              for i, d in enumerate(di for di in self.datas if len(di)):
                  out = ['Data%d' % i, d._name, '%004d' % len(d),
                         str(d.datetime.datetime()), str(d.close[0])]
                  print('\t'.join(out))
      
              self.log('TEST: %.2f SMA: %.3f TREND: %s' %
                       (self.data_spy.close[0],
                        self.sma[0],
                        self.trend))
      
      
      def runstrategy():
          # Create a cerebro entity
          cerebro = bt.Cerebro()
      
          # Add a strategy
          cerebro.addstrategy(TestDVStrategy)
      
          # Parse static SPY data file
          bfdata0 = btfeed.GenericCSVData(dataname='./datas/SPY-1D.csv',
                                          dtformat=('%Y-%m-%d'),
                                          datetime=0,
                                          high=1,
                                          low=2,
                                          open=3,
                                          close=4,
                                          volume=5,
                                          openinterest=6,
                                          sessionstart=dt.time(9, 30),
                                          sessionend=dt.time(16, 0),
                                          tz=EST)
      
          storekwargs = dict(
              host='127.0.4.1',
              port=7465,
              clientId=110,
              _debug=False
          )
      
          ibstore = bt.stores.IBStore(**storekwargs)
          broker = ibstore.getbroker()
          cerebro.setbroker(broker)
      
          # SPY Live data timeframe resampled to 1 Day
          data0 = ibstore.getdata(dataname='SPY-STK-SMART-USD',
                                  # backfill_from=bfdata0,
                                  timeframe=bt.TimeFrame.Days, compression=1,
                                  sessionstart=dt.time(9, 30),
                                  sessionend=dt.time(16, 0),
                                  tz='EST5EDT')
          # cerebro.resampledata(data0, name='SPY',
          #                      timeframe=bt.TimeFrame.Days, compression=1)
      
          cerebro.adddata(data0, name='SPY')
          cerebro.run(runonce=False, tradehistory=True, exactbars=0)
      
      
      if __name__ == '__main__':
          runstrategy()
      
      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        Worth looking into it.

        1. Using .adddata() shows the expected output.

        This statement is probably not meant so. Because without resampledata, the feed returned by the IBStore will download data in Days (because the timeframe indicates it to do so), but it will later deliver the ticks.

        It would be much better to pass bt.TimeFrame.Ticks (which is what the feed actually delivers) and let resampledata do its thing to download the historical data in bt.TimeFrame.Days

        This is an area which would probably benefit from some rework (you already mentioned it and it is done with the VisualChart live feed, for other reasons):

        • Having getdata (actually IBData) to directly return a resampled version of itself for the given timeframe

        What would also require:

        • To separate the historical (for backfilling) download from the internals of IBData

        In order to be able to do a 2nd resampledata (and a 3rd if needed be) with a different (larger, smaller) timeframe/compression combination.

        Some extra thought should have gone into the aforementioned bullet points at the time of development.

        1 Reply Last reply Reply Quote 0
        • RandyT
          RandyT last edited by

          I somehow think the backfill_from= is part of the issue here. Also, not sure from your comments but note that in this script, I am only requesting one feed from IB.

          The issue is seen when I backfill static data and for some reason, when using the .resampledata() method, the system is dropping every other bar. This is identical to another issue we were seeing when I was trying to .resampledata() from the InfluxDB feed. In that case, .adddata() produced all bars. .resampledata() dropped every other bar.

          B 1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @RandyT last edited by

            @RandyT said in .resampledata() skips data:

            The issue is seen when I backfill static data and for some reason, when using the .resampledata() method, the system is dropping every other bar. This is identical to another issue we were seeing when I was trying to .resampledata() from the InfluxDB feed. In that case, .adddata() produced all bars. .resampledata() dropped every other bar.

            This issue was due to not having specified the sessionend for the data, which collided with the algorithm in charge of knowing when the day has come to an end. This was also the reason to add the default sessionend to, at least, all CSV data feeds, which has uncovered that some users were even not actually specifying the incoming timeframe, which was confusing the resampling code.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors