Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Getting daily replay data

    General Code/Help
    2
    7
    2654
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RandyT
      RandyT last edited by RandyT

      If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

      Here is what I am seeing: (I've placed the # in front of the Day timeframe values)

      Datetime                Feed            Open    High    Low    Close    SMA200
      ...
      2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
      2017-01-17T11:57:07	SPY-daily	226.48	226.48	226.48	226.48	226.028
      2017-01-17T11:57:08	SPY-daily	226.48	226.48	226.48	226.48	226.028
      ...
      2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
      2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
      2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
      #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
      #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
      2017-01-17T11:57:19	SPY-daily	226.48	226.54	226.46	226.46	226.027
      

      Printing the above with the following code:

              txtfields = list()
              txtfields.append(self.datas[1].datetime.datetime(tz=CST).strftime("%Y-%m-%dT%H:%M:%S"))
              txtfields.append(self.datas[1]._name)
              txtfields.append('%.2f' % self.datas[1].open[0])
              txtfields.append('%.2f' % self.datas[1].high[0])
              txtfields.append('%.2f' % self.datas[1].low[0])
              txtfields.append('%.2f' % self.datas[1].close[0])
              txtfields.append('%.3f' % self.sma[0])
              print('\t'.join(txtfields))
      

      I'm creating the feeds as so:

          staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
          staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)
      
          # QQQ Live Data
          data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                                  timeframe=bt.TimeFrame.Minutes, compression=1,
                                  sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
          cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)
      
          # SPY Live Data
          data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                                  timeframe=bt.TimeFrame.Days, compression=1,
                                  sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
          cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)
      

      The value at 18:00 appears to be previous day's close.
      The other values are value of every tick.

      How do I get the calculated value for the day for SPY in this example? For both the indicators that are using it as well as the feed value in Daily timeframe?

      B 1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators @RandyT last edited by

        @RandyT said in Getting daily replay data:

        If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

        replaydata will give you a bar in the sought timeframe / compression pair with multiple ticks. And that means:

        • The len of the data feed doesn't change as long as the next timeframe / compression boundary has been reached.

        • With each tick, the timestamp will be udpated to that of the last tick

        • With each tick, there will a call to next to let you process the tick.

        The above log has some surprising things.

        Datetime                Feed            Open    High    Low    Close    SMA200
        ...
        2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
        ...
        

        The same daily bar is delivered 4 times in a row with absolutely no change in timestamp, price values or indicator values.

        2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
        #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
        #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
        

        You already said you placed a # before daily values. Do these lines appear in between the bars with the updated ticks?

        Note: Not that you should know it, but here the most important information would be the len the different objects playing a role here. For example:

        • len(self) for the strategy
        • len(self.data0) for the QQQ
          and last but not least:
        • len(self.data1) for the SPY

        Also: What is the last date in the SPY static data source?

        1 Reply Last reply Reply Quote 0
        • RandyT
          RandyT last edited by RandyT

          The output comes just as it appears with exception that I snipped out some repeated bits with '...'

          I provide the script below to recreate this. It is as simple as I can get it I believe. The last date in the static source file is 2017-01-06. Perhaps that is the problem? I'll try it with full update as well. It was generated from yahoo with your utils script so should be easy to recreate.

          from __future__ import (absolute_import, division, print_function,
                                  unicode_literals)
          
          import datetime as dt
          import os.path
          import sys
          import pandas as pd
          import pytz
          
          # Import the backtrader platform
          import backtrader as bt
          import backtrader.feeds as btfeeds
          
          EST = pytz.timezone('US/Eastern')
          CST = pytz.timezone('US/Central')
          
          # Create a Stratey
          class TestStrategy(bt.Strategy):
              params = (
                  ('maperiod', 15),
              )
          
              def log(self, txt, dt=None):
                  ''' Logging function fot this strategy'''
                  dt = dt or self.datas[0].datetime.datetime(tz=EST)
                  print('%s: %s, %s' % (self.datas[0]._name, dt.isoformat(), txt))
                  if len(self.datas) > 1:
                      dt = dt or self.datas[1].datetime.datetime(tz=CST)
                      print('%s: %s, %s' % (self.datas[1]._name, dt.isoformat(), txt))
          
              def __init__(self):
                  self.dataclose = self.datas[0].close
          
                  # To keep track of pending orders and buy price/commission
                  self.order = None
                  self.buyprice = None
                  self.buycomm = None
                  self.datastatus = False
          
                  # Add a MovingAverageSimple indicator
                  self.sma = bt.indicators.MovingAverageSimple(self.datas[1], period=self.params.maperiod)
          
              def notify_order(self, order):
                  if order.status in [order.Submitted, order.Accepted]:
                      # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                      return
          
                  # Check if an order has been completed
                  # Attention: broker could reject order if not enougth cash
                  if order.status in [order.Completed, order.Canceled, order.Margin]:
                      if order.isbuy():
                          self.log(
                              'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                              (order.executed.price,
                               order.executed.value,
                               order.executed.comm))
          
                          self.buyprice = order.executed.price
                          self.buycomm = order.executed.comm
                      else:  # Sell
                          self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                                   (order.executed.price,
                                    order.executed.value,
                                    order.executed.comm))
          
                      self.bar_executed = len(self)
          
                  # Write down: no pending order
                  self.order = None
          
              def notify_data(self, data, status, *args, **kwargs):
                  if len(data._name) > 0:
                      fname = data._name
          
                  print('*' * 3, '%s DATA NOTIF: %s %s' %
                        (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                         fname, data._getstatusname(status)), *args)
          
                  if status == data.LIVE:
                      self.datastatus = True
          
              def notify_trade(self, trade):
                  if not trade.isclosed:
                      return
          
                  self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                           (trade.pnl, trade.pnlcomm))
          
              def start(self):
                  txtfields = list()
                  txtfields.append('Datetime')
                  txtfields.append('Feed')
                  txtfields.append('Open')
                  txtfields.append('High')
                  txtfields.append('Low')
                  txtfields.append('Close')
                  txtfields.append('SMA')
                  print('\t'.join(txtfields))
          
              def next(self):
                  # Check if an order is pending ... if yes, we cannot send a 2nd one
                  if self.order:
                      return
          
                  if self.len_data0 == 0:
                      self.len_data0 = len(self.data0)
                      return
                  elif self.len_data1 == 0:
                      self.len_data1 = len(self.data1)
                      return
                  elif self.len_data0 < len(self.data0):
                      idx = 0
                      self.len_data0 += 1
                  elif self.len_data1 < len(self.data1):
                      idx = 1
                      self.len_data1 += 1
          
                  print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))
          
                  txtfields = list()
                  txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
                  txtfields.append(self.datas[idx]._name)
                  txtfields.append('%.2f' % self.datas[idx].open[0])
                  txtfields.append('%.2f' % self.datas[idx].high[0])
                  txtfields.append('%.2f' % self.datas[idx].low[0])
                  txtfields.append('%.2f' % self.datas[idx].close[0])
                  txtfields.append('%.3f' % self.sma[0])
                  print('\t'.join(txtfields))
          
          
          
          if __name__ == '__main__':
              # Create a cerebro entity
              cerebro = bt.Cerebro()
          
              # Add a strategy
              cerebro.addstrategy(TestStrategy)
          
              # Datas are in a subfolder of the samples. Need to find where the script is
              # because it could have been called from anywhere
              modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
          
              # Parse CSI SPY data file
              staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
              staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)
          
              storekwargs = dict(
                  host='127.0.0.1',
                  port=7465,
                  clientId=90,
                  timeoffset=True,
                  reconnect=3,
                  timeout=3,
                  _debug=False
              )
          
              ibstore = bt.stores.IBStore(**storekwargs)
          
              broker = ibstore.getbroker()
          
              cerebro.setbroker(broker)
          
              # QQQ Live Data
              data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                                      timeframe=bt.TimeFrame.Minutes, compression=1,
                                      sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
              cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)
          
              # SPY Live Data
              data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                                      timeframe=bt.TimeFrame.Days, compression=1,
                                      sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
              cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)
          
              # Add a FixedSize sizer according to the stake
              cerebro.addsizer(bt.sizers.FixedSize, stake=1)
          
              # Set the commission - 0.1% ... divide by 100 to remove the %
              cerebro.broker.setcommission(stocklike=True, commission=0.005,
                                           commtype=bt.CommInfoBase.COMM_FIXED)
          
              # Print out the starting conditions
              print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
          
              # Run over everything
              cerebro.run()
          
              # Print out the final result
              print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
          
          B 1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @RandyT last edited by

            @RandyT said in Getting daily replay data:

            The last date in the static source file is 2017-01-06. Perhaps that is the problem?

            No. It simply seems that the problems you seem to be able to trigger have backfill_from in common.

            1 Reply Last reply Reply Quote 0
            • RandyT
              RandyT last edited by

              Agreed. FWIW, I saw this same behavior in the more complex system I am working on, so this is just my attempt to recreate it. I know that system is always run with backfill data current through previous day of trading.

              1 Reply Last reply Reply Quote 0
              • RandyT
                RandyT last edited by

                I've made a small change to the script in an attempt to get a bit more visibility. While I am not seeing the same interleaving of values (the market is pretty slow right now), I do show exactly what data I am seeing and have a complete print of what happens. I've updated the script above to include this change and am displaying the change to cerebro.next() below.

                Here is a link to the output: https://dl.dropboxusercontent.com/u/4390007/backfill-log-SPY-QQQ.zip

                    def next(self):
                        # Check if an order is pending ... if yes, we cannot send a 2nd one
                        if self.order:
                            return
                
                        if self.len_data0 == 0:
                            self.len_data0 = len(self.data0)
                            return
                        elif self.len_data1 == 0:
                            self.len_data1 = len(self.data1)
                            return
                        elif self.len_data0 < len(self.data0):
                            idx = 0
                            self.len_data0 += 1
                        elif self.len_data1 < len(self.data1):
                            idx = 1
                            self.len_data1 += 1
                
                        print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))
                
                        txtfields = list()
                        txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
                        txtfields.append(self.datas[idx]._name)
                        txtfields.append('%.2f' % self.datas[idx].open[0])
                        txtfields.append('%.2f' % self.datas[idx].high[0])
                        txtfields.append('%.2f' % self.datas[idx].low[0])
                        txtfields.append('%.2f' % self.datas[idx].close[0])
                        txtfields.append('%.3f' % self.sma[0])
                        print('\t'.join(txtfields))
                
                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators last edited by

                  See the answer to your other post.

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors