Getting daily replay data



  • If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

    Here is what I am seeing: (I've placed the # in front of the Day timeframe values)

    Datetime                Feed            Open    High    Low    Close    SMA200
    ...
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
    2017-01-17T11:57:07	SPY-daily	226.48	226.48	226.48	226.48	226.028
    2017-01-17T11:57:08	SPY-daily	226.48	226.48	226.48	226.48	226.028
    ...
    2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
    2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
    2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
    #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
    #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-17T11:57:19	SPY-daily	226.48	226.54	226.46	226.46	226.027
    

    Printing the above with the following code:

            txtfields = list()
            txtfields.append(self.datas[1].datetime.datetime(tz=CST).strftime("%Y-%m-%dT%H:%M:%S"))
            txtfields.append(self.datas[1]._name)
            txtfields.append('%.2f' % self.datas[1].open[0])
            txtfields.append('%.2f' % self.datas[1].high[0])
            txtfields.append('%.2f' % self.datas[1].low[0])
            txtfields.append('%.2f' % self.datas[1].close[0])
            txtfields.append('%.3f' % self.sma[0])
            print('\t'.join(txtfields))
    

    I'm creating the feeds as so:

        staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
        staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)
    
        # QQQ Live Data
        data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                                timeframe=bt.TimeFrame.Minutes, compression=1,
                                sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
        cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)
    
        # SPY Live Data
        data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                                timeframe=bt.TimeFrame.Days, compression=1,
                                sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
        cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)
    

    The value at 18:00 appears to be previous day's close.
    The other values are value of every tick.

    How do I get the calculated value for the day for SPY in this example? For both the indicators that are using it as well as the feed value in Daily timeframe?


  • administrators

    @RandyT said in Getting daily replay data:

    If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

    replaydata will give you a bar in the sought timeframe / compression pair with multiple ticks. And that means:

    • The len of the data feed doesn't change as long as the next timeframe / compression boundary has been reached.

    • With each tick, the timestamp will be udpated to that of the last tick

    • With each tick, there will a call to next to let you process the tick.

    The above log has some surprising things.

    Datetime                Feed            Open    High    Low    Close    SMA200
    ...
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
    ...
    

    The same daily bar is delivered 4 times in a row with absolutely no change in timestamp, price values or indicator values.

    2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
    #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
    #2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
    

    You already said you placed a # before daily values. Do these lines appear in between the bars with the updated ticks?

    Note: Not that you should know it, but here the most important information would be the len the different objects playing a role here. For example:

    • len(self) for the strategy
    • len(self.data0) for the QQQ
      and last but not least:
    • len(self.data1) for the SPY

    Also: What is the last date in the SPY static data source?



  • The output comes just as it appears with exception that I snipped out some repeated bits with '...'

    I provide the script below to recreate this. It is as simple as I can get it I believe. The last date in the static source file is 2017-01-06. Perhaps that is the problem? I'll try it with full update as well. It was generated from yahoo with your utils script so should be easy to recreate.

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import datetime as dt
    import os.path
    import sys
    import pandas as pd
    import pytz
    
    # Import the backtrader platform
    import backtrader as bt
    import backtrader.feeds as btfeeds
    
    EST = pytz.timezone('US/Eastern')
    CST = pytz.timezone('US/Central')
    
    # Create a Stratey
    class TestStrategy(bt.Strategy):
        params = (
            ('maperiod', 15),
        )
    
        def log(self, txt, dt=None):
            ''' Logging function fot this strategy'''
            dt = dt or self.datas[0].datetime.datetime(tz=EST)
            print('%s: %s, %s' % (self.datas[0]._name, dt.isoformat(), txt))
            if len(self.datas) > 1:
                dt = dt or self.datas[1].datetime.datetime(tz=CST)
                print('%s: %s, %s' % (self.datas[1]._name, dt.isoformat(), txt))
    
        def __init__(self):
            self.dataclose = self.datas[0].close
    
            # To keep track of pending orders and buy price/commission
            self.order = None
            self.buyprice = None
            self.buycomm = None
            self.datastatus = False
    
            # Add a MovingAverageSimple indicator
            self.sma = bt.indicators.MovingAverageSimple(self.datas[1], period=self.params.maperiod)
    
        def notify_order(self, order):
            if order.status in [order.Submitted, order.Accepted]:
                # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                return
    
            # Check if an order has been completed
            # Attention: broker could reject order if not enougth cash
            if order.status in [order.Completed, order.Canceled, order.Margin]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
                self.bar_executed = len(self)
    
            # Write down: no pending order
            self.order = None
    
        def notify_data(self, data, status, *args, **kwargs):
            if len(data._name) > 0:
                fname = data._name
    
            print('*' * 3, '%s DATA NOTIF: %s %s' %
                  (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                   fname, data._getstatusname(status)), *args)
    
            if status == data.LIVE:
                self.datastatus = True
    
        def notify_trade(self, trade):
            if not trade.isclosed:
                return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                     (trade.pnl, trade.pnlcomm))
    
        def start(self):
            txtfields = list()
            txtfields.append('Datetime')
            txtfields.append('Feed')
            txtfields.append('Open')
            txtfields.append('High')
            txtfields.append('Low')
            txtfields.append('Close')
            txtfields.append('SMA')
            print('\t'.join(txtfields))
    
        def next(self):
            # Check if an order is pending ... if yes, we cannot send a 2nd one
            if self.order:
                return
    
            if self.len_data0 == 0:
                self.len_data0 = len(self.data0)
                return
            elif self.len_data1 == 0:
                self.len_data1 = len(self.data1)
                return
            elif self.len_data0 < len(self.data0):
                idx = 0
                self.len_data0 += 1
            elif self.len_data1 < len(self.data1):
                idx = 1
                self.len_data1 += 1
    
            print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))
    
            txtfields = list()
            txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
            txtfields.append(self.datas[idx]._name)
            txtfields.append('%.2f' % self.datas[idx].open[0])
            txtfields.append('%.2f' % self.datas[idx].high[0])
            txtfields.append('%.2f' % self.datas[idx].low[0])
            txtfields.append('%.2f' % self.datas[idx].close[0])
            txtfields.append('%.3f' % self.sma[0])
            print('\t'.join(txtfields))
    
    
    
    if __name__ == '__main__':
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        # Add a strategy
        cerebro.addstrategy(TestStrategy)
    
        # Datas are in a subfolder of the samples. Need to find where the script is
        # because it could have been called from anywhere
        modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    
        # Parse CSI SPY data file
        staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
        staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)
    
        storekwargs = dict(
            host='127.0.0.1',
            port=7465,
            clientId=90,
            timeoffset=True,
            reconnect=3,
            timeout=3,
            _debug=False
        )
    
        ibstore = bt.stores.IBStore(**storekwargs)
    
        broker = ibstore.getbroker()
    
        cerebro.setbroker(broker)
    
        # QQQ Live Data
        data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                                timeframe=bt.TimeFrame.Minutes, compression=1,
                                sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
        cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)
    
        # SPY Live Data
        data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                                timeframe=bt.TimeFrame.Days, compression=1,
                                sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
        cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)
    
        # Add a FixedSize sizer according to the stake
        cerebro.addsizer(bt.sizers.FixedSize, stake=1)
    
        # Set the commission - 0.1% ... divide by 100 to remove the %
        cerebro.broker.setcommission(stocklike=True, commission=0.005,
                                     commtype=bt.CommInfoBase.COMM_FIXED)
    
        # Print out the starting conditions
        print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
    
        # Run over everything
        cerebro.run()
    
        # Print out the final result
        print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    

  • administrators

    @RandyT said in Getting daily replay data:

    The last date in the static source file is 2017-01-06. Perhaps that is the problem?

    No. It simply seems that the problems you seem to be able to trigger have backfill_from in common.



  • Agreed. FWIW, I saw this same behavior in the more complex system I am working on, so this is just my attempt to recreate it. I know that system is always run with backfill data current through previous day of trading.



  • I've made a small change to the script in an attempt to get a bit more visibility. While I am not seeing the same interleaving of values (the market is pretty slow right now), I do show exactly what data I am seeing and have a complete print of what happens. I've updated the script above to include this change and am displaying the change to cerebro.next() below.

    Here is a link to the output: https://dl.dropboxusercontent.com/u/4390007/backfill-log-SPY-QQQ.zip

        def next(self):
            # Check if an order is pending ... if yes, we cannot send a 2nd one
            if self.order:
                return
    
            if self.len_data0 == 0:
                self.len_data0 = len(self.data0)
                return
            elif self.len_data1 == 0:
                self.len_data1 = len(self.data1)
                return
            elif self.len_data0 < len(self.data0):
                idx = 0
                self.len_data0 += 1
            elif self.len_data1 < len(self.data1):
                idx = 1
                self.len_data1 += 1
    
            print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))
    
            txtfields = list()
            txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
            txtfields.append(self.datas[idx]._name)
            txtfields.append('%.2f' % self.datas[idx].open[0])
            txtfields.append('%.2f' % self.datas[idx].high[0])
            txtfields.append('%.2f' % self.datas[idx].low[0])
            txtfields.append('%.2f' % self.datas[idx].close[0])
            txtfields.append('%.3f' % self.sma[0])
            print('\t'.join(txtfields))
    

  • administrators

    See the answer to your other post.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.