Backtrader Community
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Getting daily replay data

    General Code/Help
    2
    7
    2870
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RandyT
      RandyT last edited by RandyT

      If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

      Here is what I am seeing: (I've placed the # in front of the Day timeframe values)

      Datetime                Feed            Open    High    Low    Close    SMA200
...
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
2017-01-17T11:57:07	SPY-daily	226.48	226.48	226.48	226.48	226.028
2017-01-17T11:57:08	SPY-daily	226.48	226.48	226.48	226.48	226.028
...
2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
2017-01-17T11:57:15	SPY-daily	226.48	226.54	226.46	226.46	226.027
2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
#2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
#2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-17T11:57:19	SPY-daily	226.48	226.54	226.46	226.46	226.027

      Printing the above with the following code:

              txtfields = list()
        txtfields.append(self.datas[1].datetime.datetime(tz=CST).strftime("%Y-%m-%dT%H:%M:%S"))
        txtfields.append(self.datas[1]._name)
        txtfields.append('%.2f' % self.datas[1].open[0])
        txtfields.append('%.2f' % self.datas[1].high[0])
        txtfields.append('%.2f' % self.datas[1].low[0])
        txtfields.append('%.2f' % self.datas[1].close[0])
        txtfields.append('%.3f' % self.sma[0])
        print('\t'.join(txtfields))

      I'm creating the feeds as so:

          staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
    staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)

    # QQQ Live Data
    data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                            timeframe=bt.TimeFrame.Minutes, compression=1,
                            sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
    cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)

    # SPY Live Data
    data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                            timeframe=bt.TimeFrame.Days, compression=1,
                            sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
    cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)

      The value at 18:00 appears to be previous day's close.
      The other values are value of every tick.

      How do I get the calculated value for the day for SPY in this example? For both the indicators that are using it as well as the feed value in Daily timeframe?

      B 1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators @RandyT last edited by

        @RandyT said in Getting daily replay data:

        If I understand the .replaydata() concept correctly, I should expect to see the value of the timeframe that we are replaying at in the datas value.

        replaydata will give you a bar in the sought timeframe / compression pair with multiple ticks. And that means:

        • The len of the data feed doesn't change as long as the next timeframe / compression boundary has been reached.

        • With each tick, the timestamp will be udpated to that of the last tick

        • With each tick, there will a call to next to let you process the tick.

        The above log has some surprising things.

        Datetime                Feed            Open    High    Low    Close    SMA200
...
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-17T11:57:05	SPY-daily	226.48	226.48	226.48	226.48	226.028
...

        The same daily bar is delivered 4 times in a row with absolutely no change in timestamp, price values or indicator values.

        2017-01-17T11:57:16	SPY-daily	226.48	226.54	226.46	226.46	226.027
#2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977
2017-01-17T11:57:17	SPY-daily	226.48	226.54	226.46	226.47	226.027
#2017-01-16T18:00:00	SPY-daily	226.18	226.78	226.07	226.48	225.977

        You already said you placed a # before daily values. Do these lines appear in between the bars with the updated ticks?

        Note: Not that you should know it, but here the most important information would be the len the different objects playing a role here. For example:

        • len(self) for the strategy
        • len(self.data0) for the QQQ
          and last but not least:
        • len(self.data1) for the SPY

        Also: What is the last date in the SPY static data source?

        1 Reply Last reply Reply Quote 0
        • RandyT
          RandyT last edited by RandyT

          The output comes just as it appears with exception that I snipped out some repeated bits with '...'

          I provide the script below to recreate this. It is as simple as I can get it I believe. The last date in the static source file is 2017-01-06. Perhaps that is the problem? I'll try it with full update as well. It was generated from yahoo with your utils script so should be easy to recreate.

          from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime as dt
import os.path
import sys
import pandas as pd
import pytz

# Import the backtrader platform
import backtrader as bt
import backtrader.feeds as btfeeds

EST = pytz.timezone('US/Eastern')
CST = pytz.timezone('US/Central')

# Create a Stratey
class TestStrategy(bt.Strategy):
    params = (
        ('maperiod', 15),
    )

    def log(self, txt, dt=None):
        ''' Logging function fot this strategy'''
        dt = dt or self.datas[0].datetime.datetime(tz=EST)
        print('%s: %s, %s' % (self.datas[0]._name, dt.isoformat(), txt))
        if len(self.datas) > 1:
            dt = dt or self.datas[1].datetime.datetime(tz=CST)
            print('%s: %s, %s' % (self.datas[1]._name, dt.isoformat(), txt))

    def __init__(self):
        self.dataclose = self.datas[0].close

        # To keep track of pending orders and buy price/commission
        self.order = None
        self.buyprice = None
        self.buycomm = None
        self.datastatus = False

        # Add a MovingAverageSimple indicator
        self.sma = bt.indicators.MovingAverageSimple(self.datas[1], period=self.params.maperiod)

    def notify_order(self, order):
        if order.status in [order.Submitted, order.Accepted]:
            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return

        # Check if an order has been completed
        # Attention: broker could reject order if not enougth cash
        if order.status in [order.Completed, order.Canceled, order.Margin]:
            if order.isbuy():
                self.log(
                    'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                    (order.executed.price,
                     order.executed.value,
                     order.executed.comm))

                self.buyprice = order.executed.price
                self.buycomm = order.executed.comm
            else:  # Sell
                self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                         (order.executed.price,
                          order.executed.value,
                          order.executed.comm))

            self.bar_executed = len(self)

        # Write down: no pending order
        self.order = None

    def notify_data(self, data, status, *args, **kwargs):
        if len(data._name) > 0:
            fname = data._name

        print('*' * 3, '%s DATA NOTIF: %s %s' %
              (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
               fname, data._getstatusname(status)), *args)

        if status == data.LIVE:
            self.datastatus = True

    def notify_trade(self, trade):
        if not trade.isclosed:
            return

        self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                 (trade.pnl, trade.pnlcomm))

    def start(self):
        txtfields = list()
        txtfields.append('Datetime')
        txtfields.append('Feed')
        txtfields.append('Open')
        txtfields.append('High')
        txtfields.append('Low')
        txtfields.append('Close')
        txtfields.append('SMA')
        print('\t'.join(txtfields))

    def next(self):
        # Check if an order is pending ... if yes, we cannot send a 2nd one
        if self.order:
            return

        if self.len_data0 == 0:
            self.len_data0 = len(self.data0)
            return
        elif self.len_data1 == 0:
            self.len_data1 = len(self.data1)
            return
        elif self.len_data0 < len(self.data0):
            idx = 0
            self.len_data0 += 1
        elif self.len_data1 < len(self.data1):
            idx = 1
            self.len_data1 += 1

        print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))

        txtfields = list()
        txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
        txtfields.append(self.datas[idx]._name)
        txtfields.append('%.2f' % self.datas[idx].open[0])
        txtfields.append('%.2f' % self.datas[idx].high[0])
        txtfields.append('%.2f' % self.datas[idx].low[0])
        txtfields.append('%.2f' % self.datas[idx].close[0])
        txtfields.append('%.3f' % self.sma[0])
        print('\t'.join(txtfields))



if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    # Datas are in a subfolder of the samples. Need to find where the script is
    # because it could have been called from anywhere
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))

    # Parse CSI SPY data file
    staticdata0path = os.path.join(modpath, '../../datas/SPY-yahoo.csv')
    staticdata0 = btfeeds.YahooFinanceCSVData(dataname=staticdata0path)

    storekwargs = dict(
        host='127.0.0.1',
        port=7465,
        clientId=90,
        timeoffset=True,
        reconnect=3,
        timeout=3,
        _debug=False
    )

    ibstore = bt.stores.IBStore(**storekwargs)

    broker = ibstore.getbroker()

    cerebro.setbroker(broker)

    # QQQ Live Data
    data0 = ibstore.getdata(dataname='QQQ-STK-SMART-USD',
                            timeframe=bt.TimeFrame.Minutes, compression=1,
                            sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
    cerebro.replaydata(data0, name='QQQ-minutes', timeframe=bt.TimeFrame.Minutes, compression=1)

    # SPY Live Data
    data1 = ibstore.getdata(dataname='SPY-STK-SMART-USD', backfill_from=staticdata0,
                            timeframe=bt.TimeFrame.Days, compression=1,
                            sessionstart=dt.time(9,30), sessionend=dt.time(16, 0))
    cerebro.replaydata(data1, name='SPY-daily', timeframe=bt.TimeFrame.Days, compression=1)

    # Add a FixedSize sizer according to the stake
    cerebro.addsizer(bt.sizers.FixedSize, stake=1)

    # Set the commission - 0.1% ... divide by 100 to remove the %
    cerebro.broker.setcommission(stocklike=True, commission=0.005,
                                 commtype=bt.CommInfoBase.COMM_FIXED)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
          B 1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @RandyT last edited by

            @RandyT said in Getting daily replay data:

            The last date in the static source file is 2017-01-06. Perhaps that is the problem?

            No. It simply seems that the problems you seem to be able to trigger have backfill_from in common.

            1 Reply Last reply Reply Quote 0
            • RandyT
              RandyT last edited by

              Agreed. FWIW, I saw this same behavior in the more complex system I am working on, so this is just my attempt to recreate it. I know that system is always run with backfill data current through previous day of trading.

              1 Reply Last reply Reply Quote 0
              • RandyT
                RandyT last edited by

                I've made a small change to the script in an attempt to get a bit more visibility. While I am not seeing the same interleaving of values (the market is pretty slow right now), I do show exactly what data I am seeing and have a complete print of what happens. I've updated the script above to include this change and am displaying the change to cerebro.next() below.

                Here is a link to the output: https://dl.dropboxusercontent.com/u/4390007/backfill-log-SPY-QQQ.zip

                    def next(self):
        # Check if an order is pending ... if yes, we cannot send a 2nd one
        if self.order:
            return

        if self.len_data0 == 0:
            self.len_data0 = len(self.data0)
            return
        elif self.len_data1 == 0:
            self.len_data1 = len(self.data1)
            return
        elif self.len_data0 < len(self.data0):
            idx = 0
            self.len_data0 += 1
        elif self.len_data1 < len(self.data1):
            idx = 1
            self.len_data1 += 1

        print('len self: %d  data0: %d  data1: %d' % (len(self), len(self.data0), len(self.data1)))

        txtfields = list()
        txtfields.append(self.datas[idx].datetime.datetime(tz=EST).strftime("%Y-%m-%dT%H:%M:%S"))
        txtfields.append(self.datas[idx]._name)
        txtfields.append('%.2f' % self.datas[idx].open[0])
        txtfields.append('%.2f' % self.datas[idx].high[0])
        txtfields.append('%.2f' % self.datas[idx].low[0])
        txtfields.append('%.2f' % self.datas[idx].close[0])
        txtfields.append('%.3f' % self.sma[0])
        print('\t'.join(txtfields))
                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators last edited by

                  See the answer to your other post.

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors