Strategy execution on different timeframes and multi-data

[RandyT]

I'm attempting to complete a system that generates indicators and signals based on a daily timeframe and execute trades against a data feed running on a minute timeframe. As I have written in other threads, I am looking at minute data for the ES and daily data for the SPY. I have set sessionend=dt.time(16, 00) for the daily timeframe resample of the SPY data so I would only expect to see the daily bar for SPY at 16:00 EST.

Note: I am also making an assumption that the sessionend= variable is set to the time in the timezone of that instrument and not my local timezone.

In Strategy next(), how do I ignore the ticks from the minute timeframe data (data0) and only evaluate conditions based on the daily closing bar (data1) that the indicators are processing? As it stands, it appears I am hitting next() for every tick of data0 (ES).

[RandyT]

I think part of what I am seeing here is a replay of all backfilled data from my IB feed on ES data0.

Here is how I am sourcing data:

        # ES Futures Live data timeframe resampled to 1 Minute
        data0 = ibstore.getdata(dataname=args.live_es, fromdate=fetchfrom,
                                timeframe=bt.TimeFrame.Minutes, compression=1)
        cerebro.resampledata(data0, name="ES", timeframe=bt.TimeFrame.Minutes, compression=1)

        # SPY Live data timeframe resampled to 1 Day
        data1 = ibstore.getdata(dataname=args.live_spy, backfill_from=bfdata0,
                                timeframe=bt.TimeFrame.Days, compression=1,
                                sessionend=dt.time(16, 00))
        cerebro.resampledata(data1, name="SPY", timeframe=bt.TimeFrame.Days, compression=1)

After a full replay of backfilled minute data0 (ES), I now see a tick per minute playing through Strategy next()

I'd like to ignore these ticks since the only purpose for the ES feed is to execute a trade on that instrument.

[backtrader]

next is called for any tick which any data produces. Concentrating on a specific data is a matter of looking at the len of the data.

When the len of the data changes (it will always go up) you know you got a tick of that data

def start(self):
    self.lendata1 = 0
def next(self):
    if len(self.data1) > self.lendata1:
        self.lendata1 += 1
        do_something_here()

[RandyT]

@backtrader thank you sir

[Systematica]

this is very helpful - I typically backtest over same time series with 1min resolution for executions. Thanks for posting.

[hbf]

@backtrader @RandyT @Systematica

I'm trying to implement this with the following live code:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)


import datetime

import backtrader as bt

class St(bt.Strategy):

    def __init__(self):
        self.lendata1 = 0

    data_live = False

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, data._name ,'DATA NOTIF:', data._getstatusname(status),
              *args)
        if status == data.LIVE:
            self.data_live = True

    def next(self):
        txt = list()
        txt.append('Data0')
        txt.append('%04d' % len(self.data0))
        dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
        txt.append('{}'.format(self.data.datetime[0]))
        txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
        txt.append('{}'.format(self.data.open[0]))
        txt.append('{}'.format(self.data.high[0]))
        txt.append('{}'.format(self.data.low[0]))
        txt.append('{}'.format(self.data.close[0]))
        txt.append('{}'.format(self.data.volume[0]))
        txt.append('{}'.format(self.data.openinterest[0]))
        print(', '.join(txt))

        if len(self.datas) > 1 and len(self.data1):
            txt = list()
            txt.append('Data1')
            txt.append('%04d' % len(self.data1))
            dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
            txt.append('{}'.format(self.data1.datetime[0]))
            txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt))
            txt.append('{}'.format(self.data1.open[0]))
            txt.append('{}'.format(self.data1.high[0]))
            txt.append('{}'.format(self.data1.low[0]))
            txt.append('{}'.format(self.data1.close[0]))
            txt.append('{}'.format(self.data1.volume[0]))
            txt.append('{}'.format(self.data1.openinterest[0]))
            print(', '.join(txt))

        if len(self.data1) > self.lendata1:
            print(len(self.data1))
            self.lendata1 += 1
            print(self.lendata1)

def runstrat(args=None):

    cerebro = bt.Cerebro()

    # Store
    store = bt.stores.IBStore(port=7497, host='127.0.0.1')

    # Data feed
    data = store.getdata(dataname='AUD.CAD-CASH-IDEALPRO')
    data1 = store.getdata(dataname='AUD.CAD-CASH-IDEALPRO')
    tf0n = str('1m')
    tf1n = str('5m')
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes,
                         compression=1, name=tf0n)
    cerebro.resampledata(data1, timeframe=bt.TimeFrame.Minutes,
                         compression=5, name=tf1n)
    # Broker
    cerebro.broker = store.getbroker()

    # Strategy
    cerebro.addstrategy(St)

    # Execute
    cerebro.run()

if __name__ == '__main__':
    runstrat()

The problem I'm having is that even though I have initiated lendata1 on init, it doesn't seem to be updating on next. The line printed 1 should be 1180, 2 should be 1181 and 282 should be 1197.

Server Version: 76
TWS Time at connection:20181210 13:50:51 AEST
***** 1m DATA NOTIF: DELAYED
***** 5m DATA NOTIF: DELAYED
1340,2018-12-10T01:34:00,1m,0.95982,0.95984,0.95971,0.95975,-1.00000
1180
1
1341,2018-12-10T01:35:00,1m,0.95975,0.95983,0.95963,0.95964,-1.00000
1181
2
...
***** 1m DATA NOTIF: LIVE
***** 5m DATA NOTIF: LIVE
Data0, 0282, 737038.122222, 2018-12-09T21:56:00.000000, 0.9598, 0.95981, 0.95979, 0.95979, 0.0, 0.0
Data1, 1197, 737038.121528, 2018-12-09T21:55:00.000000, 0.95981, 0.95982, 0.95978, 0.95981, -1.0, 0.0
1197
282

I assume that it is because of the backfill. I was wondering if you had any suggestions? Thank you.

Weirdly, it has been working when testing on different timeframes. The following code modification stores the correct values for self.lendata1

    # Data feed
    data = store.getdata(dataname='AUD.CAD-CASH-IDEALPRO')
    data1 = store.getdata(dataname='AUD.CAD-CASH-IDEALPRO')
    tf0n = str('5m')
    tf1n = str('1h')
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes,
                         compression=5, name=tf0n)
    cerebro.resampledata(data1, timeframe=bt.TimeFrame.Minutes,
                         compression=60, name=tf1n)

UPDATE: I have used the following:

def nextstart(self):
     self.lendata1 = len(self.data1)

Updated code:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)


import datetime

import backtrader as bt

class St(bt.Strategy):

    def __init__(self):
        self.lendata1 = 0

    data_live = False

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, data._name ,'DATA NOTIF:', data._getstatusname(status),
              *args)
        if status == data.LIVE:
            self.data_live = True

    def nextstart(self):
        self.lendata1 = len(self.data1)

    def next(self):
        txt = list()
        txt.append('Data0')
        txt.append('{}'.format(len(self.data0)))
        txt.append('{}'.format(self.data.datetime.datetime(0).isoformat()))
        txt.append('{}'.format(self.data.open[0]))
        txt.append('{}'.format(self.data.high[0]))
        txt.append('{}'.format(self.data.low[0]))
        txt.append('{}'.format(self.data.close[0]))
        txt.append('{}'.format(self.data.volume[0]))
        txt.append('{}'.format(self.data.openinterest[0]))
        print(', '.join(txt))

        if len(self.datas) > 1 and len(self.data1):
            txt = list()
            txt.append('Data1')
            txt.append('{}'.format(len(self.data1)))
            txt.append('{}'.format(self.data1.datetime.datetime(0).isoformat()))
            txt.append('{}'.format(self.data1.open[0]))
            txt.append('{}'.format(self.data1.high[0]))
            txt.append('{}'.format(self.data1.low[0]))
            txt.append('{}'.format(self.data1.close[0]))
            txt.append('{}'.format(self.data1.volume[0]))
            txt.append('{}'.format(self.data1.openinterest[0]))
            print(', '.join(txt))

        if len(self.data1) > self.lendata1:
            print(len(self.data1))
            self.lendata1 += 1
            print(self.lendata1)
            print('new higher tf bar')

def runstrat(args=None):

    cerebro = bt.Cerebro()

    # Store
    store = bt.stores.IBStore(port=7497, host='127.0.0.1')

    # Data feed
    data = store.getdata(dataname='AUD.CAD-CASH-IDEALPRO',
                         timeframe=bt.TimeFrame.Ticks)
    
    tf0n = str('30s')
    tf1n = str('2m')

    cerebro.resampledata(data, timeframe=bt.TimeFrame.Seconds,
                         compression=30, name=tf0n)

    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes,
                         compression=2, name=tf1n)

    # Broker
    cerebro.broker = store.getbroker()

    # Strategy
    cerebro.addstrategy(St)

    # Execute
    cerebro.run()

if __name__ == '__main__':
    runstrat()

[stochastick]

I am revisiting this old thread as it seems to be a good jumping off point for a problem I'm stuck on. How might we extend this code to incorporate multiple unique symbols?

@backtrader said in Strategy execution on different timeframes and multi-data:

next is called for any tick which any data produces. Concentrating on a specific data is a matter of looking at the len of the data.

When the len of the data changes (it will always go up) you know you got a tick of that data

def start(self):
    self.lendata1 = 0
def next(self):
    if len(self.data1) > self.lendata1:
        self.lendata1 += 1
        do_something_here()

I am trying to run a test across multiple symbols by evaluating trade signals on one time frame and completing executions on another.

When using the aforementioned code within an enumeration, only trades for the first added data feed are acted upon.

This is a snippet of the code I am working with, which parallel's that @backtrader outlines above:

'''python

def start(self):
    self.datas[1].len = 0

def next(self):
    for i, d in enumerate(self.datas):
        if len(self.datas[1]) > self.datas[1].len:
            self.datas[1].len += 1
            TRADE_LOGIC_HERE

'''

It seems like something else needs to be done other than just wrapping this in enumerate(). I have tried many different versions and have been stuck on this problem for the last several weeks, so any help is appreciated. If needed, I can create a unique thread for this problem.

[bigdavediode]

@stochastick

Just enumerate through the datas in a pythonic way by referencing d. if len(d) > self.lenofdata....

If you reference self.datas[1] you'll only ever access self.datas[1]. Each timeframe and datafeed will have its own data in datas. datas[0] datas[1] datas[2]... datas[n]. Also self.len has the potential for a keyword conflict. I suggest following backtraders example above.

Have you reviewed the multiple datas example? You might want to search that out.