IB delayed vs live timeframes (interday trading)

[seano]

Hi Backtrader (and community),

I'm automating my existing manual system. I've recently hit something I'm struggling to troubleshoot so hope you can help. Intent is to use backtrader for live trading strategies that take a pre-screened short-list of LSE stocks and waits for break-outs (hitting new highs) before buying with a fixed/trailing stop. I can get the multiple datafeeds and trade triggers working ok, but have issues with the timeframes when implementing live system. Need to use daily data, looking at new highs of most recent close for the break-outs - only really need to run the script daily as most positions will be bought at market and held for more than a day (potentially a week or more).

I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB. If I use resample then I'm getting only a small set of the daily data (missing days) so suspect it's actually resampling the daily data.

Some specific questions:

• Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...
• Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?

Below is main snippet I'm using to test:

def runstrategy():
    
    # Hardcode the args
    args = dict(
        host='127.0.0.1',
        port=7497, #7496 for live,
        clientId=None,
        data0='VOD-STK-LSE', #'TWTR', 'EUR.USD-CASH-IDEALPRO', 'RWA-STK-LSE'
        data1=None,
        resample=False, #False,
        timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days'
        compression=1, #5,
    )
    
    # Create a cerebro
    cerebro = bt.Cerebro()

    #Use store method
    ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId'])
    broker = ibstore.getbroker()
    cerebro.setbroker(broker)

    #Set the timeframe
    timeframe = bt.TimeFrame.TFrame(args['timeframe'])

    if args['resample']:
        datatf = bt.TimeFrame.Ticks
        datacomp = 1
    else:
        datatf = timeframe
        datacomp = args['compression']

    #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData
    IBDataFactory = ibstore.getdata

    #Create the data using datatf and datacomp
    data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp)

    #Feed the strategy with resampled data if requested
    if args['resample']:
        cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression'])
    else:
        cerebro.adddata(data0)

    # Add the strategy
    cerebro.addstrategy(TestStrategy)

    cerebro.run()

Sample output:

Data0, 0248, 737092.0, 2019-02-01T23:59:59.999989, 138.28, 139.42, 136.8, 138.84, 54488196.0, 0.0, 136.928
Data0, 0249, 737095.0, 2019-02-04T23:59:59.999989, 139.2, 139.76, 136.7, 138.1, 52807693.0, 0.0, 137.46
Data0, 0250, 737096.0, 2019-02-05T23:59:59.999989, 137.84, 141.54, 137.82, 141.06, 55387568.0, 0.0, 138.656
Data0, 0251, 737097.0, 2019-02-06T23:59:59.999989, 142.2, 143.62, 142.0, 142.4, 55193955.0, 0.0, 139.796
Data0, 0252, 737098.0, 2019-02-07T23:59:59.999989, 141.5, 141.76, 138.84, 139.38, 60064548.0, 0.0, 139.956
Data0, 0253, 737098.0, 2019-02-08T00:00:00.000000, 139.0, 139.46, 138.52, 138.96, 5210654.0, 0.0, 139.98
***** DATA NOTIF: LIVE
Data0, 0254, 737098.396565, 2019-02-08T09:31:03.195999, 138.96, 138.96, 138.96, 138.96, 8127.0, 0.0, 140.152
Data0, 0255, 737098.397154, 2019-02-08T09:31:54.083995, 138.96, 138.96, 138.96, 138.96, 600.0, 0.0, 139.732
Data0, 0256, 737098.397257, 2019-02-08T09:32:03.009997, 138.92, 138.92, 138.92, 138.92, 24490.0, 0.0, 139.036
Data0, 0257, 737098.397405, 2019-02-08T09:32:15.790003, 138.9, 138.9, 138.9, 138.9, 4412.0, 0.0, 138.94
Data0, 0258, 737098.397521, 2019-02-08T09:32:25.818997, 138.88, 138.88, 138.88, 138.88, 3223.0, 0.0, 138.924

Loving the package and support! Appreciate any guidance. Many thanks.

[backtrader]

@seano said in IB delayed vs live timeframes (interday trading):

I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB

There is no switch in backtrader. IB doesn't provide full bars in real-time, it does only provide ticks (or 5-seconds real-time bars). Those ticks need to be resampled to days. The execution snippets for ibtest.py show it.

@seano said in IB delayed vs live timeframes (interday trading):

• Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...

The daily close is obviously not available until the market closes, so you may want to reformulate your question.

@seano said in IB delayed vs live timeframes (interday trading):

• Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?

Resampling should do it, but you say you have missing bars, but we haven't seen that above.

@seano said in IB delayed vs live timeframes (interday trading):

Sample output:

What was the input? (Because you have different execution paths)

[seano]

@backtrader said in IB delayed vs live timeframes (interday trading):

Thanks for quick response. To rephrase my question, I was asking whether there was intentional logic to 'switch' from daily to tick (or 5-seconds real-time bars) when going from delayed to live data... but as explained, there is no such switch so problem must be elsewhere.

Just ran the script for TWTR using resampling and you will see that the delayed data below has missing dates:

Data0, 0108, 737069.208333, 2019-01-10T00:00:00.000000, 29.9, 32.4, 29.76, 32.02, 470227.0, 0.0, 29.28
Data0, 0109, 737072.208333, 2019-01-13T00:00:00.000000, 32.02, 33.5, 32.02, 32.77, 462363.0, 0.0, 30.448
Data0, 0110, 737076.208333, 2019-01-17T00:00:00.000000, 32.75, 33.35, 32.12, 32.49, 313474.0, 0.0, 31.232
Data0, 0111, 737079.208333, 2019-01-20T00:00:00.000000, 32.32, 33.89, 32.24, 33.2, 255231.0, 0.0, 32.116
Data0, 0112, 737084.208333, 2019-01-25T00:00:00.000000, 33.01, 33.35, 30.72, 31.58, 455923.0, 0.0, 32.412
Data0, 0113, 737089.208333, 2019-01-30T00:00:00.000000, 31.66, 33.67, 31.46, 31.91, 602043.0, 0.0, 32.39
Data0, 0114, 737092.208333, 2019-02-02T00:00:00.000000, 32.0, 34.09, 31.42, 33.2, 526106.0, 0.0, 32.476
Data0, 0115, 737097.208333, 2019-02-07T00:00:00.000000, 33.3, 35.29, 33.24, 35.05, 621249.0, 0.0, 32.988
***** DATA NOTIF: LIVE

Full script below:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import argparse
import datetime

# The above could be sent to an independent module
import backtrader as bt
from backtrader.utils import flushfile  # win32 quick stdout flushing


class TestStrategy(bt.Strategy):
    params = dict(
        smaperiod=5,
        trade=True,
        stake=10,
        exectype=bt.Order.Market,
        stopafter=20,
        valid=None,
        cancel=0,
        donotsell=False,
        stoptrail=False,
        stoptraillimit=False,
        trailamount=None,
        trailpercent=None,
        limitoffset=None,
        oca=False,
        bracket=False,
        stop_loss=0.01, # NEW: price is 5% less than the entry point (Redundant if use trail I presume)
        trail=False, # NEW: NOTE IT IS SIMILAR TO stoptrail; set trail=False if want fixed stop specified in stop_loss; OR set to a numeric value (e.g. 0.10) will tell the strategy to use a StopTrail
    )


    def __init__(self):
        # To control operation entries
        self.orderid = list()
        self.order = None

        self.counttostop = 0
        self.datastatus = 0

        # NEW: Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.data0.close
        
        # NEW: Add a Highest Close indicator
        # TODO: Note that doing this with period=100 means trades don't start until after 100 periods
        # Should look at a way to simply keep a running record of highest high (in all history) instead
        self.the_highest_close = bt.ind.Highest(self.data0.close, period=5, subplot=False)
        
        # Create SMA on 2nd data
        self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod)

        print('--------------------------------------------------')
        print('Strategy Created')
        print('--------------------------------------------------')

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
        if status == data.LIVE:
            self.counttostop = self.p.stopafter
            self.datastatus = 1

    def notify_store(self, msg, *args, **kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    def notify_order(self, order):
        if order.status in [order.Completed, order.Cancelled, order.Rejected]:
            self.order = None

        print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
        print(order)
        print('-' * 50, 'ORDER END')

    def notify_trade(self, trade):
        print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
        print(trade)
        print('-' * 50, 'TRADE END')

    def prenext(self):
        self.next(frompre=True)

    def next(self, frompre=False):
        txt = list()
        txt.append('Data0')
        txt.append('%04d' % len(self.data0))
        dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
        txt.append('{}'.format(self.data.datetime[0]))
        txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
        txt.append('{}'.format(self.data.open[0]))
        txt.append('{}'.format(self.data.high[0]))
        txt.append('{}'.format(self.data.low[0]))
        txt.append('{}'.format(self.data.close[0]))
        txt.append('{}'.format(self.data.volume[0]))
        txt.append('{}'.format(self.data.openinterest[0]))
        txt.append('{}'.format(self.sma[0]))
        #txt.append('{}'.format(self.the_highest_close[0])) #NEW: to see if it should trigger
        #txt.append('{}'.format(self.dataclose[0])) #NEW: to see if it should trigger
        print(', '.join(txt))

        if len(self.datas) > 1 and len(self.data1):
            txt = list()
            txt.append('Data1')
            txt.append('%04d' % len(self.data1))
            dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
            txt.append('{}'.format(self.data1.datetime[0]))
            txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt))
            txt.append('{}'.format(self.data1.open[0]))
            txt.append('{}'.format(self.data1.high[0]))
            txt.append('{}'.format(self.data1.low[0]))
            txt.append('{}'.format(self.data1.close[0]))
            txt.append('{}'.format(self.data1.volume[0]))
            txt.append('{}'.format(self.data1.openinterest[0]))
            txt.append('{}'.format(float('NaN')))
            print(', '.join(txt))

        if self.counttostop:  # stop after x live lines
            self.counttostop -= 1
            if not self.counttostop:
                self.env.runstop()
                return

        if not self.p.trade:
            return

        if self.datastatus and not self.position: # and len(self.orderid) < 1: #NEW Removed this
            exectype = self.p.exectype if not self.p.oca else bt.Order.Limit
            close = self.data0.close[0]
            price = round(close * 0.90, 2)
            
            # NEW: Buy rule => Not yet ... we MIGHT BUY if ... 
            if self.dataclose[0] > self.the_highest_close[-1]:
                    # A BREAKOUT !!!

                    # BUY, BUY, BUY!!! (with default parameters)
                    # self.log('BUY CREATE, %.2f' % self.dataclose[0])

                    # Keep track of the created order to avoid a 2nd order
                    # self.order = self.buy()
                    
                    self.order = self.buy(size=self.p.stake,
                                  exectype=exectype,
                                  price=price,
                                  valid=self.p.valid,
                                  transmit=not self.p.bracket)
                    
                    # Set a stop loss at the time of buy (need to enable cheat on close in __init__() first)
                    if not self.params.trail:
                        stop_price = round(self.dataclose[0] * (1.0 - self.params.stop_loss), 2)
                        self.sell(size=self.p.stake, exectype=bt.Order.Stop, price=stop_price)
                    else:
                        # Originally used trailamount... but I changed to trailpercent
                        # https://www.backtrader.com/docu/order-creation-execution/bracket/bracket.html?highlight=stoptrail
                        self.sell(size=self.p.stake, exectype=bt.Order.StopTrail,
                                  trailpercent=self.p.trail)
            
            
            self.orderid.append(self.order)
            
            # TODO: Look into if I should be using these instead of my stop logic above
            if self.p.bracket:
                # low side
                self.sell(size=self.p.stake,
                          exectype=bt.Order.Stop,
                          price=round(price * 0.90, 2),
                          valid=self.p.valid,
                          transmit=False,
                          parent=self.order)

                # high side
                self.sell(size=self.p.stake,
                          exectype=bt.Order.Limit,
                          price=round(close * 1.10, 2),
                          valid=self.p.valid,
                          transmit=True,
                          parent=self.order)

            elif self.p.oca:
                self.buy(size=self.p.stake,
                         exectype=bt.Order.Limit,
                         price=round(self.data0.close[0] * 0.80, 2),
                         oco=self.order)

            elif self.p.stoptrail:
                self.sell(size=self.p.stake,
                          exectype=bt.Order.StopTrail,
                          # price=round(self.data0.close[0] * 0.90, 2),
                          valid=self.p.valid,
                          trailamount=self.p.trailamount,
                          trailpercent=self.p.trailpercent)

            elif self.p.stoptraillimit:
                p = round(self.data0.close[0] - self.p.trailamount, 2)
                # p = self.data0.close[0]
                self.sell(size=self.p.stake,
                          exectype=bt.Order.StopTrailLimit,
                          price=p,
                          plimit=p + self.p.limitoffset,
                          valid=self.p.valid,
                          trailamount=self.p.trailamount,
                          trailpercent=self.p.trailpercent)

        # TAKE THIS OUT AND JUST HAVE THE STOP LOSS
        #elif self.position.size > 0 and not self.p.donotsell:
        #    if self.order is None:
        #        self.order = self.sell(size=self.p.stake // 2,
        #                               exectype=bt.Order.Market,
        #                               price=self.data0.close[0])

        elif self.order is not None and self.p.cancel:
            if self.datastatus > self.p.cancel:
                self.cancel(self.order)

        if self.datastatus:
            self.datastatus += 1

    def start(self):
        if self.data0.contractdetails is not None:
            print('Timezone from ContractDetails: {}'.format(
                  self.data0.contractdetails.m_timeZoneId))

        header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume',
                  'OpenInterest', 'SMA']
        print(', '.join(header))

        self.done = False


def runstrategy():
    
    # Hardcode the args
    args = dict(
        host='127.0.0.1',
        port=7497, #7496 for live,
        clientId=None,
        data0='TWTR', #'TWTR', #None, #'EUR.USD-CASH-IDEALPRO'RWA-STK-LSE'
        data1=None,
        resample=True, #False,
        timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days'
        compression=1, #5,
    )
    
    # Create a cerebro
    cerebro = bt.Cerebro()

    #Use store method
    ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId'])
    broker = ibstore.getbroker()
    cerebro.setbroker(broker)

    #Set the timeframe
    timeframe = bt.TimeFrame.TFrame(args['timeframe'])

    if args['resample']:
        datatf = bt.TimeFrame.Ticks
        datacomp = 1
    else:
        datatf = timeframe
        datacomp = args['compression']

    #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData
    IBDataFactory = ibstore.getdata

    #Create the data using datatf and datacomp
    data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp)

    #Feed the strategy with resampled data if requested
    if args['resample']:
        cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression'])
    else:
        cerebro.adddata(data0)

    # Add the strategy
    cerebro.addstrategy(TestStrategy)

    cerebro.run()


if __name__ == '__main__':
    runstrategy()