For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

IB delayed vs live timeframes (interday trading)



  • Hi Backtrader (and community),

    I'm automating my existing manual system. I've recently hit something I'm struggling to troubleshoot so hope you can help. Intent is to use backtrader for live trading strategies that take a pre-screened short-list of LSE stocks and waits for break-outs (hitting new highs) before buying with a fixed/trailing stop. I can get the multiple datafeeds and trade triggers working ok, but have issues with the timeframes when implementing live system. Need to use daily data, looking at new highs of most recent close for the break-outs - only really need to run the script daily as most positions will be bought at market and held for more than a day (potentially a week or more).

    I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB. If I use resample then I'm getting only a small set of the daily data (missing days) so suspect it's actually resampling the daily data.

    Some specific questions:

    • Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...
    • Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?

    Below is main snippet I'm using to test:

    def runstrategy():
        
        # Hardcode the args
        args = dict(
            host='127.0.0.1',
            port=7497, #7496 for live,
            clientId=None,
            data0='VOD-STK-LSE', #'TWTR', 'EUR.USD-CASH-IDEALPRO', 'RWA-STK-LSE'
            data1=None,
            resample=False, #False,
            timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days'
            compression=1, #5,
        )
        
        # Create a cerebro
        cerebro = bt.Cerebro()
    
        #Use store method
        ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId'])
        broker = ibstore.getbroker()
        cerebro.setbroker(broker)
    
        #Set the timeframe
        timeframe = bt.TimeFrame.TFrame(args['timeframe'])
    
        if args['resample']:
            datatf = bt.TimeFrame.Ticks
            datacomp = 1
        else:
            datatf = timeframe
            datacomp = args['compression']
    
        #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData
        IBDataFactory = ibstore.getdata
    
        #Create the data using datatf and datacomp
        data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp)
    
        #Feed the strategy with resampled data if requested
        if args['resample']:
            cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression'])
        else:
            cerebro.adddata(data0)
    
        # Add the strategy
        cerebro.addstrategy(TestStrategy)
    
        cerebro.run()
    

    Sample output:

    Data0, 0248, 737092.0, 2019-02-01T23:59:59.999989, 138.28, 139.42, 136.8, 138.84, 54488196.0, 0.0, 136.928
    Data0, 0249, 737095.0, 2019-02-04T23:59:59.999989, 139.2, 139.76, 136.7, 138.1, 52807693.0, 0.0, 137.46
    Data0, 0250, 737096.0, 2019-02-05T23:59:59.999989, 137.84, 141.54, 137.82, 141.06, 55387568.0, 0.0, 138.656
    Data0, 0251, 737097.0, 2019-02-06T23:59:59.999989, 142.2, 143.62, 142.0, 142.4, 55193955.0, 0.0, 139.796
    Data0, 0252, 737098.0, 2019-02-07T23:59:59.999989, 141.5, 141.76, 138.84, 139.38, 60064548.0, 0.0, 139.956
    Data0, 0253, 737098.0, 2019-02-08T00:00:00.000000, 139.0, 139.46, 138.52, 138.96, 5210654.0, 0.0, 139.98
    ***** DATA NOTIF: LIVE
    Data0, 0254, 737098.396565, 2019-02-08T09:31:03.195999, 138.96, 138.96, 138.96, 138.96, 8127.0, 0.0, 140.152
    Data0, 0255, 737098.397154, 2019-02-08T09:31:54.083995, 138.96, 138.96, 138.96, 138.96, 600.0, 0.0, 139.732
    Data0, 0256, 737098.397257, 2019-02-08T09:32:03.009997, 138.92, 138.92, 138.92, 138.92, 24490.0, 0.0, 139.036
    Data0, 0257, 737098.397405, 2019-02-08T09:32:15.790003, 138.9, 138.9, 138.9, 138.9, 4412.0, 0.0, 138.94
    Data0, 0258, 737098.397521, 2019-02-08T09:32:25.818997, 138.88, 138.88, 138.88, 138.88, 3223.0, 0.0, 138.924
    

    Loving the package and support! Appreciate any guidance. Many thanks.


  • administrators

    @seano said in IB delayed vs live timeframes (interday trading):

    I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB

    There is no switch in backtrader. IB doesn't provide full bars in real-time, it does only provide ticks (or 5-seconds real-time bars). Those ticks need to be resampled to days. The execution snippets for ibtest.py show it.

    @seano said in IB delayed vs live timeframes (interday trading):

    • Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...

    The daily close is obviously not available until the market closes, so you may want to reformulate your question.

    @seano said in IB delayed vs live timeframes (interday trading):

    • Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?

    Resampling should do it, but you say you have missing bars, but we haven't seen that above.

    @seano said in IB delayed vs live timeframes (interday trading):

    Sample output:

    What was the input? (Because you have different execution paths)



  • @backtrader said in IB delayed vs live timeframes (interday trading):

    Thanks for quick response. To rephrase my question, I was asking whether there was intentional logic to 'switch' from daily to tick (or 5-seconds real-time bars) when going from delayed to live data... but as explained, there is no such switch so problem must be elsewhere.

    Just ran the script for TWTR using resampling and you will see that the delayed data below has missing dates:

    Data0, 0108, 737069.208333, 2019-01-10T00:00:00.000000, 29.9, 32.4, 29.76, 32.02, 470227.0, 0.0, 29.28
    Data0, 0109, 737072.208333, 2019-01-13T00:00:00.000000, 32.02, 33.5, 32.02, 32.77, 462363.0, 0.0, 30.448
    Data0, 0110, 737076.208333, 2019-01-17T00:00:00.000000, 32.75, 33.35, 32.12, 32.49, 313474.0, 0.0, 31.232
    Data0, 0111, 737079.208333, 2019-01-20T00:00:00.000000, 32.32, 33.89, 32.24, 33.2, 255231.0, 0.0, 32.116
    Data0, 0112, 737084.208333, 2019-01-25T00:00:00.000000, 33.01, 33.35, 30.72, 31.58, 455923.0, 0.0, 32.412
    Data0, 0113, 737089.208333, 2019-01-30T00:00:00.000000, 31.66, 33.67, 31.46, 31.91, 602043.0, 0.0, 32.39
    Data0, 0114, 737092.208333, 2019-02-02T00:00:00.000000, 32.0, 34.09, 31.42, 33.2, 526106.0, 0.0, 32.476
    Data0, 0115, 737097.208333, 2019-02-07T00:00:00.000000, 33.3, 35.29, 33.24, 35.05, 621249.0, 0.0, 32.988
    ***** DATA NOTIF: LIVE
    

    Full script below:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import argparse
    import datetime
    
    # The above could be sent to an independent module
    import backtrader as bt
    from backtrader.utils import flushfile  # win32 quick stdout flushing
    
    
    class TestStrategy(bt.Strategy):
        params = dict(
            smaperiod=5,
            trade=True,
            stake=10,
            exectype=bt.Order.Market,
            stopafter=20,
            valid=None,
            cancel=0,
            donotsell=False,
            stoptrail=False,
            stoptraillimit=False,
            trailamount=None,
            trailpercent=None,
            limitoffset=None,
            oca=False,
            bracket=False,
            stop_loss=0.01, # NEW: price is 5% less than the entry point (Redundant if use trail I presume)
            trail=False, # NEW: NOTE IT IS SIMILAR TO stoptrail; set trail=False if want fixed stop specified in stop_loss; OR set to a numeric value (e.g. 0.10) will tell the strategy to use a StopTrail
        )
    
    
        def __init__(self):
            # To control operation entries
            self.orderid = list()
            self.order = None
    
            self.counttostop = 0
            self.datastatus = 0
    
            # NEW: Keep a reference to the "close" line in the data[0] dataseries
            self.dataclose = self.data0.close
            
            # NEW: Add a Highest Close indicator
            # TODO: Note that doing this with period=100 means trades don't start until after 100 periods
            # Should look at a way to simply keep a running record of highest high (in all history) instead
            self.the_highest_close = bt.ind.Highest(self.data0.close, period=5, subplot=False)
            
            # Create SMA on 2nd data
            self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod)
    
            print('--------------------------------------------------')
            print('Strategy Created')
            print('--------------------------------------------------')
    
        def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
            if status == data.LIVE:
                self.counttostop = self.p.stopafter
                self.datastatus = 1
    
        def notify_store(self, msg, *args, **kwargs):
            print('*' * 5, 'STORE NOTIF:', msg)
    
        def notify_order(self, order):
            if order.status in [order.Completed, order.Cancelled, order.Rejected]:
                self.order = None
    
            print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
            print(order)
            print('-' * 50, 'ORDER END')
    
        def notify_trade(self, trade):
            print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
            print(trade)
            print('-' * 50, 'TRADE END')
    
        def prenext(self):
            self.next(frompre=True)
    
        def next(self, frompre=False):
            txt = list()
            txt.append('Data0')
            txt.append('%04d' % len(self.data0))
            dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
            txt.append('{}'.format(self.data.datetime[0]))
            txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
            txt.append('{}'.format(self.data.open[0]))
            txt.append('{}'.format(self.data.high[0]))
            txt.append('{}'.format(self.data.low[0]))
            txt.append('{}'.format(self.data.close[0]))
            txt.append('{}'.format(self.data.volume[0]))
            txt.append('{}'.format(self.data.openinterest[0]))
            txt.append('{}'.format(self.sma[0]))
            #txt.append('{}'.format(self.the_highest_close[0])) #NEW: to see if it should trigger
            #txt.append('{}'.format(self.dataclose[0])) #NEW: to see if it should trigger
            print(', '.join(txt))
    
            if len(self.datas) > 1 and len(self.data1):
                txt = list()
                txt.append('Data1')
                txt.append('%04d' % len(self.data1))
                dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
                txt.append('{}'.format(self.data1.datetime[0]))
                txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt))
                txt.append('{}'.format(self.data1.open[0]))
                txt.append('{}'.format(self.data1.high[0]))
                txt.append('{}'.format(self.data1.low[0]))
                txt.append('{}'.format(self.data1.close[0]))
                txt.append('{}'.format(self.data1.volume[0]))
                txt.append('{}'.format(self.data1.openinterest[0]))
                txt.append('{}'.format(float('NaN')))
                print(', '.join(txt))
    
            if self.counttostop:  # stop after x live lines
                self.counttostop -= 1
                if not self.counttostop:
                    self.env.runstop()
                    return
    
            if not self.p.trade:
                return
    
            if self.datastatus and not self.position: # and len(self.orderid) < 1: #NEW Removed this
                exectype = self.p.exectype if not self.p.oca else bt.Order.Limit
                close = self.data0.close[0]
                price = round(close * 0.90, 2)
                
                # NEW: Buy rule => Not yet ... we MIGHT BUY if ... 
                if self.dataclose[0] > self.the_highest_close[-1]:
                        # A BREAKOUT !!!
    
                        # BUY, BUY, BUY!!! (with default parameters)
                        # self.log('BUY CREATE, %.2f' % self.dataclose[0])
    
                        # Keep track of the created order to avoid a 2nd order
                        # self.order = self.buy()
                        
                        self.order = self.buy(size=self.p.stake,
                                      exectype=exectype,
                                      price=price,
                                      valid=self.p.valid,
                                      transmit=not self.p.bracket)
                        
                        # Set a stop loss at the time of buy (need to enable cheat on close in __init__() first)
                        if not self.params.trail:
                            stop_price = round(self.dataclose[0] * (1.0 - self.params.stop_loss), 2)
                            self.sell(size=self.p.stake, exectype=bt.Order.Stop, price=stop_price)
                        else:
                            # Originally used trailamount... but I changed to trailpercent
                            # https://www.backtrader.com/docu/order-creation-execution/bracket/bracket.html?highlight=stoptrail
                            self.sell(size=self.p.stake, exectype=bt.Order.StopTrail,
                                      trailpercent=self.p.trail)
                
                
                self.orderid.append(self.order)
                
                # TODO: Look into if I should be using these instead of my stop logic above
                if self.p.bracket:
                    # low side
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.Stop,
                              price=round(price * 0.90, 2),
                              valid=self.p.valid,
                              transmit=False,
                              parent=self.order)
    
                    # high side
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.Limit,
                              price=round(close * 1.10, 2),
                              valid=self.p.valid,
                              transmit=True,
                              parent=self.order)
    
                elif self.p.oca:
                    self.buy(size=self.p.stake,
                             exectype=bt.Order.Limit,
                             price=round(self.data0.close[0] * 0.80, 2),
                             oco=self.order)
    
                elif self.p.stoptrail:
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.StopTrail,
                              # price=round(self.data0.close[0] * 0.90, 2),
                              valid=self.p.valid,
                              trailamount=self.p.trailamount,
                              trailpercent=self.p.trailpercent)
    
                elif self.p.stoptraillimit:
                    p = round(self.data0.close[0] - self.p.trailamount, 2)
                    # p = self.data0.close[0]
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.StopTrailLimit,
                              price=p,
                              plimit=p + self.p.limitoffset,
                              valid=self.p.valid,
                              trailamount=self.p.trailamount,
                              trailpercent=self.p.trailpercent)
    
            # TAKE THIS OUT AND JUST HAVE THE STOP LOSS
            #elif self.position.size > 0 and not self.p.donotsell:
            #    if self.order is None:
            #        self.order = self.sell(size=self.p.stake // 2,
            #                               exectype=bt.Order.Market,
            #                               price=self.data0.close[0])
    
            elif self.order is not None and self.p.cancel:
                if self.datastatus > self.p.cancel:
                    self.cancel(self.order)
    
            if self.datastatus:
                self.datastatus += 1
    
        def start(self):
            if self.data0.contractdetails is not None:
                print('Timezone from ContractDetails: {}'.format(
                      self.data0.contractdetails.m_timeZoneId))
    
            header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume',
                      'OpenInterest', 'SMA']
            print(', '.join(header))
    
            self.done = False
    
    
    def runstrategy():
        
        # Hardcode the args
        args = dict(
            host='127.0.0.1',
            port=7497, #7496 for live,
            clientId=None,
            data0='TWTR', #'TWTR', #None, #'EUR.USD-CASH-IDEALPRO'RWA-STK-LSE'
            data1=None,
            resample=True, #False,
            timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days'
            compression=1, #5,
        )
        
        # Create a cerebro
        cerebro = bt.Cerebro()
    
        #Use store method
        ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId'])
        broker = ibstore.getbroker()
        cerebro.setbroker(broker)
    
        #Set the timeframe
        timeframe = bt.TimeFrame.TFrame(args['timeframe'])
    
        if args['resample']:
            datatf = bt.TimeFrame.Ticks
            datacomp = 1
        else:
            datatf = timeframe
            datacomp = args['compression']
    
        #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData
        IBDataFactory = ibstore.getdata
    
        #Create the data using datatf and datacomp
        data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp)
    
        #Feed the strategy with resampled data if requested
        if args['resample']:
            cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression'])
        else:
            cerebro.adddata(data0)
    
        # Add the strategy
        cerebro.addstrategy(TestStrategy)
    
        cerebro.run()
    
    
    if __name__ == '__main__':
        runstrategy()
    

Log in to reply
 

});