For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
IB delayed vs live timeframes (interday trading)
-
Hi Backtrader (and community),
I'm automating my existing manual system. I've recently hit something I'm struggling to troubleshoot so hope you can help. Intent is to use backtrader for live trading strategies that take a pre-screened short-list of LSE stocks and waits for break-outs (hitting new highs) before buying with a fixed/trailing stop. I can get the multiple datafeeds and trade triggers working ok, but have issues with the timeframes when implementing live system. Need to use daily data, looking at new highs of most recent close for the break-outs - only really need to run the script daily as most positions will be bought at market and held for more than a day (potentially a week or more).
I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB. If I use resample then I'm getting only a small set of the daily data (missing days) so suspect it's actually resampling the daily data.
Some specific questions:
- Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...
- Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?
Below is main snippet I'm using to test:
def runstrategy(): # Hardcode the args args = dict( host='127.0.0.1', port=7497, #7496 for live, clientId=None, data0='VOD-STK-LSE', #'TWTR', 'EUR.USD-CASH-IDEALPRO', 'RWA-STK-LSE' data1=None, resample=False, #False, timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days' compression=1, #5, ) # Create a cerebro cerebro = bt.Cerebro() #Use store method ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId']) broker = ibstore.getbroker() cerebro.setbroker(broker) #Set the timeframe timeframe = bt.TimeFrame.TFrame(args['timeframe']) if args['resample']: datatf = bt.TimeFrame.Ticks datacomp = 1 else: datatf = timeframe datacomp = args['compression'] #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData IBDataFactory = ibstore.getdata #Create the data using datatf and datacomp data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp) #Feed the strategy with resampled data if requested if args['resample']: cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression']) else: cerebro.adddata(data0) # Add the strategy cerebro.addstrategy(TestStrategy) cerebro.run()Sample output:
Data0, 0248, 737092.0, 2019-02-01T23:59:59.999989, 138.28, 139.42, 136.8, 138.84, 54488196.0, 0.0, 136.928 Data0, 0249, 737095.0, 2019-02-04T23:59:59.999989, 139.2, 139.76, 136.7, 138.1, 52807693.0, 0.0, 137.46 Data0, 0250, 737096.0, 2019-02-05T23:59:59.999989, 137.84, 141.54, 137.82, 141.06, 55387568.0, 0.0, 138.656 Data0, 0251, 737097.0, 2019-02-06T23:59:59.999989, 142.2, 143.62, 142.0, 142.4, 55193955.0, 0.0, 139.796 Data0, 0252, 737098.0, 2019-02-07T23:59:59.999989, 141.5, 141.76, 138.84, 139.38, 60064548.0, 0.0, 139.956 Data0, 0253, 737098.0, 2019-02-08T00:00:00.000000, 139.0, 139.46, 138.52, 138.96, 5210654.0, 0.0, 139.98 ***** DATA NOTIF: LIVE Data0, 0254, 737098.396565, 2019-02-08T09:31:03.195999, 138.96, 138.96, 138.96, 138.96, 8127.0, 0.0, 140.152 Data0, 0255, 737098.397154, 2019-02-08T09:31:54.083995, 138.96, 138.96, 138.96, 138.96, 600.0, 0.0, 139.732 Data0, 0256, 737098.397257, 2019-02-08T09:32:03.009997, 138.92, 138.92, 138.92, 138.92, 24490.0, 0.0, 139.036 Data0, 0257, 737098.397405, 2019-02-08T09:32:15.790003, 138.9, 138.9, 138.9, 138.9, 4412.0, 0.0, 138.94 Data0, 0258, 737098.397521, 2019-02-08T09:32:25.818997, 138.88, 138.88, 138.88, 138.88, 3223.0, 0.0, 138.924Loving the package and support! Appreciate any guidance. Many thanks.
-
@seano said in IB delayed vs live timeframes (interday trading):
I used ibtest.py as original template. When I set timeframe = days and compression = 1 in datafeed the historical 'delayed' data is daily but it switches to tick data when switching to 'live' feed from IB
There is no switch in backtrader.
IBdoesn't provide full bars in real-time, it does only provide ticks (or 5-seconds real-time bars). Those ticks need to be resampled to days. The execution snippets foribtest.pyshow it.@seano said in IB delayed vs live timeframes (interday trading):
- Am I missing something obvious here in terms of parameters/approach? Or is this intentional logic seeing as daily close data wouldn't ever be available until market close anyway...
The daily close is obviously not available until the market closes, so you may want to reformulate your question.
@seano said in IB delayed vs live timeframes (interday trading):
- Should I just ignore the fact that live will call next() for every tick and just build functionality to only look at first tick on calling (if triggering script daily) or perhaps using timers?
Resampling should do it, but you say you have missing bars, but we haven't seen that above.
@seano said in IB delayed vs live timeframes (interday trading):
Sample output:
What was the input? (Because you have different execution paths)
-
@backtrader said in IB delayed vs live timeframes (interday trading):
Thanks for quick response. To rephrase my question, I was asking whether there was intentional logic to 'switch' from daily to tick (or 5-seconds real-time bars) when going from delayed to live data... but as explained, there is no such switch so problem must be elsewhere.
Just ran the script for TWTR using resampling and you will see that the delayed data below has missing dates:
Data0, 0108, 737069.208333, 2019-01-10T00:00:00.000000, 29.9, 32.4, 29.76, 32.02, 470227.0, 0.0, 29.28 Data0, 0109, 737072.208333, 2019-01-13T00:00:00.000000, 32.02, 33.5, 32.02, 32.77, 462363.0, 0.0, 30.448 Data0, 0110, 737076.208333, 2019-01-17T00:00:00.000000, 32.75, 33.35, 32.12, 32.49, 313474.0, 0.0, 31.232 Data0, 0111, 737079.208333, 2019-01-20T00:00:00.000000, 32.32, 33.89, 32.24, 33.2, 255231.0, 0.0, 32.116 Data0, 0112, 737084.208333, 2019-01-25T00:00:00.000000, 33.01, 33.35, 30.72, 31.58, 455923.0, 0.0, 32.412 Data0, 0113, 737089.208333, 2019-01-30T00:00:00.000000, 31.66, 33.67, 31.46, 31.91, 602043.0, 0.0, 32.39 Data0, 0114, 737092.208333, 2019-02-02T00:00:00.000000, 32.0, 34.09, 31.42, 33.2, 526106.0, 0.0, 32.476 Data0, 0115, 737097.208333, 2019-02-07T00:00:00.000000, 33.3, 35.29, 33.24, 35.05, 621249.0, 0.0, 32.988 ***** DATA NOTIF: LIVEFull script below:
from __future__ import (absolute_import, division, print_function, unicode_literals) import argparse import datetime # The above could be sent to an independent module import backtrader as bt from backtrader.utils import flushfile # win32 quick stdout flushing class TestStrategy(bt.Strategy): params = dict( smaperiod=5, trade=True, stake=10, exectype=bt.Order.Market, stopafter=20, valid=None, cancel=0, donotsell=False, stoptrail=False, stoptraillimit=False, trailamount=None, trailpercent=None, limitoffset=None, oca=False, bracket=False, stop_loss=0.01, # NEW: price is 5% less than the entry point (Redundant if use trail I presume) trail=False, # NEW: NOTE IT IS SIMILAR TO stoptrail; set trail=False if want fixed stop specified in stop_loss; OR set to a numeric value (e.g. 0.10) will tell the strategy to use a StopTrail ) def __init__(self): # To control operation entries self.orderid = list() self.order = None self.counttostop = 0 self.datastatus = 0 # NEW: Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.data0.close # NEW: Add a Highest Close indicator # TODO: Note that doing this with period=100 means trades don't start until after 100 periods # Should look at a way to simply keep a running record of highest high (in all history) instead self.the_highest_close = bt.ind.Highest(self.data0.close, period=5, subplot=False) # Create SMA on 2nd data self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod) print('--------------------------------------------------') print('Strategy Created') print('--------------------------------------------------') def notify_data(self, data, status, *args, **kwargs): print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args) if status == data.LIVE: self.counttostop = self.p.stopafter self.datastatus = 1 def notify_store(self, msg, *args, **kwargs): print('*' * 5, 'STORE NOTIF:', msg) def notify_order(self, order): if order.status in [order.Completed, order.Cancelled, order.Rejected]: self.order = None print('-' * 50, 'ORDER BEGIN', datetime.datetime.now()) print(order) print('-' * 50, 'ORDER END') def notify_trade(self, trade): print('-' * 50, 'TRADE BEGIN', datetime.datetime.now()) print(trade) print('-' * 50, 'TRADE END') def prenext(self): self.next(frompre=True) def next(self, frompre=False): txt = list() txt.append('Data0') txt.append('%04d' % len(self.data0)) dtfmt = '%Y-%m-%dT%H:%M:%S.%f' txt.append('{}'.format(self.data.datetime[0])) txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt)) txt.append('{}'.format(self.data.open[0])) txt.append('{}'.format(self.data.high[0])) txt.append('{}'.format(self.data.low[0])) txt.append('{}'.format(self.data.close[0])) txt.append('{}'.format(self.data.volume[0])) txt.append('{}'.format(self.data.openinterest[0])) txt.append('{}'.format(self.sma[0])) #txt.append('{}'.format(self.the_highest_close[0])) #NEW: to see if it should trigger #txt.append('{}'.format(self.dataclose[0])) #NEW: to see if it should trigger print(', '.join(txt)) if len(self.datas) > 1 and len(self.data1): txt = list() txt.append('Data1') txt.append('%04d' % len(self.data1)) dtfmt = '%Y-%m-%dT%H:%M:%S.%f' txt.append('{}'.format(self.data1.datetime[0])) txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt)) txt.append('{}'.format(self.data1.open[0])) txt.append('{}'.format(self.data1.high[0])) txt.append('{}'.format(self.data1.low[0])) txt.append('{}'.format(self.data1.close[0])) txt.append('{}'.format(self.data1.volume[0])) txt.append('{}'.format(self.data1.openinterest[0])) txt.append('{}'.format(float('NaN'))) print(', '.join(txt)) if self.counttostop: # stop after x live lines self.counttostop -= 1 if not self.counttostop: self.env.runstop() return if not self.p.trade: return if self.datastatus and not self.position: # and len(self.orderid) < 1: #NEW Removed this exectype = self.p.exectype if not self.p.oca else bt.Order.Limit close = self.data0.close[0] price = round(close * 0.90, 2) # NEW: Buy rule => Not yet ... we MIGHT BUY if ... if self.dataclose[0] > self.the_highest_close[-1]: # A BREAKOUT !!! # BUY, BUY, BUY!!! (with default parameters) # self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order # self.order = self.buy() self.order = self.buy(size=self.p.stake, exectype=exectype, price=price, valid=self.p.valid, transmit=not self.p.bracket) # Set a stop loss at the time of buy (need to enable cheat on close in __init__() first) if not self.params.trail: stop_price = round(self.dataclose[0] * (1.0 - self.params.stop_loss), 2) self.sell(size=self.p.stake, exectype=bt.Order.Stop, price=stop_price) else: # Originally used trailamount... but I changed to trailpercent # https://www.backtrader.com/docu/order-creation-execution/bracket/bracket.html?highlight=stoptrail self.sell(size=self.p.stake, exectype=bt.Order.StopTrail, trailpercent=self.p.trail) self.orderid.append(self.order) # TODO: Look into if I should be using these instead of my stop logic above if self.p.bracket: # low side self.sell(size=self.p.stake, exectype=bt.Order.Stop, price=round(price * 0.90, 2), valid=self.p.valid, transmit=False, parent=self.order) # high side self.sell(size=self.p.stake, exectype=bt.Order.Limit, price=round(close * 1.10, 2), valid=self.p.valid, transmit=True, parent=self.order) elif self.p.oca: self.buy(size=self.p.stake, exectype=bt.Order.Limit, price=round(self.data0.close[0] * 0.80, 2), oco=self.order) elif self.p.stoptrail: self.sell(size=self.p.stake, exectype=bt.Order.StopTrail, # price=round(self.data0.close[0] * 0.90, 2), valid=self.p.valid, trailamount=self.p.trailamount, trailpercent=self.p.trailpercent) elif self.p.stoptraillimit: p = round(self.data0.close[0] - self.p.trailamount, 2) # p = self.data0.close[0] self.sell(size=self.p.stake, exectype=bt.Order.StopTrailLimit, price=p, plimit=p + self.p.limitoffset, valid=self.p.valid, trailamount=self.p.trailamount, trailpercent=self.p.trailpercent) # TAKE THIS OUT AND JUST HAVE THE STOP LOSS #elif self.position.size > 0 and not self.p.donotsell: # if self.order is None: # self.order = self.sell(size=self.p.stake // 2, # exectype=bt.Order.Market, # price=self.data0.close[0]) elif self.order is not None and self.p.cancel: if self.datastatus > self.p.cancel: self.cancel(self.order) if self.datastatus: self.datastatus += 1 def start(self): if self.data0.contractdetails is not None: print('Timezone from ContractDetails: {}'.format( self.data0.contractdetails.m_timeZoneId)) header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume', 'OpenInterest', 'SMA'] print(', '.join(header)) self.done = False def runstrategy(): # Hardcode the args args = dict( host='127.0.0.1', port=7497, #7496 for live, clientId=None, data0='TWTR', #'TWTR', #None, #'EUR.USD-CASH-IDEALPRO'RWA-STK-LSE' data1=None, resample=True, #False, timeframe='Days', #bt.TimeFrame.Names[0], #Try 'Minutes' or 'Days' compression=1, #5, ) # Create a cerebro cerebro = bt.Cerebro() #Use store method ibstore = bt.stores.IBStore(host=args['host'], port=args['port'], clientId=args['clientId']) broker = ibstore.getbroker() cerebro.setbroker(broker) #Set the timeframe timeframe = bt.TimeFrame.TFrame(args['timeframe']) if args['resample']: datatf = bt.TimeFrame.Ticks datacomp = 1 else: datatf = timeframe datacomp = args['compression'] #IBDataFactory = ibstore.getdata if args['usestore'] else bt.feeds.IBData IBDataFactory = ibstore.getdata #Create the data using datatf and datacomp data0 = IBDataFactory(dataname=args['data0'], timeframe=datatf, compression=datacomp) #Feed the strategy with resampled data if requested if args['resample']: cerebro.resampledata(data0, timeframe=timeframe, compression=args['compression']) else: cerebro.adddata(data0) # Add the strategy cerebro.addstrategy(TestStrategy) cerebro.run() if __name__ == '__main__': runstrategy()