For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
Indicators show different values with LIVE streaming data
-
Hello,
During my testing across days, I've observed that Indicator values show correctly when using historical data. However when using LIVE streaming data, example, 1 minute resample of IB LIVE data, I could always see that generated indicator values are different to that of actual values. I compared these values with IB-TWS and even with tradingview. This consistently deviates from actual values.
I have tested using different sample periods etc. Not able to figure out what could be the reason. Even tried playing with resampling flags like boundoff, rightedge, rtbar, qcheck et all
Indicators I've tested and found to be not consistent with LIVE data are Stochastic Oscillator, MACD, ADX, PSAR etc
Is this working fine for anyone with IB live data?
Thanks in advance.
-
@mv do you use the same number of bars to calculate indicator values with historical data and with live data?
-
@ab_trader just to clarify - not indicator period, but total number of bars available?
-
Hello @ab_trader, appreciate your response. Yes, I try to keep all parameters same for both scenarios.
For example, at end of the day I stop the Live run and run the script for historical data and I see that values generated for all indicators are off. Sometimes during live run I restart script and I see values match before LIVE data and slowly values deviate.Have you been using indicators with IB Live data? If it’s working for you then I could doing something wrong. Can you please test the same duration for both live and historical and see if they match.
Is there anything that I could test or tweak to debug this issue further?
Im suspecting issue with resampling as I could see data being sent from IB Gateway continuously at 5secs without a miss, didn’t validate the data though.
Thanks @ab_trader once again for looking into this
-
Indicators depend on the prices and on the data feed length in case of recursive indicators. Indicators mentioned above are mostly recursive. There are no logs and script provided, therefore hard to say what is going on.
To continue debugging please provide the following:
- script
- prices and indicator values with the timestamps and data feed length logged on every
next()during the LIVE session - prices and indicator values with the timestamps and data feed length logged on every
next()made during historical backtest session - timestamps (bar length) should be the same for both logs
-
Hi @ab_trader ,
Today I have run with fresh data to help us debug further. Looks like data is shifted by 1 bar during Live resampling
- I used ibtest.py from framework samples and added indicators to it, below is the complete script. I made sure of minimal changes required
- I've used writer functionality and CSV's generated by the script with data and indicator values are uploaded to shared GogoleDrive folder below
- Drive Link has logs of both Live and Historical runs
- I made an XLS comparison of Live vs Historical Vs TradingView data. I could see that till data is LIVE all values matched and once data is LIVE values are off. This can be clearly seen between LIVE and Historical. Historical is very close to TradingView data
Observations from below Comparison. I'm using just Stochastic K and D values
- Values match till data goes LIVE. Len 1-44 matches between Live and Historical
- After data goes LIVE from Len 44, Live data/indicator values deviate
- After few lines you can see that Live data is shifted by a bar/minute. In the below image, you can see lines highlighted blue. After about 11-14 lines which is the indicator period, you can clearly see that Live data matches with previous line of historical and TradingView. Not just for indicator values, even the CLOSE values follow this +1 bar pattern for Live data.
Original size image is available in GoogleDrive folder link posted
Your help is very much appreciated.
#!/usr/bin/env python from __future__ import (absolute_import, division, print_function, unicode_literals) import argparse import datetime import os.path # To manage paths import sys # To find out the script name (in argv[0]) from backtrader import backtrader as bt from backtrader.utils import flushfile # win32 quick stdout flushing class TestStrategy(bt.Strategy): params = dict( smaperiod=5, trade=False, stake=10, exectype=bt.Order.Market, stopafter=0, valid=None, cancel=0, donotsell=False, stoptrail=False, stoptraillimit=False, trailamount=None, trailpercent=None, limitoffset=None, oca=False, bracket=False, period=14, pfast=3, pslow=3, upperLimit=80, lowerLimit=20, # PSAR psarPeriod=2, psarAf=0.02, psarAfMax=0.2, # MACD macd_1=5, macd_2=20, macd_sig=11, ) def __init__(self): # To control operation entries self.orderid = list() self.order = None self.counttostop = 0 self.datastatus = 0 # Create SMA on 2nd data self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod) self.sma.csv = True self.stochData0 = bt.indicators.Stochastic(self.datas[0], period=self.params.period, period_dfast=self.params.pfast, period_dslow=self.params.pslow, upperband=self.params.upperLimit, lowerband=self.params.lowerLimit, plot = True) self.stochData0.csv = True # Add PSAR Indicator self.psarData0 = bt.indicators.ParabolicSAR(self.datas[0], period=self.params.psarPeriod, af=self.params.psarAf, afmax=self.params.psarAfMax, plot = True) self.psarData0.csv = True self.macdData0 = bt.indicators.MACD(self.datas[0], period_me1=self.params.macd_1, period_me2=self.params.macd_2, period_signal=self.params.macd_sig, plot=True) self.macdData0.csv = True print('--------------------------------------------------') print('Strategy Created') print('--------------------------------------------------') def notify_data(self, data, status, *args, **kwargs): print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args) if status == data.LIVE: self.counttostop = self.p.stopafter self.datastatus = 1 def notify_store(self, msg, *args, **kwargs): print('*' * 5, 'STORE NOTIF:', msg) def notify_order(self, order): if order.status in [order.Completed, order.Cancelled, order.Rejected]: self.order = None print('-' * 50, 'ORDER BEGIN', datetime.datetime.now()) print(order) print('-' * 50, 'ORDER END') def notify_trade(self, trade): print('-' * 50, 'TRADE BEGIN', datetime.datetime.now()) print(trade) print('-' * 50, 'TRADE END') def prenext(self): self.next(frompre=True) def next(self, frompre=False): txt = list() txt.append('Data0') txt.append('%04d' % len(self.data0)) dtfmt = '%Y-%m-%dT%H:%M:%S.%f' txt.append('{}'.format(self.data.datetime[0])) txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt)) txt.append('{}'.format(self.data.open[0])) txt.append('{}'.format(self.data.high[0])) txt.append('{}'.format(self.data.low[0])) txt.append('{}'.format(self.data.close[0])) txt.append('{}'.format(self.data.volume[0])) txt.append('{}'.format(self.data.openinterest[0])) txt.append('{}'.format(self.sma[0])) txt.append('{}'.format(self.stochData0.lines.percK[0])) txt.append('{}'.format(self.stochData0.lines.percD[0])) txt.append('{}'.format(self.macdData0.lines.macd[0])) txt.append('{}'.format(self.macdData0.lines.signal[0])) txt.append('{}'.format(self.psarData0.lines.psar[0])) print(', '.join(txt)) if len(self.datas) > 1 and len(self.data1): txt = list() txt.append('Data1') txt.append('%04d' % len(self.data1)) dtfmt = '%Y-%m-%dT%H:%M:%S.%f' txt.append('{}'.format(self.data1.datetime[0])) txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt)) txt.append('{}'.format(self.data1.open[0])) txt.append('{}'.format(self.data1.high[0])) txt.append('{}'.format(self.data1.low[0])) txt.append('{}'.format(self.data1.close[0])) txt.append('{}'.format(self.data1.volume[0])) txt.append('{}'.format(self.data1.openinterest[0])) txt.append('{}'.format(float('NaN'))) print(', '.join(txt)) if self.counttostop: # stop after x live lines self.counttostop -= 1 if not self.counttostop: self.env.runstop() return if not self.p.trade: return if self.datastatus and not self.position and len(self.orderid) < 1: exectype = self.p.exectype if not self.p.oca else bt.Order.Limit close = self.data0.close[0] price = round(close * 0.90, 2) self.order = self.buy(size=self.p.stake, exectype=exectype, price=price, valid=self.p.valid, transmit=not self.p.bracket) self.orderid.append(self.order) if self.p.bracket: # low side self.sell(size=self.p.stake, exectype=bt.Order.Stop, price=round(price * 0.90, 2), valid=self.p.valid, transmit=False, parent=self.order) # high side self.sell(size=self.p.stake, exectype=bt.Order.Limit, price=round(close * 1.10, 2), valid=self.p.valid, transmit=True, parent=self.order) elif self.p.oca: self.buy(size=self.p.stake, exectype=bt.Order.Limit, price=round(self.data0.close[0] * 0.80, 2), oco=self.order) elif self.p.stoptrail: self.sell(size=self.p.stake, exectype=bt.Order.StopTrail, # price=round(self.data0.close[0] * 0.90, 2), valid=self.p.valid, trailamount=self.p.trailamount, trailpercent=self.p.trailpercent) elif self.p.stoptraillimit: p = round(self.data0.close[0] - self.p.trailamount, 2) # p = self.data0.close[0] self.sell(size=self.p.stake, exectype=bt.Order.StopTrailLimit, price=p, plimit=p + self.p.limitoffset, valid=self.p.valid, trailamount=self.p.trailamount, trailpercent=self.p.trailpercent) elif self.position.size > 0 and not self.p.donotsell: if self.order is None: self.order = self.sell(size=self.p.stake // 2, exectype=bt.Order.Market, price=self.data0.close[0]) elif self.order is not None and self.p.cancel: if self.datastatus > self.p.cancel: self.cancel(self.order) if self.datastatus: self.datastatus += 1 def start(self): if self.data0.contractdetails is not None: print('Timezone from ContractDetails: {}'.format( self.data0.contractdetails.m_timeZoneId)) header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume', 'OpenInterest', 'SMA','StochK','StochD','MACD','MACDSig', 'PSAR'] print(', '.join(header)) self.done = False def runstrategy(): args = parse_args() # Create a cerebro cerebro = bt.Cerebro() storekwargs = dict( host=args.host, port=args.port, clientId=args.clientId, timeoffset=not args.no_timeoffset, reconnect=args.reconnect, timeout=args.timeout, notifyall=args.notifyall, _debug=args.debug ) if args.usestore: ibstore = bt.stores.IBStore(**storekwargs) if args.broker: if args.usestore: broker = ibstore.getbroker() else: broker = bt.brokers.IBBroker(**storekwargs) cerebro.setbroker(broker) timeframe = bt.TimeFrame.TFrame(args.timeframe) # Manage data1 parameters tf1 = args.timeframe1 tf1 = bt.TimeFrame.TFrame(tf1) if tf1 is not None else timeframe cp1 = args.compression1 cp1 = cp1 if cp1 is not None else args.compression if args.resample or args.replay: datatf = datatf1 = bt.TimeFrame.Ticks datacomp = datacomp1 = 1 else: datatf = timeframe datacomp = args.compression datatf1 = tf1 datacomp1 = cp1 fromdate = None if args.fromdate: dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate)) fromdate = datetime.datetime.strptime(args.fromdate, dtformat) todate = None if args.todate: dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.todate)) todate = datetime.datetime.strptime(args.todate, dtformat) IBDataFactory = ibstore.getdata if args.usestore else bt.feeds.IBData datakwargs = dict( timeframe=datatf, compression=datacomp, historical=args.historical, fromdate=fromdate, todate=todate, rtbar=args.rtbar, qcheck=args.qcheck, what=args.what, backfill_start=not args.no_backfill_start, backfill=not args.no_backfill, latethrough=args.latethrough, tz=args.timezone ) if not args.usestore and not args.broker: # neither store nor broker datakwargs.update(storekwargs) # pass the store args over the data data0 = IBDataFactory(dataname=args.data0, **datakwargs) data1 = None if args.data1 is not None: if args.data1 != args.data0: datakwargs['timeframe'] = datatf1 datakwargs['compression'] = datacomp1 data1 = IBDataFactory(dataname=args.data1, **datakwargs) else: data1 = data0 rekwargs = dict( timeframe=timeframe, compression=args.compression, bar2edge=not args.no_bar2edge, adjbartime=not args.no_adjbartime, rightedge=not args.no_rightedge, takelate=not args.no_takelate, ) if args.replay: cerebro.replaydata(data0, **rekwargs) if data1 is not None: rekwargs['timeframe'] = tf1 rekwargs['compression'] = cp1 cerebro.replaydata(data1, **rekwargs) elif args.resample: cerebro.resampledata(data0, **rekwargs) if data1 is not None: rekwargs['timeframe'] = tf1 rekwargs['compression'] = cp1 cerebro.resampledata(data1, **rekwargs) else: cerebro.adddata(data0) if data1 is not None: cerebro.adddata(data1) if args.valid is None: valid = None else: valid = datetime.timedelta(seconds=args.valid) # Add the strategy cerebro.addstrategy(TestStrategy, smaperiod=args.smaperiod, trade=args.trade, exectype=bt.Order.ExecType(args.exectype), stake=args.stake, stopafter=args.stopafter, valid=valid, cancel=args.cancel, donotsell=args.donotsell, stoptrail=args.stoptrail, stoptraillimit=args.traillimit, trailamount=args.trailamount, trailpercent=args.trailpercent, limitoffset=args.limitoffset, oca=args.oca, bracket=args.bracket) if args.write: now = datetime.datetime.today() scriptName = os.path.basename(__file__).split(".")[0] logFileName = "logs/" + now.strftime("%Y_%m_%d_%H%M%S_") + \ scriptName + "_ibtest" + "_log.csv" cerebro.addwriter(bt.WriterFile, csv=True, out=logFileName, rounding=2) # Live data ... avoid long data accumulation by switching to "exactbars" cerebro.run(exactbars=args.exactbars) if args.plot and args.exactbars < 1: # plot if possible cerebro.plot() def parse_args(): parser = argparse.ArgumentParser( formatter_class=argparse.ArgumentDefaultsHelpFormatter, description='Test Interactive Brokers integration') parser.add_argument('--exactbars', default=1, type=int, required=False, action='store', help='exactbars level, use 0/-1/-2 to enable plotting') parser.add_argument('--plot', required=False, action='store_true', help='Plot if possible') parser.add_argument('--stopafter', default=0, type=int, required=False, action='store', help='Stop after x lines of LIVE data') parser.add_argument('--usestore', default=False, required=False, action='store_true', help='Use the store pattern') parser.add_argument('--notifyall', required=False, action='store_true', help='Notify all messages to strategy as store notifs') parser.add_argument('--debug', default=False, required=False, action='store_true', help='Display all info received form IB') parser.add_argument('--host', default='127.0.0.1', required=False, action='store', help='Host for the Interactive Brokers TWS Connection') parser.add_argument('--qcheck', default=2.0, type=float, required=False, action='store', help=('Timeout for periodic ' 'notification/resampling/replaying check')) parser.add_argument('--port', default=4002, type=int, required=False, action='store', help='Port for the Interactive Brokers TWS Connection') parser.add_argument('--clientId', default='466824', type=int, required=False, action='store', help='Client Id to connect to TWS (default: random)') parser.add_argument('--no-timeoffset', required=False, action='store_true', help=('Do not Use TWS/System time offset for non ' 'timestamped prices and to align resampling')) parser.add_argument('--reconnect', default=3, type=int, required=False, action='store', help='Number of recconnection attempts to TWS') parser.add_argument('--timeout', default=3.0, type=float, required=False, action='store', help='Timeout between reconnection attempts to TWS') parser.add_argument('--data0', default='BANKNIFTY-IND-NSE-INR', required=False, action='store', help='data 0 into the system') parser.add_argument('--data1', default=None, required=False, action='store', help='data 1 into the system') parser.add_argument('--timezone', default=None, required=False, action='store', help='timezone to get time output into (pytz names)') parser.add_argument('--what', default=None, required=False, action='store', help='specific price type for historical requests') parser.add_argument('--no-backfill_start', required=False, action='store_true', help='Disable backfilling at the start') parser.add_argument('--latethrough', required=False, action='store_true', help=('if resampling replaying, adjusting time ' 'and disabling time offset, let late samples ' 'through')) parser.add_argument('--no-backfill', required=False, action='store_true', help='Disable backfilling after a disconnection') parser.add_argument('--rtbar', default=True, required=False, action='store_true', help='Use 5 seconds real time bar updates if possible') parser.add_argument('--historical', default=True, required=False, action='store_true', help='do only historical download') parser.add_argument('--fromdate', default="2020-09-16T09:15:00", required=False, action='store', help=('Starting date for historical download ' 'with format: YYYY-MM-DD[THH:MM:SS]')) parser.add_argument('--todate', default="2020-09-17T15:30:00", required=False, action='store', help=('Ending date for historical download ' 'with format: YYYY-MM-DD[THH:MM:SS]')) parser.add_argument('--smaperiod', default=10, type=int, required=False, action='store', help='Period to apply to the Simple Moving Average') pgroup = parser.add_mutually_exclusive_group(required=False) pgroup.add_argument('--replay', required=False, action='store_true', help='replay to chosen timeframe') pgroup.add_argument('--resample', default=True, required=False, action='store_true', help='resample to chosen timeframe') parser.add_argument('--timeframe', default='Minutes', choices=bt.TimeFrame.Names, required=False, action='store', help='TimeFrame for Resample/Replay') parser.add_argument('--compression', default=1, type=int, required=False, action='store', help='Compression for Resample/Replay') parser.add_argument('--timeframe1', default=None, choices=bt.TimeFrame.Names, required=False, action='store', help='TimeFrame for Resample/Replay - Data1') parser.add_argument('--compression1', default=None, type=int, required=False, action='store', help='Compression for Resample/Replay - Data1') parser.add_argument('--no-takelate', required=False, action='store_true', help=('resample/replay, do not accept late samples ' 'in new bar if the data source let them through ' '(latethrough)')) parser.add_argument('--no-bar2edge', required=False, action='store_true', help='no bar2edge for resample/replay') parser.add_argument('--no-adjbartime', required=False, action='store_true', help='no adjbartime for resample/replay') parser.add_argument('--no-rightedge', required=False, action='store_true', help='no rightedge for resample/replay') parser.add_argument('--broker', default=False, required=False, action='store_true', help='Use IB as broker') parser.add_argument('--trade', default= False, required=False, action='store_true', help='Do Sample Buy/Sell operations') parser.add_argument('--donotsell', required=False, action='store_true', help='Do not sell after a buy') parser.add_argument('--exectype', default=bt.Order.ExecTypes[0], choices=bt.Order.ExecTypes, required=False, action='store', help='Execution to Use when opening position') parser.add_argument('--stake', default=10, type=int, required=False, action='store', help='Stake to use in buy operations') parser.add_argument('--valid', default=None, type=int, required=False, action='store', help='Seconds to keep the order alive (0 means DAY)') pgroup = parser.add_mutually_exclusive_group(required=False) pgroup.add_argument('--stoptrail', required=False, action='store_true', help='Issue a stoptraillimit after buy( do not sell') pgroup.add_argument('--traillimit', required=False, action='store_true', help='Issue a stoptrail after buying (do not sell') pgroup.add_argument('--oca', required=False, action='store_true', help='Test oca by putting 2 orders in a group') pgroup.add_argument('--bracket', required=False, action='store_true', help='Test bracket orders by issuing high/low sides') pgroup = parser.add_mutually_exclusive_group(required=False) pgroup.add_argument('--trailamount', default=None, type=float, required=False, action='store', help='trailamount for StopTrail order') pgroup.add_argument('--trailpercent', default=None, type=float, required=False, action='store', help='trailpercent for StopTrail order') parser.add_argument('--limitoffset', default=None, type=float, required=False, action='store', help='limitoffset for StopTrailLimit orders') parser.add_argument('--cancel', default=0, type=int, required=False, action='store', help=('Cancel a buy order after n bars in operation,' ' to be combined with orders like Limit')) parser.add_argument('--write', default=True, required=False, action='store', help='Enable write observer data to file') return parser.parse_args() if __name__ == '__main__': runstrategy() -
It seems to me that you encountered the issue discussed here several times even in 2017. Try to search the forum for resampling with live data. As I remember
btuses one approach when do resampling, andibuses another approach for already formed bars. Honestly I didn't follow that discussions, and don't know if this was resolved or not.@vladisld may help if he has time.
-
@vladisld, Can you please help me with this issue. This is becoming a deal breaker as Indicator values are key to any automated trading systems. Do you use IB data or some other data source with backtrader? Did you notice any discrepancies with indicator values?