For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Indicators show different values with LIVE streaming data



  • Hello,

    During my testing across days, I've observed that Indicator values show correctly when using historical data. However when using LIVE streaming data, example, 1 minute resample of IB LIVE data, I could always see that generated indicator values are different to that of actual values. I compared these values with IB-TWS and even with tradingview. This consistently deviates from actual values.

    I have tested using different sample periods etc. Not able to figure out what could be the reason. Even tried playing with resampling flags like boundoff, rightedge, rtbar, qcheck et all

    Indicators I've tested and found to be not consistent with LIVE data are Stochastic Oscillator, MACD, ADX, PSAR etc

    Is this working fine for anyone with IB live data?

    Thanks in advance.



  • @mv do you use the same number of bars to calculate indicator values with historical data and with live data?



  • @ab_trader just to clarify - not indicator period, but total number of bars available?



  • Hello @ab_trader, appreciate your response. Yes, I try to keep all parameters same for both scenarios.
    For example, at end of the day I stop the Live run and run the script for historical data and I see that values generated for all indicators are off. Sometimes during live run I restart script and I see values match before LIVE data and slowly values deviate.

    Have you been using indicators with IB Live data? If it’s working for you then I could doing something wrong. Can you please test the same duration for both live and historical and see if they match.

    Is there anything that I could test or tweak to debug this issue further?

    Im suspecting issue with resampling as I could see data being sent from IB Gateway continuously at 5secs without a miss, didn’t validate the data though.

    Thanks @ab_trader once again for looking into this



  • Indicators depend on the prices and on the data feed length in case of recursive indicators. Indicators mentioned above are mostly recursive. There are no logs and script provided, therefore hard to say what is going on.

    To continue debugging please provide the following:

    • script
    • prices and indicator values with the timestamps and data feed length logged on every next() during the LIVE session
    • prices and indicator values with the timestamps and data feed length logged on every next() made during historical backtest session
    • timestamps (bar length) should be the same for both logs


  • Hi @ab_trader ,

    Today I have run with fresh data to help us debug further. Looks like data is shifted by 1 bar during Live resampling

    • I used ibtest.py from framework samples and added indicators to it, below is the complete script. I made sure of minimal changes required
    • I've used writer functionality and CSV's generated by the script with data and indicator values are uploaded to shared GogoleDrive folder below
    • Drive Link has logs of both Live and Historical runs
    • I made an XLS comparison of Live vs Historical Vs TradingView data. I could see that till data is LIVE all values matched and once data is LIVE values are off. This can be clearly seen between LIVE and Historical. Historical is very close to TradingView data

    Observations from below Comparison. I'm using just Stochastic K and D values

    1. Values match till data goes LIVE. Len 1-44 matches between Live and Historical
    2. After data goes LIVE from Len 44, Live data/indicator values deviate
    3. After few lines you can see that Live data is shifted by a bar/minute. In the below image, you can see lines highlighted blue. After about 11-14 lines which is the indicator period, you can clearly see that Live data matches with previous line of historical and TradingView. Not just for indicator values, even the CLOSE values follow this +1 bar pattern for Live data.

    Original size image is available in GoogleDrive folder link posted

    Google Drive Link

    Your help is very much appreciated.

    XLSData.jpg

    #!/usr/bin/env python
    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import argparse
    import datetime
    import os.path # To manage paths
    
    import sys # To find out the script name (in argv[0])
    
    from backtrader import backtrader as bt
    
    from backtrader.utils import flushfile  # win32 quick stdout flushing
    
    
    class TestStrategy(bt.Strategy):
        params = dict(
            smaperiod=5,
            trade=False,
            stake=10,
            exectype=bt.Order.Market,
            stopafter=0,
            valid=None,
            cancel=0,
            donotsell=False,
            stoptrail=False,
            stoptraillimit=False,
            trailamount=None,
            trailpercent=None,
            limitoffset=None,
            oca=False,
            bracket=False,
            period=14, pfast=3, pslow=3, upperLimit=80, lowerLimit=20,
            # PSAR
            psarPeriod=2,
            psarAf=0.02,
            psarAfMax=0.2,
            # MACD
            macd_1=5,
            macd_2=20,
            macd_sig=11,
    
        )
    
        def __init__(self):
            # To control operation entries
            self.orderid = list()
            self.order = None
    
            self.counttostop = 0
            self.datastatus = 0
    
            # Create SMA on 2nd data
            self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod)
            self.sma.csv = True
    
            self.stochData0 = bt.indicators.Stochastic(self.datas[0], period=self.params.period,
                                                   period_dfast=self.params.pfast,
                                                   period_dslow=self.params.pslow,
                                                   upperband=self.params.upperLimit,
                                                   lowerband=self.params.lowerLimit,
                                                   plot = True)
            self.stochData0.csv = True
    
            # Add PSAR Indicator
            self.psarData0 = bt.indicators.ParabolicSAR(self.datas[0],
                                                    period=self.params.psarPeriod,
                                                    af=self.params.psarAf,
                                                    afmax=self.params.psarAfMax,
                                                    plot = True)
            self.psarData0.csv = True
    
            self.macdData0 = bt.indicators.MACD(self.datas[0],
                                            period_me1=self.params.macd_1,
                                            period_me2=self.params.macd_2,
                                            period_signal=self.params.macd_sig,
                                            plot=True)
            self.macdData0.csv = True
    
            print('--------------------------------------------------')
            print('Strategy Created')
            print('--------------------------------------------------')
    
        def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
            if status == data.LIVE:
                self.counttostop = self.p.stopafter
                self.datastatus = 1
    
        def notify_store(self, msg, *args, **kwargs):
            print('*' * 5, 'STORE NOTIF:', msg)
    
        def notify_order(self, order):
            if order.status in [order.Completed, order.Cancelled, order.Rejected]:
                self.order = None
    
            print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
            print(order)
            print('-' * 50, 'ORDER END')
    
        def notify_trade(self, trade):
            print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
            print(trade)
            print('-' * 50, 'TRADE END')
    
        def prenext(self):
            self.next(frompre=True)
    
        def next(self, frompre=False):
            txt = list()
            txt.append('Data0')
            txt.append('%04d' % len(self.data0))
            dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
            txt.append('{}'.format(self.data.datetime[0]))
            txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
            txt.append('{}'.format(self.data.open[0]))
            txt.append('{}'.format(self.data.high[0]))
            txt.append('{}'.format(self.data.low[0]))
            txt.append('{}'.format(self.data.close[0]))
            txt.append('{}'.format(self.data.volume[0]))
            txt.append('{}'.format(self.data.openinterest[0]))
            txt.append('{}'.format(self.sma[0]))
            txt.append('{}'.format(self.stochData0.lines.percK[0]))
            txt.append('{}'.format(self.stochData0.lines.percD[0]))
            txt.append('{}'.format(self.macdData0.lines.macd[0]))
            txt.append('{}'.format(self.macdData0.lines.signal[0]))
            txt.append('{}'.format(self.psarData0.lines.psar[0]))
    
            print(', '.join(txt))
    
            if len(self.datas) > 1 and len(self.data1):
                txt = list()
                txt.append('Data1')
                txt.append('%04d' % len(self.data1))
                dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
                txt.append('{}'.format(self.data1.datetime[0]))
                txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt))
                txt.append('{}'.format(self.data1.open[0]))
                txt.append('{}'.format(self.data1.high[0]))
                txt.append('{}'.format(self.data1.low[0]))
                txt.append('{}'.format(self.data1.close[0]))
                txt.append('{}'.format(self.data1.volume[0]))
                txt.append('{}'.format(self.data1.openinterest[0]))
                txt.append('{}'.format(float('NaN')))
                print(', '.join(txt))
    
            if self.counttostop:  # stop after x live lines
                self.counttostop -= 1
                if not self.counttostop:
                    self.env.runstop()
                    return
    
            if not self.p.trade:
                return
    
            if self.datastatus and not self.position and len(self.orderid) < 1:
                exectype = self.p.exectype if not self.p.oca else bt.Order.Limit
                close = self.data0.close[0]
                price = round(close * 0.90, 2)
                self.order = self.buy(size=self.p.stake,
                                      exectype=exectype,
                                      price=price,
                                      valid=self.p.valid,
                                      transmit=not self.p.bracket)
    
                self.orderid.append(self.order)
    
                if self.p.bracket:
                    # low side
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.Stop,
                              price=round(price * 0.90, 2),
                              valid=self.p.valid,
                              transmit=False,
                              parent=self.order)
    
                    # high side
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.Limit,
                              price=round(close * 1.10, 2),
                              valid=self.p.valid,
                              transmit=True,
                              parent=self.order)
    
                elif self.p.oca:
                    self.buy(size=self.p.stake,
                             exectype=bt.Order.Limit,
                             price=round(self.data0.close[0] * 0.80, 2),
                             oco=self.order)
    
                elif self.p.stoptrail:
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.StopTrail,
                              # price=round(self.data0.close[0] * 0.90, 2),
                              valid=self.p.valid,
                              trailamount=self.p.trailamount,
                              trailpercent=self.p.trailpercent)
    
                elif self.p.stoptraillimit:
                    p = round(self.data0.close[0] - self.p.trailamount, 2)
                    # p = self.data0.close[0]
                    self.sell(size=self.p.stake,
                              exectype=bt.Order.StopTrailLimit,
                              price=p,
                              plimit=p + self.p.limitoffset,
                              valid=self.p.valid,
                              trailamount=self.p.trailamount,
                              trailpercent=self.p.trailpercent)
    
            elif self.position.size > 0 and not self.p.donotsell:
                if self.order is None:
                    self.order = self.sell(size=self.p.stake // 2,
                                           exectype=bt.Order.Market,
                                           price=self.data0.close[0])
    
            elif self.order is not None and self.p.cancel:
                if self.datastatus > self.p.cancel:
                    self.cancel(self.order)
    
            if self.datastatus:
                self.datastatus += 1
    
        def start(self):
            if self.data0.contractdetails is not None:
                print('Timezone from ContractDetails: {}'.format(
                      self.data0.contractdetails.m_timeZoneId))
    
            header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume',
                      'OpenInterest', 'SMA','StochK','StochD','MACD','MACDSig',
                      'PSAR']
            print(', '.join(header))
    
            self.done = False
    
    
    def runstrategy():
        args = parse_args()
    
        # Create a cerebro
        cerebro = bt.Cerebro()
    
        storekwargs = dict(
            host=args.host, port=args.port,
            clientId=args.clientId, timeoffset=not args.no_timeoffset,
            reconnect=args.reconnect, timeout=args.timeout,
            notifyall=args.notifyall, _debug=args.debug
        )
    
        if args.usestore:
            ibstore = bt.stores.IBStore(**storekwargs)
    
        if args.broker:
            if args.usestore:
                broker = ibstore.getbroker()
            else:
                broker = bt.brokers.IBBroker(**storekwargs)
    
            cerebro.setbroker(broker)
    
        timeframe = bt.TimeFrame.TFrame(args.timeframe)
        # Manage data1 parameters
        tf1 = args.timeframe1
        tf1 = bt.TimeFrame.TFrame(tf1) if tf1 is not None else timeframe
        cp1 = args.compression1
        cp1 = cp1 if cp1 is not None else args.compression
    
        if args.resample or args.replay:
            datatf = datatf1 = bt.TimeFrame.Ticks
            datacomp = datacomp1 = 1
        else:
            datatf = timeframe
            datacomp = args.compression
            datatf1 = tf1
            datacomp1 = cp1
    
        fromdate = None
        if args.fromdate:
            dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.fromdate))
            fromdate = datetime.datetime.strptime(args.fromdate, dtformat)
    
        todate = None
        if args.todate:
            dtformat = '%Y-%m-%d' + ('T%H:%M:%S' * ('T' in args.todate))
            todate = datetime.datetime.strptime(args.todate, dtformat)
    
        IBDataFactory = ibstore.getdata if args.usestore else bt.feeds.IBData
    
        datakwargs = dict(
            timeframe=datatf, compression=datacomp,
            historical=args.historical, fromdate=fromdate, todate=todate,
            rtbar=args.rtbar,
            qcheck=args.qcheck,
            what=args.what,
            backfill_start=not args.no_backfill_start,
            backfill=not args.no_backfill,
            latethrough=args.latethrough,
            tz=args.timezone
        )
    
        if not args.usestore and not args.broker:   # neither store nor broker
            datakwargs.update(storekwargs)  # pass the store args over the data
    
        data0 = IBDataFactory(dataname=args.data0, **datakwargs)
    
        data1 = None
        if args.data1 is not None:
            if args.data1 != args.data0:
                datakwargs['timeframe'] = datatf1
                datakwargs['compression'] = datacomp1
                data1 = IBDataFactory(dataname=args.data1, **datakwargs)
            else:
                data1 = data0
    
        rekwargs = dict(
            timeframe=timeframe, compression=args.compression,
            bar2edge=not args.no_bar2edge,
            adjbartime=not args.no_adjbartime,
            rightedge=not args.no_rightedge,
            takelate=not args.no_takelate,
        )
    
        if args.replay:
            cerebro.replaydata(data0, **rekwargs)
    
            if data1 is not None:
                rekwargs['timeframe'] = tf1
                rekwargs['compression'] = cp1
                cerebro.replaydata(data1, **rekwargs)
    
        elif args.resample:
            cerebro.resampledata(data0, **rekwargs)
    
            if data1 is not None:
                rekwargs['timeframe'] = tf1
                rekwargs['compression'] = cp1
                cerebro.resampledata(data1, **rekwargs)
    
        else:
            cerebro.adddata(data0)
            if data1 is not None:
                cerebro.adddata(data1)
    
        if args.valid is None:
            valid = None
        else:
            valid = datetime.timedelta(seconds=args.valid)
        # Add the strategy
        cerebro.addstrategy(TestStrategy,
                            smaperiod=args.smaperiod,
                            trade=args.trade,
                            exectype=bt.Order.ExecType(args.exectype),
                            stake=args.stake,
                            stopafter=args.stopafter,
                            valid=valid,
                            cancel=args.cancel,
                            donotsell=args.donotsell,
                            stoptrail=args.stoptrail,
                            stoptraillimit=args.traillimit,
                            trailamount=args.trailamount,
                            trailpercent=args.trailpercent,
                            limitoffset=args.limitoffset,
                            oca=args.oca,
                            bracket=args.bracket)
    
        if args.write:
            now = datetime.datetime.today()
            scriptName = os.path.basename(__file__).split(".")[0]
            logFileName = "logs/" + now.strftime("%Y_%m_%d_%H%M%S_") + \
                          scriptName + "_ibtest" + "_log.csv"
            cerebro.addwriter(bt.WriterFile, csv=True, out=logFileName, rounding=2)
    
        # Live data ... avoid long data accumulation by switching to "exactbars"
        cerebro.run(exactbars=args.exactbars)
    
        if args.plot and args.exactbars < 1:  # plot if possible
            cerebro.plot()
    
    
    def parse_args():
        parser = argparse.ArgumentParser(
            formatter_class=argparse.ArgumentDefaultsHelpFormatter,
            description='Test Interactive Brokers integration')
    
        parser.add_argument('--exactbars', default=1, type=int,
                            required=False, action='store',
                            help='exactbars level, use 0/-1/-2 to enable plotting')
    
        parser.add_argument('--plot',
                            required=False, action='store_true',
                            help='Plot if possible')
    
        parser.add_argument('--stopafter', default=0, type=int,
                            required=False, action='store',
                            help='Stop after x lines of LIVE data')
    
        parser.add_argument('--usestore', default=False,
                            required=False, action='store_true',
                            help='Use the store pattern')
    
        parser.add_argument('--notifyall',
                            required=False, action='store_true',
                            help='Notify all messages to strategy as store notifs')
    
        parser.add_argument('--debug', default=False,
                            required=False, action='store_true',
                            help='Display all info received form IB')
    
        parser.add_argument('--host', default='127.0.0.1',
                            required=False, action='store',
                            help='Host for the Interactive Brokers TWS Connection')
    
        parser.add_argument('--qcheck', default=2.0, type=float,
                            required=False, action='store',
                            help=('Timeout for periodic '
                                  'notification/resampling/replaying check'))
    
        parser.add_argument('--port', default=4002, type=int,
                            required=False, action='store',
                            help='Port for the Interactive Brokers TWS Connection')
    
        parser.add_argument('--clientId', default='466824', type=int,
                            required=False, action='store',
                            help='Client Id to connect to TWS (default: random)')
    
        parser.add_argument('--no-timeoffset',
                            required=False, action='store_true',
                            help=('Do not Use TWS/System time offset for non '
                                  'timestamped prices and to align resampling'))
    
        parser.add_argument('--reconnect', default=3, type=int,
                            required=False, action='store',
                            help='Number of recconnection attempts to TWS')
    
        parser.add_argument('--timeout', default=3.0, type=float,
                            required=False, action='store',
                            help='Timeout between reconnection attempts to TWS')
    
        parser.add_argument('--data0', default='BANKNIFTY-IND-NSE-INR',
                            required=False, action='store',
                            help='data 0 into the system')
    
        parser.add_argument('--data1', default=None,
                            required=False, action='store',
                            help='data 1 into the system')
    
        parser.add_argument('--timezone', default=None,
                            required=False, action='store',
                            help='timezone to get time output into (pytz names)')
    
        parser.add_argument('--what', default=None,
                            required=False, action='store',
                            help='specific price type for historical requests')
    
        parser.add_argument('--no-backfill_start',
                            required=False, action='store_true',
                            help='Disable backfilling at the start')
    
        parser.add_argument('--latethrough',
                            required=False, action='store_true',
                            help=('if resampling replaying, adjusting time '
                                  'and disabling time offset, let late samples '
                                  'through'))
    
        parser.add_argument('--no-backfill',
                            required=False, action='store_true',
                            help='Disable backfilling after a disconnection')
    
        parser.add_argument('--rtbar', default=True,
                            required=False, action='store_true',
                            help='Use 5 seconds real time bar updates if possible')
    
        parser.add_argument('--historical', default=True,
                            required=False, action='store_true',
                            help='do only historical download')
    
        parser.add_argument('--fromdate', default="2020-09-16T09:15:00",
                            required=False, action='store',
                            help=('Starting date for historical download '
                                  'with format: YYYY-MM-DD[THH:MM:SS]'))
    
        parser.add_argument('--todate', default="2020-09-17T15:30:00",
                            required=False, action='store',
                            help=('Ending date for historical download '
                                  'with format: YYYY-MM-DD[THH:MM:SS]'))
    
        parser.add_argument('--smaperiod', default=10, type=int,
                            required=False, action='store',
                            help='Period to apply to the Simple Moving Average')
    
        pgroup = parser.add_mutually_exclusive_group(required=False)
    
        pgroup.add_argument('--replay',
                            required=False, action='store_true',
                            help='replay to chosen timeframe')
    
        pgroup.add_argument('--resample',  default=True,
                            required=False, action='store_true',
                            help='resample to chosen timeframe')
    
        parser.add_argument('--timeframe', default='Minutes',
                            choices=bt.TimeFrame.Names,
                            required=False, action='store',
                            help='TimeFrame for Resample/Replay')
    
        parser.add_argument('--compression', default=1, type=int,
                            required=False, action='store',
                            help='Compression for Resample/Replay')
    
        parser.add_argument('--timeframe1', default=None,
                            choices=bt.TimeFrame.Names,
                            required=False, action='store',
                            help='TimeFrame for Resample/Replay - Data1')
    
        parser.add_argument('--compression1', default=None, type=int,
                            required=False, action='store',
                            help='Compression for Resample/Replay - Data1')
    
        parser.add_argument('--no-takelate',
                            required=False, action='store_true',
                            help=('resample/replay, do not accept late samples '
                                  'in new bar if the data source let them through '
                                  '(latethrough)'))
    
        parser.add_argument('--no-bar2edge',
                            required=False, action='store_true',
                            help='no bar2edge for resample/replay')
    
        parser.add_argument('--no-adjbartime',
                            required=False, action='store_true',
                            help='no adjbartime for resample/replay')
    
        parser.add_argument('--no-rightedge',
                            required=False, action='store_true',
                            help='no rightedge for resample/replay')
    
        parser.add_argument('--broker', default=False,
                            required=False, action='store_true',
                            help='Use IB as broker')
    
        parser.add_argument('--trade', default= False,
                            required=False, action='store_true',
                            help='Do Sample Buy/Sell operations')
    
        parser.add_argument('--donotsell',
                            required=False, action='store_true',
                            help='Do not sell after a buy')
    
        parser.add_argument('--exectype', default=bt.Order.ExecTypes[0],
                            choices=bt.Order.ExecTypes,
                            required=False, action='store',
                            help='Execution to Use when opening position')
    
        parser.add_argument('--stake', default=10, type=int,
                            required=False, action='store',
                            help='Stake to use in buy operations')
    
        parser.add_argument('--valid', default=None, type=int,
                            required=False, action='store',
                            help='Seconds to keep the order alive (0 means DAY)')
    
        pgroup = parser.add_mutually_exclusive_group(required=False)
        pgroup.add_argument('--stoptrail',
                            required=False, action='store_true',
                            help='Issue a stoptraillimit after buy( do not sell')
    
        pgroup.add_argument('--traillimit',
                            required=False, action='store_true',
                            help='Issue a stoptrail after buying (do not sell')
    
        pgroup.add_argument('--oca',
                            required=False, action='store_true',
                            help='Test oca by putting 2 orders in a group')
    
        pgroup.add_argument('--bracket',
                            required=False, action='store_true',
                            help='Test bracket orders by issuing high/low sides')
    
        pgroup = parser.add_mutually_exclusive_group(required=False)
        pgroup.add_argument('--trailamount', default=None, type=float,
                            required=False, action='store',
                            help='trailamount for StopTrail order')
    
        pgroup.add_argument('--trailpercent', default=None, type=float,
                            required=False, action='store',
                            help='trailpercent for StopTrail order')
    
        parser.add_argument('--limitoffset', default=None, type=float,
                            required=False, action='store',
                            help='limitoffset for StopTrailLimit orders')
    
        parser.add_argument('--cancel', default=0, type=int,
                            required=False, action='store',
                            help=('Cancel a buy order after n bars in operation,'
                                  ' to be combined with orders like Limit'))
    
        parser.add_argument('--write', default=True,
                            required=False, action='store',
                            help='Enable write observer data to file')
    
        return parser.parse_args()
    
    
    if __name__ == '__main__':
        runstrategy()
    


  • It seems to me that you encountered the issue discussed here several times even in 2017. Try to search the forum for resampling with live data. As I remember bt uses one approach when do resampling, and ib uses another approach for already formed bars. Honestly I didn't follow that discussions, and don't know if this was resolved or not.

    @vladisld may help if he has time.



  • @vladisld, Can you please help me with this issue. This is becoming a deal breaker as Indicator values are key to any automated trading systems. Do you use IB data or some other data source with backtrader? Did you notice any discrepancies with indicator values?


Log in to reply
 

});