IB Live Data Feed: differences between resampled data and backfilled data

borodiliz

Hi all,

I'm doing some tests with backtrader + IB + live data feeds and I'm getting some differences between resampled data and backfilled data.

Initially my goal is simple: A log with exactly the same bars/data than the TWS chart shows.

Given the following code:

class Test(bt.Strategy):

    ######################
    #
    ######################
    def notify_data(self, data, status, * args, ** kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), * args)


    ######################
    #
    ######################
    def notify_store(self, msg, * args, ** kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    ######################
    #
    ######################
    def __init__(self):
        ''''''
    ######################
    #
    ######################
    def prenext(self):
        ''''''
        #self.next(frompre=True)



    ######################
    #
    ######################
    def log(self, txt):
        now = datetime.datetime.now()
        print('%s:  %s' % (
              now.strftime(dtfmt), txt))

    def next (self, frompre=False):

        self.log("Frompre: %d data0 %s open %f high %f low %f close %f vol %d" %
                 (frompre,
                 self.data0.datetime.datetime(0).strftime(dtfmt),
                 self.data0.open[0],
                 self.data0.high[0],
                 self.data0.low[0],
                 self.data0.close[0],
                 self.data0.volume[0]))

def parse_args():
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

    parser.add_argument('--notifyall',
                        required=False, action='store_true',
                        help='Notify all messages to strategy as store notifs')

    parser.add_argument('--debug',
                        required=False, action='store_true',
                        help='Display all info received form IB')


    parser.add_argument('--host', default='127.0.0.1',
                        required=False, action='store',
                        help='Host for the Interactive Brokers TWS Connection')

    parser.add_argument('--port', default=4002, type=int,
                        required=False, action='store',
                        help='Port for the Interactive Brokers TWS Connection')


    parser.add_argument('--clientId', default=None, type=int,
                        required=False, action='store',
                        help='Client Id to connect to TWS (default: random)')

    parser.add_argument('--no-timeoffset',
                        required=False, action='store_true',
                        help=('Do not Use TWS/System time offset for non '
                        'timestamped prices and to align resampling'))

    parser.add_argument('--reconnect', default=3, type=int,
                        required=False, action='store',
                        help='Number of recconnection attempts to TWS')

    parser.add_argument('--timeout', default=3.0, type=float,
                        required=False, action='store',
                        help='Timeout between reconnection attempts to TWS')

    parser.add_argument('--data0', default=None,
                        required=False, action='store',
                        help='data 0 into the system')

    return parser.parse_args()

if __name__ == '__main__':

    dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
    now = datetime.datetime.now();
    print ("Now is %s" % (now.strftime(dtfmt)))


    args = parse_args()

    # Create a cerebro entity
    cerebro = bt.Cerebro(exactbars=True, runonce=False, preload=False)


    ibstore = bt.stores.IBStore(
                                host=args.host, port=args.port,
                                clientId=args.clientId, timeoffset=not args.no_timeoffset,
                                reconnect=args.reconnect, timeout=args.timeout,
                                notifyall=args.notifyall, _debug=args.debug)


    cerebro.setbroker(ibstore.getbroker())



    data0 = ibstore.getdata(dataname=args.data0,
                            backfill_start=True,
                            timeframe=bt.TimeFrame.Minutes,
                            compression=1,
                            name='d0',
                            qcheck=20)

    # New resampler
    cerebro.resampledata(
                         data0,
                         timeframe=bt.TimeFrame.Minutes,
                         compression=1)

    # Add a strategy

    cerebro.addstrategy(Test)

    cerebro.run()

First run: backfilled data + resampled data:
resampled_vs_backfilled.py --port 4002 --clientId 15 --data0 GBL-201709-DTB

2017-06-16T09:48:19.251639: Frompre: 0 data0 2017-06-16T07:44:00.000000 open 164.41 high 164.41 low 164.34 close 164.35 vol 2712
2017-06-16T09:48:19.257446: Frompre: 0 data0 2017-06-16T07:45:00.000000 open 164.35 high 164.36 low 164.33 close 164.35 vol 1530
2017-06-16T09:48:19.259503: Frompre: 0 data0 2017-06-16T07:46:00.000000 open 164.34 high 164.35 low 164.32 close 164.34 vol 1330
2017-06-16T09:48:19.261535: Frompre: 0 data0 2017-06-16T07:47:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 409
2017-06-16T09:48:19.263397: Frompre: 0 data0 2017-06-16T07:48:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 165
***** DATA NOTIF: LIVE
2017-06-16T09:49:02.211599: Frompre: 0 data0 2017-06-16T07:49:00.000000 open 164.35 high 164.39 low 164.35 close 164.38 vol 798
2017-06-16T09:50:03.389798: Frompre: 0 data0 2017-06-16T07:50:00.000000 open 164.38 high 164.38 low 164.36 close 164.37 vol 538
2017-06-16T09:51:02.751163: Frompre: 0 data0 2017-06-16T07:51:00.000000 open 164.37 high 164.38 low 164.36 close 164.37 vol 780
2017-06-16T09:52:04.579715: Frompre: 0 data0 2017-06-16T07:52:00.000000 open 164.36 high 164.38 low 164.36 close 164.37 vol 747
2017-06-16T09:53:02.244969: Frompre: 0 data0 2017-06-16T07:53:00.000000 open 164.37 high 164.37 low 164.36 close 164.36 vol 246
2017-06-16T09:54:00.565664: Frompre: 0 data0 2017-06-16T07:54:00.000000 open 164.36 high 164.37 low 164.34 close 164.36 vol 896
2017-06-16T09:55:11.430551: Frompre: 0 data0 2017-06-16T07:55:00.000000 open 164.35 high 164.36 low 164.34 close 164.34 vol 414
2017-06-16T09:56:04.272170: Frompre: 0 data0 2017-06-16T07:56:00.000000 open 164.34 high 164.34 low 164.31 close 164.32 vol 2116
2017-06-16T09:57:01.123514: Frompre: 0 data0 2017-06-16T07:57:00.000000 open 164.33 high 164.35 low 164.33 close 164.35 vol 931

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T09:57:59.194470: Frompre: 0 data0 2017-06-16T07:44:00.000000 open 164.41 high 164.41 low 164.34 close 164.35 vol 2712
2017-06-16T09:57:59.196351: Frompre: 0 data0 2017-06-16T07:45:00.000000 open 164.35 high 164.36 low 164.33 close 164.35 vol 1530
2017-06-16T09:57:59.201949: Frompre: 0 data0 2017-06-16T07:46:00.000000 open 164.34 high 164.35 low 164.32 close 164.34 vol 1330
2017-06-16T09:57:59.204162: Frompre: 0 data0 2017-06-16T07:47:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 409
2017-06-16T09:57:59.206197: Frompre: 0 data0 2017-06-16T07:48:00.000000 open 164.34 high 164.39 low 164.34 close 164.38 vol 961
2017-06-16T09:57:59.207977: Frompre: 0 data0 2017-06-16T07:49:00.000000 open 164.38 high 164.38 low 164.36 close 164.37 vol 540
2017-06-16T09:57:59.210999: Frompre: 0 data0 2017-06-16T07:50:00.000000 open 164.37 high 164.38 low 164.36 close 164.37 vol 780
2017-06-16T09:57:59.215024: Frompre: 0 data0 2017-06-16T07:51:00.000000 open 164.37 high 164.38 low 164.35 close 164.37 vol 747
2017-06-16T09:57:59.216797: Frompre: 0 data0 2017-06-16T07:52:00.000000 open 164.37 high 164.37 low 164.36 close 164.36 vol 246
2017-06-16T09:57:59.218651: Frompre: 0 data0 2017-06-16T07:53:00.000000 open 164.36 high 164.37 low 164.34 close 164.36 vol 890
2017-06-16T09:57:59.220877: Frompre: 0 data0 2017-06-16T07:54:00.000000 open 164.36 high 164.36 low 164.34 close 164.34 vol 415
2017-06-16T09:57:59.223913: Frompre: 0 data0 2017-06-16T07:55:00.000000 open 164.34 high 164.34 low 164.31 close 164.33 vol 2114
2017-06-16T09:57:59.229085: Frompre: 0 data0 2017-06-16T07:56:00.000000 open 164.33 high 164.35 low 164.32 close 164.35 vol 934
2017-06-16T09:57:59.230990: Frompre: 0 data0 2017-06-16T07:57:00.000000 open 164.35 high 164.35 low 164.33 close 164.35 vol 427

Problems detected:

1.- Some bars are different. e.g: the bar with 211X volume

resampled data -> close 164.32 vol 2116
backfilled data -> close 164.33 vol 2114

2.- Different timestamps, e.g: the bar with volume 246:

resampled data time-> 07:53
backfilled data time -> 07:52

I've tried increasing the qcheck up-to 20 without success

Thanks in advance.

backtrader

@borodiliz said in IB Live Data Feed: differences between resampled data and backfilled data:

I've tried increasing the qcheck up-to 20 without success

You should decrease the qcheck parameter for more checks per second, which it's probably your intention (qcheck=0.5, the default, forces the system to wake-up every 0.5 seconds to do resampling clean-up for example, which in the absence of negotiation will allow the resampler to spit out bars by checking the real-time clock)

Without claiming that backtrader is perfect and its resampling is perfect and ... and ... the expectation that the backfilled data and the real-time resampled data will be equal is too much to expect. Add to the equation that Interactive Brokers dismisses itself as a data provider and has disclaimers about the data.

Reasons:

Ticks may be missed
Ticks may come in too late and be discarded (there is a parameter to control it and add them to the next resampled bar)
The resampler has to deliver in real-time using the real-time clock and cannot wait for ticks to show up if they are delayed. At 2017-06-16T07:57:00 it has to give you a bar, but it may not have all information from Interactive Brokers
The 4 times a second snapshots are not guaranteed to contain exact information.

Some of this problems can be alleviated by switching to Real-Time Bars for which Interactive Brokers guarantees consolidated information (check the rtbar parameter in Docs - Interactive Brokers. But this is not available for all products.

You can also face the following problem:

Backfill today and get bars with information X
Backfill tomorrow and get bars with information Y (slight changes to X), because some information has been consolidated.

@borodiliz said in IB Live Data Feed: differences between resampled data and backfilled data:

2.- Different timestamps, e.g: the bar with volume 246:

resampled data time-> 07:53
backfilled data time -> 07:52

This is because IB is moving the data pointer to the beginning of the minute rather than to the end. It's just a convention.

borodiliz

Thanks for your quick reply @backtrader

You can't claim backtrader is perfect, but I can claim backtrader is ... awesome!

I can understand the complexity using tickString, so let's go with RealTimeBars data:

Because I do not want bars as be delivered as much in real-time as possible and I perfer to "wait" a while I've set the following (qcheck=2 , timeoffset=True and rtbar=True):

    ibstore = bt.stores.IBStore(
                                host=args.host, port=args.port,
                                clientId=args.clientId, timeoffset=True,
                                reconnect=args.reconnect, timeout=args.timeout,
                                notifyall=args.notifyall, _debug=args.debug)

    cerebro.setbroker(ibstore.getbroker())

    data0 = ibstore.getdata(dataname=args.data0,
                            backfill_start=True,
                            timeframe=bt.TimeFrame.Minutes,
                            compression=1,
                            name='d0',
                            qcheck=2,
                            rtbar=True)

First run: check resampled data with debug mode enabled:

...........
<realtimeBar reqId=16777217, time=1497608880, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608885, open=164.35, high=164.35, low=164.35, close=164.35, volume=4, wap=164.35, count=3>
<realtimeBar reqId=16777217, time=1497608890, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608895, open=164.35, high=164.35, low=164.34, close=164.34, volume=9, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608900, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608905, open=164.34, high=164.34, low=164.34, close=164.34, volume=205, wap=164.34, count=2>
<currentTime time=1497608915>
<realtimeBar reqId=16777217, time=1497608910, open=164.34, high=164.34, low=164.34, close=164.34, volume=133, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497608915, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497608920, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608925, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608930, open=164.35, high=164.35, low=164.35, close=164.35, volume=79, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608935, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
2017-06-16T12:29:03.091121: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
<realtimeBar reqId=16777217, time=1497608940, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608945, open=164.35, high=164.35, low=164.35, close=164.35, volume=30, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608950, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608955, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608960, open=164.35, high=164.35, low=164.34, close=164.34, volume=13, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497608965, open=164.34, high=164.34, low=164.34, close=164.34, volume=15, wap=164.34, count=1>
<currentTime time=1497608975>
<realtimeBar reqId=16777217, time=1497608970, open=164.34, high=164.34, low=164.34, close=164.34, volume=13, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608975, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608980, open=164.35, high=164.35, low=164.35, close=164.35, volume=13, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608985, open=164.35, high=164.36, low=164.35, close=164.36, volume=48, wap=164.35, count=15>
<realtimeBar reqId=16777217, time=1497608990, open=164.35, high=164.35, low=164.35, close=164.35, volume=129, wap=164.35, count=19>
<realtimeBar reqId=16777217, time=1497608995, open=164.35, high=164.35, low=164.35, close=164.35, volume=26, wap=164.35, count=2>
2017-06-16T12:30:03.100416: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 289
<realtimeBar reqId=16777217, time=1497609000, open=164.34, high=164.34, low=164.34, close=164.34, volume=21, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609005, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609010, open=164.34, high=164.34, low=164.34, close=164.34, volume=104, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497609015, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609020, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609025, open=164.33, high=164.33, low=164.33, close=164.33, volume=1, wap=164.33, count=1>
<currentTime time=1497609035>
<realtimeBar reqId=16777217, time=1497609030, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609035, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609040, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609045, open=164.33, high=164.33, low=164.33, close=164.33, volume=286, wap=164.33, count=24>
<realtimeBar reqId=16777217, time=1497609050, open=164.33, high=164.33, low=164.33, close=164.33, volume=51, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609055, open=164.33, high=164.34, low=164.33, close=164.34, volume=243, wap=164.34, count=5>
2017-06-16T12:31:03.092756: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 687
<realtimeBar reqId=16777217, time=1497609060, open=164.33, high=164.33, low=164.33, close=164.33, volume=160, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497609065, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609070, open=164.33, high=164.33, low=164.33, close=164.33, volume=86, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609075, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609080, open=164.34, high=164.34, low=164.34, close=164.34, volume=33, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609085, open=164.33, high=164.33, low=164.33, close=164.33, volume=48, wap=164.33, count=6>
<currentTime time=1497609095>
<realtimeBar reqId=16777217, time=1497609090, open=164.33, high=164.33, low=164.33, close=164.33, volume=24, wap=164.33, count=5>
<realtimeBar reqId=16777217, time=1497609095, open=164.33, high=164.33, low=164.33, close=164.33, volume=43, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609100, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609105, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609110, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609115, open=164.33, high=164.33, low=164.33, close=164.33, volume=54, wap=164.33, count=3>
2017-06-16T12:32:03.096349: Frompre: 0 data0 2017-06-16T10:32:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 288

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T12:33:44.813167: Frompre: 0 data0 2017-06-16T10:28:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
2017-06-16T12:33:44.814699: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 290
2017-06-16T12:33:44.816210: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 708
2017-06-16T12:33:44.817950: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 448

My conclusion: First rtbar is not taken into account

After checking ibdata.py source code and debug data I think the problem arises when datetime for a rtbar is equal to the just emitted resampled bar. I seems those minor changes solves the problem.

Test again:

2017-06-16T13:27:02.523913: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
<realtimeBar reqId=16777217, time=1497612420, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612425, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612430, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612435, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612440, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612445, open=164.34, high=164.34, low=164.34, close=164.34, volume=18, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612450, open=164.33, high=164.34, low=164.33, close=164.34, volume=11, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497612455, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612460, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<currentTime time=1497612466>
<realtimeBar reqId=16777217, time=1497612465, open=164.34, high=164.34, low=164.34, close=164.34, volume=349, wap=164.34, count=42>
<realtimeBar reqId=16777217, time=1497612470, open=164.34, high=164.34, low=164.34, close=164.34, volume=152, wap=164.34, count=17>
<realtimeBar reqId=16777217, time=1497612475, open=164.34, high=164.34, low=164.34, close=164.34, volume=20, wap=164.34, count=5>
2017-06-16T13:28:02.429118: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
<realtimeBar reqId=16777217, time=1497612480, open=164.34, high=164.34, low=164.34, close=164.34, volume=61, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612485, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612490, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612495, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612500, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612505, open=164.34, high=164.34, low=164.34, close=164.34, volume=4, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497612510, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<realtimeBar reqId=16777217, time=1497612515, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612520, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<currentTime time=1497612526>
<realtimeBar reqId=16777217, time=1497612525, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612530, open=164.34, high=164.35, low=164.34, close=164.34, volume=256, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612535, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:29:02.331917: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
<realtimeBar reqId=16777217, time=1497612540, open=164.35, high=164.35, low=164.35, close=164.35, volume=2, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497612545, open=164.35, high=164.35, low=164.35, close=164.35, volume=352, wap=164.35, count=25>
<realtimeBar reqId=16777217, time=1497612550, open=164.35, high=164.35, low=164.35, close=164.35, volume=39, wap=164.35, count=6>
<realtimeBar reqId=16777217, time=1497612555, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612560, open=164.35, high=164.35, low=164.35, close=164.35, volume=10, wap=164.35, count=5>
<realtimeBar reqId=16777217, time=1497612565, open=164.35, high=164.35, low=164.35, close=164.35, volume=36, wap=164.35, count=4>
<realtimeBar reqId=16777217, time=1497612570, open=164.35, high=164.35, low=164.35, close=164.35, volume=143, wap=164.35, count=12>
<realtimeBar reqId=16777217, time=1497612575, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612580, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<currentTime time=1497612586>
<realtimeBar reqId=16777217, time=1497612585, open=164.35, high=164.35, low=164.35, close=164.35, volume=3, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497612590, open=164.34, high=164.34, low=164.34, close=164.34, volume=240, wap=164.34, count=6>
<realtimeBar reqId=16777217, time=1497612595, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:30:02.806358: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
<realtimeBar reqId=16777217, time=1497612600, open=164.35, high=164.35, low=164.34, close=164.34, volume=15, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612605, open=164.35, high=164.35, low=164.35, close=164.35, volume=182, wap=164.35, count=32>
<realtimeBar reqId=16777217, time=1497612610, open=164.35, high=164.35, low=164.35, close=164.35, volume=203, wap=164.35, count=17>
<realtimeBar reqId=16777217, time=1497612615, open=164.36, high=164.36, low=164.36, close=164.36, volume=8, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612620, open=164.36, high=164.37, low=164.36, close=164.37, volume=372, wap=164.36, count=23>
<realtimeBar reqId=16777217, time=1497612625, open=164.36, high=164.37, low=164.36, close=164.37, volume=347, wap=164.37, count=27>
<realtimeBar reqId=16777217, time=1497612630, open=164.37, high=164.37, low=164.37, close=164.37, volume=27, wap=164.37, count=6>
<realtimeBar reqId=16777217, time=1497612635, open=164.37, high=164.37, low=164.36, close=164.37, volume=88, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612640, open=164.36, high=164.36, low=164.36, close=164.36, volume=7, wap=164.36, count=1>
<currentTime time=1497612646>
<realtimeBar reqId=16777217, time=1497612645, open=164.36, high=164.36, low=164.36, close=164.36, volume=20, wap=164.36, count=1>
<realtimeBar reqId=16777217, time=1497612650, open=164.37, high=164.37, low=164.37, close=164.37, volume=2, wap=164.37, count=2>
<realtimeBar reqId=16777217, time=1497612655, open=164.36, high=164.36, low=164.36, close=164.36, volume=14, wap=164.36, count=1>
2017-06-16T13:31:03.093858: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T13:32:02.206769: Frompre: 0 data0 2017-06-16T11:26:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
2017-06-16T13:32:02.208600: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
2017-06-16T13:32:02.210083: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
2017-06-16T13:32:02.211503: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
2017-06-16T13:32:02.212988: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285
2017-06-16T13:32:02.219781: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.360000 high 164.370000 low 164.360000 close 164.370000 vol 285

Now in my opinion data seems to be correct.

Thanks for your time

backtrader

@borodiliz said in IB Live Data Feed: differences between resampled data and backfilled data:

My conclusion: First rtbar is not taken into account

May of course be. Will be looked into. Thank you for the analysis.