For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Live Trading Data Synchronization Issues



  • Hello!

    Long time lurker of these boards, first time poster here...

    I've been using Backtrader successfully for backtesting cryptcurrency strategies for a few months now, and have begun to try and tweak the platform for live trading so as to seamlessly move from backtesting to live trading. Long story short, I've built a backend for market prices and supplementary data from scratch that communicates with backtrader through a custom store/feed/broker that underlying all of it uses WAMP for two-way communication. It was certainly helpful to leverage the Interactive Brokers and OANDA sample code to do so.

    I know that data sync issues between multiple timeframe feeds has been something that has changed substantially in recent versions, but I must say that despite the improvements I am still rather confused by certain aspects of the behavior of the platform. So as to try and stay focused about my specific question(s) here, I won't include any of the custom store code or anything for now, but can do so as needed. What I thought would be most useful would be a logging output of what is happening in the strategy itself, and I will try and notate where I am not sure what is going on (which is a few places)

    For context, I am subscribing to 3 data 'feeds' - 1 is the order book data which is sampled at 5 second intervals, 2 is the actual trade data from the exchanges (so 1 new item per new trade) which I will call 'ticks' below to differentiate from backtrader's concept of 'trades' and 3 is a minute bar feed which is just like it sounds of course. I also have an indicator being calculated on the minute bars with a period of 6, and that seems to be working fine, but it may be relevant later. I am adding them as such:

    data_orderbook = reactivestore.getorderbook(dataname='BTC-USD', exchange='GDAX')
    cerebro.adddata(data_orderbook, name='orderbook')
    data_ticks = reactivestore.getdata(dataname='BTC-USD', exchange='GDAX', backfill_start=False,
                                       timeframe=bt.TimeFrame.Ticks, compression=1)
    cerebro.adddata(data_ticks, name='ticks')
    data_min = reactivestore.getdata(dataname='BTC-USD', exchange='GDAX', backfill_start=True,
                                     timeframe=bt.TimeFrame.Minutes, compression=1)
    cerebro.adddata(data_min, name='BTC-USD-1M')
    

    The order book feed is a subclassing of DataBase to get items out beyond the usual OHLC - there was a blog post written about this ('Out of OHLC Land' or similar) which I used for inspiration. Anyways, it seems to work at a high level so I think that's OK.

    All of the feeds as I mentioned use WAMP/Autobahn and basically they all spin up a new Thread and just listen for certain topics to be broadcast, and then add prices to the queue - again, not unlike the IB or OANDA samples.

    So here is a log of what is happening (the "Got Order Book / Tick / Bar" messages are in the store themselves and represent when the 'raw' data itself came in)

    Subscribing to order book...
    Subscribing to ticks...
    Subscribing to candles...
    ***** STORE NOTIF: subscribed to order book topic
    ***** STORE NOTIF: subscribed to pricing topic
    ***** STORE NOTIF: subscribed to pricing topic
    Got order book: {'Timestamp': '2018-05-17T21:22:03.321Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.7335024, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.99': 0.6, '8199.28': 5.8, '8191.33': 2.70621365, '8191.34': 4.28630967, '8194.45': 0.0027}, 'Exchange': 'GDAX'}
    ***** DATA NOTIF: orderbook LIVE
    2018-05-17T21:22:03.320995 PRENEXT
    Length of data[0] (order books): 1
    Length of data[1] (ticks): 0
    Length of data[2] (minute bars): 0
    Got order book: {'Timestamp': '2018-05-17T21:22:04.458Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.7335024, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.99': 0.6, '8199.28': 5.8, '8191.33': 2.70621365, '8191.34': 4.28630967, '8194.45': 0.0027}, 'Exchange': 'GDAX'}
    2018-05-17T21:22:04.458004 PRENEXT
    Length of data[0] (order books): 2
    Length of data[1] (ticks): 0
    Length of data[2] (minute bars): 0
    ...Get Order Book data a bunch of times before any ticks arrive (not unexpected...)
    Got tick: {'Timestamp': '2018-05-17T21:22:06.677Z', 'Symbol': 'BTC-USD', 'Price': 8191.33, 'Exchange': 'GDAX'}
    ***** DATA NOTIF: ticks LIVE
    2018-05-17T21:22:06.444998 PRENEXT
    Length of data[0] (order books): 4
    Length of data[1] (ticks): 1
    Length of data[2] (minute bars): 0
    Got first tick...everything still seems good
    2018-05-17T21:22:10.346002 PRENEXT
    Length of data[0] (order books): 8
    Length of data[1] (ticks): 1
    Length of data[2] (minute bars): 0
    Got tick: {'Timestamp': '2018-05-17T21:22:11.468Z', 'Symbol': 'BTC-USD', 'Price': 8191.33, 'Exchange': 'GDAX'}
    Got tick: {'Timestamp': '2018-05-17T21:22:11.468Z', 'Symbol': 'BTC-USD', 'Price': 8191.33, 'Exchange': 'GDAX'}
    Got tick: {'Timestamp': '2018-05-17T21:22:11.468Z', 'Symbol': 'BTC-USD', 'Price': 8191.33, 'Exchange': 'GDAX'}
    2018-05-17T21:22:10.346002 PRENEXT
    Length of data[0] (order books): 8
    Length of data[1] (ticks): 2
    Length of data[2] (minute bars): 0
    BOOM, bunch of ticks all at once, but only one data point gets added. Is this because of the timestamp being so similar, and if so, what is the resolution beyond which all ticks are essentially 'the same'?
    ...continues...
    Got order book: {'Timestamp': '2018-05-17T21:22:17.145Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.71115292, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.28': 5.8, '8191.33': 1.72357238, '8191.34': 4.28630967, '8194.45': 0.0027, '8196.20000000': 3.65}, 'Exchange': 'GDAX'}
    2018-05-17T21:22:17.145000 PRENEXT
    Length of data[0] (order books): 15
    Length of data[1] (ticks): 2
    Length of data[2] (minute bars): 0
    ...continues, but starts to get weird...
    Got order book: {'Timestamp': '2018-05-17T21:22:57.487Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.22073863, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.28': 5.8, '8191.33': 2.2783538, '8191.34': 4.48630967, '8194.45': 0.0027, '8196.20000000': 3.65}, 'Exchange': 'GDAX'}
    2018-05-17T21:22:57.486996 PRENEXT
    Length of data[0] (order books): 55
    Length of data[1] (ticks): 5
    Length of data[2] (minute bars): 0
    Got order book: {'Timestamp': '2018-05-17T21:22:59.242Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.22073863, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.28': 5.8, '8191.33': 2.2783538, '8191.34': 4.48630967, '8194.45': 0.0027, '8196.20000000': 3.65}, 'Exchange': 'GDAX'}
    2018-05-17T21:22:59.241995 PRENEXT
    Length of data[0] (order books): 56
    Length of data[1] (ticks): 5
    Length of data[2] (minute bars): 0
    Got tick: {'Timestamp': '2018-05-17T21:23:00.33Z', 'Symbol': 'BTC-USD', 'Price': 8191.33, 'Exchange': 'GDAX'}
    Got bar: {'Volume': 0.0, 'Timestamp': '2018-05-17T21:22:00Z', 'High': 8191.33, 'Low': 8191.32, 'Close': 8191.32, 'Open': 8191.33}
    Got our first minute bar, and backfilling begins
    Trying to get historical candles for BTC-USD-1M
    Got order book: {'Timestamp': '2018-05-17T21:23:00.27Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.22073863, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.28': 5.8, '8191.33': 1.4766538, '8191.34': 4.48630967, '8194.45': 0.0027, '8196.20000000': 3.65}, 'Exchange': 'GDAX'}
    2018-05-17T21:22:59.241995 PRENEXT
    Length of data[0] (order books): 56
    Length of data[1] (ticks): 4
    Length of data[2] (minute bars): 1
    Notice how order book count did not change, even though another sample was received. ALSO, the tick data length actually went down ?!?
    ...
    2018-05-17T21:22:59.241995 PRENEXT
    Length of data[0] (order books): 56
    Length of data[1] (ticks): 4
    Length of data[2] (minute bars): 5
    Got Minute Bar - Date: 2018-05-17 16:27:00 Open: 8282.56 High: 8290.99 Low: 8282.56 Close: 8287.24
    Now have enough bars for indicator min period, so next() starts getting called...
    Next: 2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 16:28:00 Open: 8287.23 High: 8287.24 Low: 8287.23 Close: 8287.24
    Next: 2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 16:29:00 Open: 8287.24 High: 8290.77 Low: 8287.24 Close: 8290.77
    Next: 2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 18:07:00 Open: 8256.00 High: 8264.35 Low: 8255.99 Close: 8264.34
    Next: 2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 18:08:00 Open: 8264.35 High: 8264.35 Low: 8264.34 Close: 8264.34
    Got order book: {'Timestamp': '2018-05-17T21:23:01.47Z', 'Symbol': 'BTC-USD', 'Bids': {'8190': 0.54460617, '8191': 0.5, '8189': 0.97, '8189.75': 0.0225, '8190.22': 0.001, '8188.35': 0.0225, '8190.26': 0.13868193, '8191.32': 2.22073863, '8190.76': 0.003, '8190.01': 0.015}, 'Asks': {'8192': 0.5, '8199.27': 0.5, '8195.33': 0.006, '8191.95': 1.21993, '8199': 0.814, '8199.28': 5.8, '8191.33': 2.2176538, '8191.34': 4.48630967, '8194.45': 0.0027, '8196.20000000': 3.65}, 'Exchange': 'GDAX'}
    ...
    2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 21:20:00 Open: 8191.33 High: 8191.33 Low: 8191.32 Close: 8191.32
    2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 21:21:00 Open: 8191.33 High: 8191.33 Low: 8191.32 Close: 8191.32
    2018-05-17T21:22:59.241995 Bar - Date: 2018-05-17 21:22:00 Open: 8191.33 High: 8191.33 Low: 8191.32 Close: 8191.32
    ***** DATA NOTIF: BTC-USD-1M LIVE
    ** So now, all data is live, unfortunately things sort of degrade from here...**

    As a few example of a couple things which go wrong after this point,

    ** Notice the timestamp on the tick received below, it is received from the backend and q.put() in the store, but it is only after 5 later order book updates are passed into next() that the tick update is passed into next() **
    Got tick: {'Timestamp': '2018-05-17T22:51:31.361Z', 'Symbol': 'BTC-USD', 'Price': 8079.91, 'Exchange': 'GDAX'}
    Got order book: {'Timestamp': '2018-05-17T22:51:31.514Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 0.23458007, '8079.90000000': 10.0, '8074.99': 0.001, '8075': 0.1, '8076': 0.05, '8074.19': 0.001, '8076.59': 0.167, '8076.58': 6.5, '8074.14': 0.03069, '8077.31': 0.001}, 'Asks': {'8087': 15.2871807, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8087.08': 0.00117464, '8087.09': 0.00166911}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:25.642996 NEXT
    Length of data[0] (order books): 52
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:25.642996 next(): Order Book - Date: 2018-05-17 22:51:25.642996 Bid: 8086.99 (volume 0.00) Ask: 8087.00 (volume 22.85)
    Got order book: {'Timestamp': '2018-05-17T22:51:32.644Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 0.04, '8076.56000000': 0.1, '8079.90000000': 10.0, '8074.99': 0.001, '8075': 0.1, '8076': 0.05, '8074.19': 0.001, '8076.58': 6.5, '8074.14': 0.03069, '8077.31': 0.001}, 'Asks': {'8086.99000000': 0.9207, '8083.26000000': 1.0, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087': 15.2861607, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8079.92000000': 1.1585}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:26.628999 NEXT
    Length of data[0] (order books): 53
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:26.628999 next(): Order Book - Date: 2018-05-17 22:51:26.628999 Bid: 8086.99 (volume 0.00) Ask: 8087.00 (volume 23.23)
    Got order book: {'Timestamp': '2018-05-17T22:51:33.588Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 1.0, '8076.94000000': 0.1, '8079.90000000': 10.0, '8079.99000000': 3.7, '8075': 0.1, '8076': 0.05, '8074.19': 0.001, '8076.58': 6.5, '8074.99': 0.001, '8077.31': 0.001}, 'Asks': {'8087': 15.2874507, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8087.08': 0.00117464, '8087.09': 0.00166911}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:27.599000 NEXT
    Length of data[0] (order books): 54
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:27.599000 next(): Order Book - Date: 2018-05-17 22:51:27.599000 Bid: 8086.99 (volume 0.00) Ask: 8087.00 (volume 21.99)
    Got order book: {'Timestamp': '2018-05-17T22:51:34.482Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 1.0, '8079.90000000': 10.0, '8079.99000000': 3.7, '8075': 0.1, '8076': 0.05, '8080.44000000': 1.0, '8074.19': 0.001, '8076.58': 6.5, '8074.99': 0.001, '8077.31': 0.001}, 'Asks': {'8087': 15.2861607, '8087.09000000': 0.00154542, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8087.08': 0.00117464}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:28.604004 NEXT
    Length of data[0] (order books): 55
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:28.604004 next(): Order Book - Date: 2018-05-17 22:51:28.604004 Bid: 8086.99 (volume 0.04) Ask: 8087.00 (volume 21.45)
    Got order book: {'Timestamp': '2018-05-17T22:51:35.634Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 1.0, '8079.90000000': 10.0, '8080.59000000': 0.04239948, '8076.58': 6.5, '8075': 0.1, '8076': 0.05, '8080.44000000': 1.0, '8079.99000000': 5.7, '8074.99': 0.001, '8077.31': 0.001}, 'Asks': {'8087': 3.3529107, '8084.45000000': 0.9107, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8087.08': 0.00117464}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:29.632996 NEXT
    Length of data[0] (order books): 56
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:29.632996 next(): Order Book - Date: 2018-05-17 22:51:29.632996 Bid: 8086.99 (volume 0.00) Ask: 8087.00 (volume 21.99)
    Got order book: {'Timestamp': '2018-05-17T22:51:36.568Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 1.0, '8079.90000000': 10.0, '8079.99000000': 5.7, '8075': 0.1, '8076': 0.05, '8080.61000000': 0.004, '8080.44000000': 1.0, '8076.81000000': 0.1, '8076.58': 6.5, '8077.31': 0.001}, 'Asks': {'8087': 3.4029107, '8087.09000000': 0.00154542, '8087.04': 0.00129825, '8087.05': 0.00117438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459, '8087.08': 0.00117464}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:30.611004 NEXT
    Length of data[0] (order books): 57
    Length of data[1] (ticks): 8
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:30.611004 next(): Order Book - Date: 2018-05-17 22:51:30.611004 Bid: 8086.99 (volume 0.00) Ask: 8087.00 (volume 19.49)
    Got order book: {'Timestamp': '2018-05-17T22:51:37.572Z', 'Symbol': 'BTC-USD', 'Bids': {'8079.91000000': 1.00746, '8079.90000000': 10.0, '8079.99000000': 5.7, '8075': 0.1, '8080.64000000': 0.005, '8080.05000000': 1.23606, '8076': 0.05, '8080.44000000': 1.0, '8076.58': 6.5, '8077.31': 0.001}, 'Asks': {'8084.91000000': 0.8757, '8087': 3.4015207, '8085.53000000': 0.01398, '8087.04': 0.00129825, '8087.05': 1.23677438, '8087.06': 0.00142182, '8087.07': 0.00154566, '8087.01': 0.00117467, '8087.02': 0.00216393, '8087.03': 0.00117459}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:30.611004 NEXT
    Length of data[0] (order books): 57
    Length of data[1] (ticks): 9
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:30.611004 next(): Tick - Date: 2018-05-17 22:51:31.361000 Price: 8086.99

    And then the biggest issue is below. If you look at the timestamps, it is received by the backend in real time, but the next next() call passes a very delayed value that was received a long time ago. Very quickly the seconds turn into minutes, and it seems to get progressively worse.

    Got order book: {'Timestamp': '2018-05-17T22:51:59.636Z', 'Symbol': 'BTC-USD', 'Bids': {'8075.84000000': 0.5, '8074.53000000': 0.003, '8074.99': 0.001, '8074.32000000': 0.03168, '8075': 0.03, '8076': 0.05, '8077.35000000': 0.51284, '8074.19': 0.001, '8074.30000000': 0.03171, '8077.31': 0.001}, 'Asks': {'8080.35000000': 1.23692, '8077.37000000': 5.996, '8077.38000000': 2.0, '8078.71000000': 11.8, '8079.94000000': 2.65191806, '8077.40000000': 1.0, '8080.00000000': 2.0, '8079.79000000': 1.0, '8079.92000000': 0.00129948, '8078.69000000': 0.01}, 'Exchange': 'GDAX'}
    2018-05-17T22:51:51.540995 NEXT
    Length of data[0] (order books): 78
    Length of data[1] (ticks): 10
    Length of data[2] (minute bars): 301
    2018-05-17T22:51:51.540995 next(): Order Book - Date: 2018-05-17 22:51:51.540995 Bid: 8079.95 (volume 0.51) Ask: 8080.55 (volume 6.00)

    I've perused the boards as much as I can here to try and come up with ideas for settings to change, etc but am pretty stuck here about how to even go about debugging this going forward. I don't know if this is just a matter of python/backtrader not being able to keep up with the rate at which the updates are received, but I somehow doubt that given it seems to do fine in the prenext() stage, and only struggles afterwards.

    And as a related question - clearly if I were to run this sort of live strategy for any amount of time the data structures holding the streams, with all of their respective histories, could become quite a memory hog (especially the order book data) - is there a way to limit the maximum data length that is stored for historical purposes and just purge older records, so as to limit this issue?

    Any advice would be much appreciated!



  • I can actually ask a simple question that ultimately might go a long way to solving many my issues here, which could all be related in the end. To simplify things I just printed out order book data both when it was received 'raw' in its own thread, and then when it was finally received by next(). Something like below happened:

    got order book timestamp: 2018-05-19T00:55:24.347Z
    next() order book timestamp: 2018-05-19 00:55:16.418000 <-- a delay of ~8 seconds, shortly after launching
    got order book timestamp: 2018-05-19T00:55:25.388Z
    next() order book timestamp: 2018-05-19 00:55:17.447002
    ...continuing
    got order book timestamp: 2018-05-19T00:55:32.432Z
    next() order book timestamp: 2018-05-19 00:55:24.346995
    got order book timestamp: 2018-05-19T00:55:33.376Z
    next() order book timestamp: 2018-05-19 00:55:25.387997
    ...here, I stop the pricing source so new new order book updates are delivered
    got order book timestamp: 2018-05-19T00:55:34.358Z
    next() order book timestamp: 2018-05-19 00:55:26.422995
    next() order book timestamp: 2018-05-19 00:55:27.306002
    next() order book timestamp: 2018-05-19 00:55:28.438997
    next() order book timestamp: 2018-05-19 00:55:29.346003
    next() order book timestamp: 2018-05-19 00:55:30.418004
    next() order book timestamp: 2018-05-19 00:55:31.420996
    next() order book timestamp: 2018-05-19 00:55:32.431995

    It is clear that next() is just not getting called/processed through fast enough to keep up with the data received. Still, I think this can't have anything to do with actual technical limitations of the hardware or anything, as this is hardly "high frequency" at all. I'm wondering if maybe I have to explicitly specify something in regards to the order book data timeframe - essentially the order books are received when 'ready' and not at any predetermined frequency, so not sure how to do this.

    Unfortunately playing around with 'resampledata' and 'replaydata' to see if they could help (not sure why they should, but you never know...) is out of the question since this is a custom data feed and I immediately get an error "AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'high'" - which is true - whenever I try and do so...

    Thoughts?

    Thanks!


  • administrators

    @someone29926 said in Live Trading Data Synchronization Issues:

    I know that data sync issues between multiple timeframe feeds has been something that has changed substantially in recent versions

    No there have been no changes for a very long time.

    @someone29926 said in Live Trading Data Synchronization Issues:

    Unfortunately playing around with 'resampledata' and 'replaydata' to see if they could help (not sure why they should, but you never know...) is out of the question since this is a custom data feed and I immediately get an error "AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'high'" - which is true - whenever I try and do so...
    Thoughts?

    Everything revolves about your custom data feed which nobody has seen. Synchronization with the provided data feeds work.



  • It did come down to the custom store code in the end...I had been using the 'qcheck' param from the IB store code and had its default set to 0.5seconds, I am not sure entirely how, but I guess somehow this was causing backpressure against the incoming feed. Once I set the qcheck to 0 everything ran fine.


  • administrators

    qcheck=0 won't wake up the data to enable accurate real-time resampling in case trading is thin. That's probably not an issue in most cases, because when ticks come in with enough frequency, the tick will itself wake up the feed.