next() called multiple times within timeframe and bid/ask availability



  • Hello, I am new to the platform and got some questions would like to seek your advice:

    1. I am using IB as broker and called cerebro.replaydata(...) with timeframe=Minutes and compression=1. When receiving DELAYED feed, next() was called with every one minute period, however, when switched to LIVE, the next() call is called many times within the minute which looks like tick from IB. Is it the expected bahavior that the next() is called multiple times during the one minute period during LIVE feed?

    2. Is it possible to get latest bid/ask in the next() call together with other line data e.g. open, high, low or need to be a separate data feed? I tried to search some previous discussion on this topic but still not sure if it is possible yet..

    3. I am trying to build a strategy that reads AUD.USD.CASH-IDEALPRO and AUD Futures. For AUD.USD.CASH feed, I would like to build indicators base on 5 and 15 minutes period and for AUD Futures I would like to get the latest bid/ask price at the point when placing Futures orders. Should I create 3 data feeds for above e.g.

    • cerebro.replaydata(AUD.USD.CASH, timeframe=Minutes, period=5, ...)
    • cerebro.replaydata(AUD.USD.CASH, timeframe=Minutes, period=15,...)
    • cerebro.replaydata(AUD.FUTUREs, timeframe=Ticks, period=1,...) plus how to get bid/ask

    Sorry if these are very primitive questions and any help will be much appreciated.


  • administrators

    @you said in next() called multiple times within timeframe and bid/ask availability:

    1. ... the next() call is called many times within the minute which looks like tick from IB. Is it the expected bahavior that the next() is called multiple times during the one minute period during LIVE feed?

    It is the expected behavior, but no because it is LIVE. It is because of this:

    cerebro.replaydata(...)
    

    The full docs:

    The quick summary:

    • Replay gives you every single (available) tick which is giving form to the desired timeframe/compression target price bar (for example a 5 minutes bar may be made up of 1000 ticks and replay will give you the same bar 1000 times adding up the ticks)

    • Resample gives you a single instance of the bar. The ticks are considered internally but not given to you. You only get the bar when it is ready.

    You obviously want cerebro.resampledata

    1. Is it possible to get latest bid/ask in the next() call together with other line data e.g. open, high, low or need to be a separate data feed? I tried to search some previous discussion on this topic but still not sure if it is possible yet.

    No it's not possible. The close price of the resampled bar in forex in Interactive Brokers is the latest bid price. Interactive Brokers provides no actual price (like for stocks or futures) for Forex and the de-facto standard is to use the bid price (you can modify that to get the ask price if wished)

    1. I am trying to build a strategy that reads AUD.USD.CASH-IDEALPRO and AUD Futures. For AUD.USD.CASH feed, I would like to build indicators base on 5 and 15 minutes period and for AUD Futures I would like to get the latest bid/ask price at the point when placing Futures orders. Should I create 3 data feeds for above e.g.

    Short answer: yes (but with resampledata). Longer answer: resampling/replaying to ticks/1 makes no sense. That's the lowest available timefame/compression combination.

    cerebro.replaydata(AUD.USD.CASH, timeframe=Minutes, period=5, ...)
    cerebro.replaydata(AUD.USD.CASH, timeframe=Minutes, period=15,...)

    Those are not 2 data feeds. Those are 2 resampled data feeds. Due to how backfilling is implemented that will not work. You should create individual data feeds and then resample them.



  • Thanks very much for your reply. May I further check the following:

    • With IB LIVE data, when using resample and replay e.g. (data, timeframe=Minutes, compression=5), if there is really no market event triggered within the 5 minutes e.g. from 10:00am to 10:05am, no bar will be generated at 10:05am as next() has not been called due to lack of data trigger. Is there available option or standard mechanism that can force generation of the 10:05am bar or it has to be implemented locally? I thought I have the qcheck=0.5 default to force this but seems it has no effect.

    • With IB LIVE data, when using replay(data, timeframe=Minutes, compression=5), next() will be called for each tick received within the 5 mins compression period with the same len(self.data0) value e.g. len(self.data0)=299 between 10:00am to 10:05am. Is there a standard mechanism in next() that I can dintinguish the intra-period ticks and the end of period tick? Or do I need to check the event time e.g. look for event time at e.g. 10:05:00.000000

    • With IB LIVE data, when using resample(data, timeframe=Minutes, compression=1), sometime next() will be called multiple times at start of a minute e.g. 10:00am but with exact same values (open, high, low, close and indicators). I have checked the time and sales in IB and there is no trades happened from 09:59:59am and 10:00:01am so wonder what may trigger the duplicate next() call.

    -- still learning to operate the platform and really appreciate your time and effort making this available to all


  • administrators

    @you said in next() called multiple times within timeframe and bid/ask availability:

    With IB LIVE data, when using resample and replay e.g. (data, timeframe=Minutes, compression=5), if there is really no market event triggered within the 5 minutes e.g. from 10:00am to 10:05am, no bar will be generated at 10:05am as next() has not been called due to lack of data trigger. Is there available option or standard mechanism that can force generation of the 10:05am bar or it has to be implemented locally? I thought I have the qcheck=0.5 default to force this but seems it has no effect.

    qcheck does only indicate how often the feed will wake up to see if it can deliver. If there is nothing, nothing can be delivered and nothing will be delivered. The data feed (or the resampler) cannot fabricate data out of thin air.

    With IB LIVE data, when using replay(data, timeframe=Minutes, compression=5), next() will be called for each tick received within the 5 mins compression period with the same len(self.data0) value e.g. len(self.data0)=299 between 10:00am to 10:05am. Is there a standard mechanism in next() that I can dintinguish the intra-period ticks and the end of period tick? Or do I need to check the event time e.g. look for event time at e.g. 10:05:00.000000

    You can only know that the resampling of the specific bar is over when the len changes. The replay mechanism cannot know when the replay is going to be over. It will be over either because an incoming timestamp goes over the next boundary or because the real-time clock has gone over the boundary (plus more or less the value of qcheck)

    With IB LIVE data, when using resample(data, timeframe=Minutes, compression=1), sometime next() will be called multiple times at start of a minute e.g. 10:00am but with exact same values (open, high, low, close and indicators). I have checked the time and sales in IB and there is no trades happened from 09:59:59am and 10:00:01am so wonder what may trigger the duplicate next() call

    If the assumption is that this could be an error, one would need a sample and data.

    But this is most likely due to a timestamp coming in which belongs to the previous minute, but cannot be delivered with the previous minute because that one is out. In realtime, that timestamp is delivered with the next minute. Looking at historical data gives a different view of course


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.