An additional data source can be passed to do an initial layer of backfilling. Once the data source is depleted and if requested, backfilling from IB will take place. This is ideally meant to backfill from already stored sources like a file on disk, but not limited to.
Also IB Gateway version 978 seems stable and reconnects seemingly better than the older versions like 972. It also has a "log in once a week" feature, that works. If you're on foreign IB site like Canada's 972 is all it offers. Download 978 (the most recent version) from the US site.
With IB LIVE data, when using resample and replay e.g. (data, timeframe=Minutes, compression=5), if there is really no market event triggered within the 5 minutes e.g. from 10:00am to 10:05am, no bar will be generated at 10:05am as next() has not been called due to lack of data trigger. Is there available option or standard mechanism that can force generation of the 10:05am bar or it has to be implemented locally? I thought I have the qcheck=0.5 default to force this but seems it has no effect.
qcheck does only indicate how often the feed will wake up to see if it can deliver. If there is nothing, nothing can be delivered and nothing will be delivered. The data feed (or the resampler) cannot fabricate data out of thin air.
With IB LIVE data, when using replay(data, timeframe=Minutes, compression=5), next() will be called for each tick received within the 5 mins compression period with the same len(self.data0) value e.g. len(self.data0)=299 between 10:00am to 10:05am. Is there a standard mechanism in next() that I can dintinguish the intra-period ticks and the end of period tick? Or do I need to check the event time e.g. look for event time at e.g. 10:05:00.000000
You can only know that the resampling of the specific bar is over when the len changes. The replay mechanism cannot know when the replay is going to be over. It will be over either because an incoming timestamp goes over the next boundary or because the real-time clock has gone over the boundary (plus more or less the value of qcheck)
With IB LIVE data, when using resample(data, timeframe=Minutes, compression=1), sometime next() will be called multiple times at start of a minute e.g. 10:00am but with exact same values (open, high, low, close and indicators). I have checked the time and sales in IB and there is no trades happened from 09:59:59am and 10:00:01am so wonder what may trigger the duplicate next() call
If the assumption is that this could be an error, one would need a sample and data.
But this is most likely due to a timestamp coming in which belongs to the previous minute, but cannot be delivered with the previous minute because that one is out. In realtime, that timestamp is delivered with the next minute. Looking at historical data gives a different view of course