Struggling to consistently backfill
ElliotP last edited by backtrader
I have read the documentation and the forums exhaustively but I'm struggling to come up with an answer. When I start some of the strategies they permanently hang on "Data Delayed" after the initial successful connection to IB live servers. So it will receive the TWS Successfull connection and then print a "Data Delayed" which I have in the Notify Order function; but it won't progress onto the next function.
data0 = ibstore.getdata(reqRealTimeBars=True, dataname='BHP-STK-SMART-AUD', rtbar=False, timeframe=bt.TimeFrame.Seconds, compression=1, tradename ='BHP-CFD-SMART-AUD', useRTH=True, fromdate=datetime.datetime.now(pytz.timezone('Australia/Sydney')).date(), backfill=False) cerebro.resampledata(data0, timeframe=bt.TimeFrame.Seconds, compression=5)
The asx commences trading in rolling window periods by the alphabetical first letter of the stock's symbol, so some stocks will begin trading at 9.59am while some will not begin trading until 10.09am. With the larger securities, it seems to be much more consistent as I suspect there is a trade to hook in-to.
Some slightly smaller companies but still very liquid are often the ones to cause problems. So if the Stock were to open at 10.04am, I set the script to begin at 10.04am or 10.04.15am or 10.04.30am sometimes it will work, sometimes it won't.
I understand Data Delayed is backfilling in progress, but when multiple different stocks will just sit at the initial connection text with the status "Data Delayed" i feel it isn't stuck on backfilling. This also makes it hard to debug as I'm unable to see any of the "debug=True" text, but in the past this hasn't indicated any issues.
I have tried varying the qcheck value and my only other remaining ideas are to begin the start of the script later and hope to catch a trade or perhaps change the compression to a minute?
bigdavediode last edited by
First off, even though you haven't received many replies I want to point out how helpful your questions are. These are complex problems that I myself have encountered and hopefully you haven't given up despite the age of your posts. Your old questions are very interesting and useful so do come back.
WRT backfilling, I had similar problems. I would suggest that there are two independent issues in your post: 1) the lack of consistency of the backfill "sticking" on DATA DELAYED, and 2) the reason why the DATA DELAYED was not triggering the live feed when it does stick.
I am assuming that you're using your Pandas data cache routine that you've posted in another thread. Keep in mind that this all could be incorrect and is just supposition based on the below assumptions but also based on my testing.
When the data is not live the DATA DELAYED is generated and the process does not transition to the live data but ONLY when your caching pandas routine is reading from a cached file
When the data is not live the DATA DELAYED is not generated and the process proceeds to live data *when your caching pandas routine is caching the data for the FIRST TIME and therefore creating the file.
When the data is live the DATA DELAYED does not "stick" even when reading from the cached file. Thus your timing start issues you mentioned.
When the data is not live and your process is reading from the cached file and it's sticking on DATA DELAYED this may well be because the use_rth (Backtrader attribute) and outsideRth (ib's version) do not match in real time hours terms. For example, with half hour bars on the last bar of the day the IB reqdata will return no 4:00pm bar with the EOD set to 4:00pm. Therefore the data to backfill_from in your data request is not quite available/compatible.
When your pandas routine caches for the first time the IB connection is still active so the backtrader routine then backfills from the existing connection with revised parameters/attributes for the connection. This is a guess, but a reasonable one.
When the data is live the ibstore routine backfills what it needs from the live connection using its backfilling requests since the other pandas data is (in its view) incomplete.
This could all be wrong but seems to bear out with some tests I've run. One thing you can do is to change your pandas csv backfill data request to outsideRth=0 and then have your ibstore request use_rth=1. You would have to check if that will force the backfilled data to rth only, but I suspect it would.
bigdavediode last edited by
If you're not referencing backfilling with your pandas data cache routine, and your issue is simply that no data for one security is available and thus ALL the feeds don't start, then this thread might be helpful: https://community.backtrader.com/topic/476/trades-don-t-happen-until-way-late-no-they-don-t