Just enumerate through the datas in a pythonic way by referencing d. if len(d) > self.lenofdata....
If you reference self.datas[1] you'll only ever access self.datas[1]. Each timeframe and datafeed will have its own data in datas. datas[0] datas[1] datas[2]... datas[n]. Also self.len has the potential for a keyword conflict. I suggest following backtraders example above.
Have you reviewed the multiple datas example? You might want to search that out.
@sullyai Couple of points that may or may not be useful: resampledata is done in-place so you are resampling four times in-place above. Ie. if your "day" has a half an hour bar during the real time hours, then you may get stuck resampling from 60 minutes to 1 day. Resample from 1 minute to daily instead.
Another thought is that your source data may be missing some bars due to it not being traded. For example, at the 4:00pm bar there may not be a "tick" yet so it may be missing and thus cannot be resampled. Especially with ibdata which isn't real tick data.
May I suggest as an additional option just zero-padding, symmetrization, or smooth padding out the short data feed to the length of the longest line? Then all the data will start at the same bar. At the end you can simply trim the output to match the actual individual line length if you wish.
Despite the IB documentation, my impression is that that the IB API waives the limit for most historical data request cases: "IB has a limitation, 1 historical data request per 10 seconds. According to this: https://groups.io/g/insync/topic/24001598?p=Created,,,20,2,0,0::,,,0,0,0,24001598 "Currently IB waives that limit for bar sizes larger than 30 seconds." However, it does seem to add to the 50/s total request limit pacing. And that was my problem at the time. Live + historical = pacing error.
So I ran some tests on historical data requests just now and pounded IB with data requests for historical 30 min bars, and 1 D bars and was not able to trip the 60 historical requests per ten minute period, nor the 1 data request every 10 seconds. So it's likely that the above statement that the limit is waived for bar sizes larger than 30 seconds is correct. If someone else wants to test 30 second or less bars that would be interesting.
As for the sliding window request limit your statement of 250/5 secs is almost certainly correct although there is some mention on the web that it's a 4 second window. Thus my _throt_delay code assumes four to be conservative.
Not sure if the throttling code could be put into ibpy without more understanding of its structure.
My strategy subscribes to about 37 symbols which backfill_from a pandascsv along with live trading. The pandascsv has it's own non-ibstore historical request mechanism. Thus the occasional throttling required.
Very well done! May I suggest throttling the requests as well to reduce the number of disconnection/reconnections required?
If you're not referencing backfilling with your pandas data cache routine, and your issue is simply that no data for one security is available and thus ALL the feeds don't start, then this thread might be helpful: https://community.backtrader.com/topic/476/trades-don-t-happen-until-way-late-no-they-don-t
Interesting, and I tried your solution unfortunately the combination of my historical data requests combined with live requests still jointly caused a pacing error. I've created two solutions to throttling the connection, one ugly but simple, and one elegant but complex.
The first, quite evil, solution is to simply wait one second every 50 transactions to ibstore/ibdata. Then call this routine _throt_delay before each ib request in ibstore/ibdata. Terrible, but on the bright side it becomes mathematically impossible to exceed IB's pacing restriction. And for most purposes this one second delay every once in a while is acceptable. The global is due to the the fact that both historical and live must add to the throttle count by calling _throt_delay before data request calls that come from a variety of routines in ibstore/ibdata.
reqcount = 0 #throttle limits
reqlmt = 50
...
@staticmethod
def _throt_delay():
global reqcount, reqlmt
if reqcount >= reqlmt - 1:
reqcount = 0
sleep(1)
reqcount += 1
The second, more elegant solution, is to throttle nothing unless it comes to the edge of maximum allowed transaction requests, and throttle the minimum at that point, and clear out the count as time passes. This routine also exploits the fact that IB only recalculates every four seconds or so, so up to 250 requests have no delay and beyond that only 1/50s per request:
reqcount = 0 #throttle limits
lastreqtime = time.time()
reqlmt = 50
...
@staticmethod
def _throt_delay():
global reqcount, lastreqtime
throttime = 1 / reqlmt # throttle delay in fraction of seconds
reqcount += 1 # add one throt period
reqcount -= int((time.time() - lastreqtime) / throttime) # Decrement count by time passed:
reqcount = max(reqcount, 0) # keep it above zero
lastreqtime = time.time()
# sleep it off if the pacing is too fast...
if reqcount >= 4 * reqlmt - 1:
sleep(throttime)
Now understandably Daniel R. would find the sleep abhorrent, although it's called minimally and with the absolute minimum period required, but this could be substituted with a thread blocker/barrier/thread primitive if appropriate. However the minimal and rare sleep avoids any concurrency issues with the threads just fine.
Hi Elliot,
First off, even though you haven't received many replies I want to point out how helpful your questions are. These are complex problems that I myself have encountered and hopefully you haven't given up despite the age of your posts. Your old questions are very interesting and useful so do come back.
WRT backfilling, I had similar problems. I would suggest that there are two independent issues in your post: 1) the lack of consistency of the backfill "sticking" on DATA DELAYED, and 2) the reason why the DATA DELAYED was not triggering the live feed when it does stick.
I am assuming that you're using your Pandas data cache routine that you've posted in another thread. Keep in mind that this all could be incorrect and is just supposition based on the below assumptions but also based on my testing.
My suppositions:
When the data is not live the DATA DELAYED is generated and the process does not transition to the live data but ONLY when your caching pandas routine is reading from a cached file
When the data is not live the DATA DELAYED is not generated and the process proceeds to live data *when your caching pandas routine is caching the data for the FIRST TIME and therefore creating the file.
When the data is live the DATA DELAYED does not "stick" even when reading from the cached file. Thus your timing start issues you mentioned.
Why?
When the data is not live and your process is reading from the cached file and it's sticking on DATA DELAYED this may well be because the use_rth (Backtrader attribute) and outsideRth (ib's version) do not match in real time hours terms. For example, with half hour bars on the last bar of the day the IB reqdata will return no 4:00pm bar with the EOD set to 4:00pm. Therefore the data to backfill_from in your data request is not quite available/compatible.
When your pandas routine caches for the first time the IB connection is still active so the backtrader routine then backfills from the existing connection with revised parameters/attributes for the connection. This is a guess, but a reasonable one.
When the data is live the ibstore routine backfills what it needs from the live connection using its backfilling requests since the other pandas data is (in its view) incomplete.
This could all be wrong but seems to bear out with some tests I've run. One thing you can do is to change your pandas csv backfill data request to outsideRth=0 and then have your ibstore request use_rth=1. You would have to check if that will force the backfilled data to rth only, but I suspect it would.
Good luck,
B.
My guess is that threading.Thread is a constructor only that doesn't start the thread until .start(). And start() looks to be triggered by a timer.
However when downloading many symbols (37) I also receive the 'Max rate of messages per second exceeded' error. A puzzle.