BT-IBAPI
-
So, I've been trying to rework ibstore/ibdata/ibbroker to utilize https://github.com/quantrocket-llc/ibpythonic and the official IB TWS API, all went good up to a point - as soon as I request a larger download (2 years of 1-day data), the download stops at the end of the first year and doesn't continue.
I've tracked this to a call towards historicalData() (called from ibpythonic->receiver.py->dispatchMethod()).
On IBPy:
<historicalData reqId=16777221, date=20181231, open=249.8, high=250.15, low=247.53, close=250.13, volume=1630, count=1375, WAP=248.945, hasGaps=False>
<historicalData reqId=16777221, date=finished-20180101 00:00:00-20190101 00:00:00, open=-1, high=-1, low=-1, close=-1, volume=-1, count=-1, WAP=-1, hasGaps=False>"date=finished-" appears at the end of each period, however, this is not the case with IBAPI, so the condition
if msg_date.startswith('finished-'): is not met
I've tried to remedy this by creating method:
@iswrapper @ibregister def historicalDataEnd(self, msg): # <historicalDataEnd reqId=16777218, start=20180101 00:00:00, end=20190101 00:00:00> ## For multi-tiered downloads we'd need to rebind the queue to a new ## tickerId (in case tickerIds are not reusable) and instead of putting ## None, issue a new reqHistData with the new data and move formward tickerId = msg.reqId q = self.qs[tickerId] self.histfmt.pop(tickerId, None) self.histsend.pop(tickerId, None) self.histtz.pop(tickerId, None) kargs = self.histexreq.pop(tickerId, None) if kargs is not None: self.reqHistoricalDataEx(tickerId=tickerId, **kargs) return if self.validQueue(q): self.cancelQueue(q, False)This makes the history download finish the whole period, however, as soon as we hit the end and call cancelQueue - the data doesn't resume "live" state, just sits the way it is.
Does anyone have any ideas, or want to help realize this? I can place code in a separate bt2 fork, or just upload everything here.
Let me know.
-
@vladisld I'm pretty sure you can help with this :)
-
So, I've made some progress, historical downloads now resume normally, I had to pass the historicalDataEnd() message to the Queue and check the message type. Now I'm onto ibbroker and orders, hopefully all will go well. Updates to follow.
@iswrapper @ibregister def historicalDataEnd(self, msg): ## For multi-tiered downloads we'd need to rebind the queue to a new ## tickerId (in case tickerIds are not reusable) and instead of putting ## None, issue a new reqHistData with the new data and move formward tickerId = msg.reqId q = self.qs[tickerId] self.histfmt.pop(tickerId, None) self.histsend.pop(tickerId, None) self.histtz.pop(tickerId, None) kargs = self.histexreq.pop(tickerId, None) if kargs is not None: self.reqHistoricalDataEx(tickerId=tickerId, **kargs) return else: msg.start = None msg.end = None # <historicalDataEnd reqId=16777220, start=None, end=None> # Put this message on the queue, check it's type @ ibdata->_load() -> elif self._state == self._ST_HISTORBACK: q.put(msg) if self.validQueue(q): self.cancelQueue(q, False) @ibregister def historicalData(self, msg): '''Receives the events of a historical data request''' # For multi-tiered downloads we'd need to rebind the queue to a new # tickerId (in case tickerIds are not reusable) and instead of putting # None, issue a new reqHistData with the new data and move formward tickerId = msg.reqId q = self.qs[tickerId] msg_date = msg.bar.date # 'BarData' object dtstr = msg.bar.date if self.histfmt[tickerId]: sessionend = self.histsend[tickerId] dt = datetime.strptime(dtstr, '%Y%m%d') dteos = datetime.combine(dt, sessionend) tz = self.histtz[tickerId] if tz: dteostz = tz.localize(dteos) dteosutc = dteostz.astimezone(UTC).replace(tzinfo=None) # When requesting for example daily bars, the current day # will be returned with the already happened data. If the # session end were added, the new ticks wouldn't make it # through, because they happen before the end of time else: dteosutc = dteos if dteosutc <= datetime.utcnow(): dt = dteosutc msg.bar.date = dt # msg.date -> msg.bar.date else: msg.bar.date = datetime.utcfromtimestamp(long(dtstr)) q.put(msg) # . . . . . . . . . Functions in ibdata->_load() before this one: elif self._state == self._ST_HISTORBACK: msg = self.qhist.get() if msg is None: # Conn broken during historical/backfilling #print("_load _ST_HISTORBACK msg is None; DISCONNECTED") # Situation not managed. Simply bail out self._subcription_valid = False self.put_notification(status=self.DISCONNECTED) return False # error management cancelled the queue # . . . . . . . . . Functions in between msgType = msg.__class__.__name__ if msgType == "HistoricalDataEnd": #print("Historical data download end, continuing...") pass else: if msg.bar.date is not None: # msg.date -> msg.bar.date #print("_load date is not None; _load_rtbar #2, hist=True: {}".format(msg)) if self._load_rtbar(msg, hist=True): #print("_load date _load_rtbar() returned True; True") return True # loading worked # the date is from overlapping historical request #print("_load date _load_rtbar() returned False, continuing") continue # . . . . . . . . . Functions after -
Orders seem to work fine, moving on to contract request rework by using separate fields for sectype, symbol, exchange, currency, local_symbol, expiry_date, multiplier, etc, instead of a very weird 'name' conversion.
After this is done, I'll look into news feeds & market signals requests, the idea is to have a separate function for each (like next(): market_signals() && news_feeds() and possibly acting on them separately)
If someone's interested in any of this - let me know.
-
Seriously, no interest at all?
-
@chewbacca do you have any github repository with some code to try it out?
-
@chewbacca Personally I'm pretty interested, but there are quite a few BUTs.
IMHO the following issues need to be discussed:
-
What new features the new implementation will bring compared to the current IB store implementation?
-
Why IBpythonic and not other alternative like: native TWS Python API or ib_insync library?
-
AFAIU the IBpythonic library is dependent on TWS Python API which supports only Python3. Although the python 2 is deprecated now, it is still officially supported in BT. If you development presents a replacement for the current implementation - this brings the question of future python support in BT itself.
-
One of the main goals of BT was to provide a framework with as little external dependencies as possible. My hope was that one day we may get rid of IBpy dependency as well once the native TWS Python API was available. Right now you are trading IBpy for IBpythonic + TWS Python API that needs to be installed - effectively increasing the external dependencies.
-
Testing: how are you going to prove your component will have a better quality compared with the current IB store implementation.?
IMHO, the main problem with the current IB store implementation is the lack of proper testing. There are no component level tests that provide a good coverage for all the scenarios involved in live trading nor for backtesting using IB. Without such tests any change to IB store source code is prune to regressions.
The only test for IB store that I'm aware of is a sample code provided with BT that needs to be run manually. The coverage of such testing is questionable at best.
One may say that given a long time in use the IB store component is slowly getting rid of its bugs - and this may even be partially true. However we are witnessing a different story - number of bugs in IB component is growing even after many years of use - last ones are just few weeks old.
I doubt that developing a replacement (or alternative) store using the similar code base without proper discussion of the approach, without clear test plan/infrastructure would make things better.
-
-
@vladisld I'm still learning python.
I used ibpythonic because it wouldn't require as much changes to the existing code comparing to a complete transition to
the official TWS api (which right now is beyond my abilities).Also, I decided to start doing this because.. well.. nobody else did. IbPy is quite old (2015 if memory serves), there's a lot of new functionality we could be using in BT by transitioning to the TWS api. The way I'm implementing this is by keeping the old ibbroker/ibstore/ibdata files and doing everything in a copy of these files. I'll try to rename all functions in the new files so they won't interfere with the old versions, meaning one could use the old or the new, their choice.
Let me do some cleanup and I'll place the 3 files here, you can review/change and place in your github repo (backtrader2).
-
@chewbacca said in BT-IBAPI:
there's a lot of new functionality we could be using in BT by transitioning to the TWS api
It would be great if we could discuss all those new features and see if they worth it - unless the above statement was more of a general assumption of future TWS API capabilities that we will miss.
Any plans for test coverage ? How? Are you going to develop IBpythonic or even TWS API (python or even NI) mock ( this actually could help in current implementation testing) ?
@chewbacca said in BT-IBAPI:
I decided to start doing this because.. well.. nobody else did
The above list of issues to discuss is pretty tough to address - may explain why nobody has tried :-)
-
Well, I tried, I'll provide everything here and its up to the community to use/develop it or not.
