Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    BT-IBAPI

    General Code/Help
    3
    10
    346
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      chewbacca last edited by

      So, I've been trying to rework ibstore/ibdata/ibbroker to utilize https://github.com/quantrocket-llc/ibpythonic and the official IB TWS API, all went good up to a point - as soon as I request a larger download (2 years of 1-day data), the download stops at the end of the first year and doesn't continue.

      I've tracked this to a call towards historicalData() (called from ibpythonic->receiver.py->dispatchMethod()).
      On IBPy:
      <historicalData reqId=16777221, date=20181231, open=249.8, high=250.15, low=247.53, close=250.13, volume=1630, count=1375, WAP=248.945, hasGaps=False>
      <historicalData reqId=16777221, date=finished-20180101 00:00:00-20190101 00:00:00, open=-1, high=-1, low=-1, close=-1, volume=-1, count=-1, WAP=-1, hasGaps=False>

      "date=finished-" appears at the end of each period, however, this is not the case with IBAPI, so the condition

      if msg_date.startswith('finished-'): is not met

      I've tried to remedy this by creating method:

      @iswrapper
      @ibregister
      def historicalDataEnd(self, msg):
          # <historicalDataEnd reqId=16777218, start=20180101  00:00:00, end=20190101  00:00:00>
          ## For multi-tiered downloads we'd need to rebind the queue to a new
          ## tickerId (in case tickerIds are not reusable) and instead of putting
          ## None, issue a new reqHistData with the new data and move formward
          tickerId = msg.reqId
          q = self.qs[tickerId]
          self.histfmt.pop(tickerId, None)
          self.histsend.pop(tickerId, None)
          self.histtz.pop(tickerId, None)
          kargs = self.histexreq.pop(tickerId, None)
          if kargs is not None:
              self.reqHistoricalDataEx(tickerId=tickerId, **kargs)
              return
      
          if self.validQueue(q):
              self.cancelQueue(q, False)
      

      This makes the history download finish the whole period, however, as soon as we hit the end and call cancelQueue - the data doesn't resume "live" state, just sits the way it is.

      Does anyone have any ideas, or want to help realize this? I can place code in a separate bt2 fork, or just upload everything here.

      Let me know.

      1 Reply Last reply Reply Quote 0
      • C
        chewbacca last edited by

        @vladisld I'm pretty sure you can help with this :)

        1 Reply Last reply Reply Quote 0
        • C
          chewbacca last edited by

          So, I've made some progress, historical downloads now resume normally, I had to pass the historicalDataEnd() message to the Queue and check the message type. Now I'm onto ibbroker and orders, hopefully all will go well. Updates to follow.

          @iswrapper
          @ibregister
          def historicalDataEnd(self, msg):
              ## For multi-tiered downloads we'd need to rebind the queue to a new
              ## tickerId (in case tickerIds are not reusable) and instead of putting
              ## None, issue a new reqHistData with the new data and move formward
              tickerId = msg.reqId
              q = self.qs[tickerId]
              self.histfmt.pop(tickerId, None)
              self.histsend.pop(tickerId, None)
              self.histtz.pop(tickerId, None)
              kargs = self.histexreq.pop(tickerId, None)
              if kargs is not None:
                  self.reqHistoricalDataEx(tickerId=tickerId, **kargs)
                  return
              else:
                  msg.start = None
                  msg.end = None
                  # <historicalDataEnd reqId=16777220, start=None, end=None>
                  # Put this message on the queue, check it's type @ ibdata->_load() -> elif self._state == self._ST_HISTORBACK:
                  q.put(msg)
          
                  if self.validQueue(q):
                      self.cancelQueue(q, False)
          
          
          @ibregister
          def historicalData(self, msg):
              '''Receives the events of a historical data request'''
          
              # For multi-tiered downloads we'd need to rebind the queue to a new
              # tickerId (in case tickerIds are not reusable) and instead of putting
              # None, issue a new reqHistData with the new data and move formward
              tickerId = msg.reqId
              q = self.qs[tickerId]
              msg_date = msg.bar.date # 'BarData' object
          
              dtstr = msg.bar.date
              if self.histfmt[tickerId]:
                  sessionend = self.histsend[tickerId]
                  dt = datetime.strptime(dtstr, '%Y%m%d')
                  dteos = datetime.combine(dt, sessionend)
                  tz = self.histtz[tickerId]
                  if tz:
                      dteostz = tz.localize(dteos)
                      dteosutc = dteostz.astimezone(UTC).replace(tzinfo=None)
                      # When requesting for example daily bars, the current day
                      # will be returned with the already happened data. If the
                      # session end were added, the new ticks wouldn't make it
                      # through, because they happen before the end of time
                  else:
                      dteosutc = dteos
          
                  if dteosutc <= datetime.utcnow():
                      dt = dteosutc
          
                  msg.bar.date = dt # msg.date -> msg.bar.date
              else:
                  msg.bar.date = datetime.utcfromtimestamp(long(dtstr))
          
              q.put(msg)
          
          
          
                  # . . . . . . . . . Functions in ibdata->_load() before this one:
          
                  elif self._state == self._ST_HISTORBACK:
                      msg = self.qhist.get()
                      if msg is None:  # Conn broken during historical/backfilling
                          #print("_load _ST_HISTORBACK msg is None; DISCONNECTED")
          
                          # Situation not managed. Simply bail out
                          self._subcription_valid = False
                          self.put_notification(status=self.DISCONNECTED)
                          return False  # error management cancelled the queue
          
                      # . . . . . . . . . Functions in between
          
                      msgType = msg.__class__.__name__
                      if msgType == "HistoricalDataEnd":
                          #print("Historical data download end, continuing...")
                          pass
                      else:
                          if msg.bar.date is not None: # msg.date -> msg.bar.date
                              #print("_load date is not None; _load_rtbar #2, hist=True: {}".format(msg))
                              if self._load_rtbar(msg, hist=True):
                                  #print("_load date _load_rtbar() returned True; True")
                                  return True  # loading worked
          
                              # the date is from overlapping historical request
                              #print("_load date _load_rtbar() returned False, continuing")
                              continue
          
                      # . . . . . . . . . Functions after
          
          1 Reply Last reply Reply Quote 0
          • C
            chewbacca last edited by

            Orders seem to work fine, moving on to contract request rework by using separate fields for sectype, symbol, exchange, currency, local_symbol, expiry_date, multiplier, etc, instead of a very weird 'name' conversion.

            After this is done, I'll look into news feeds & market signals requests, the idea is to have a separate function for each (like next(): market_signals() && news_feeds() and possibly acting on them separately)

            If someone's interested in any of this - let me know.

            1 Reply Last reply Reply Quote 0
            • C
              chewbacca last edited by

              Seriously, no interest at all?

              vladisld 1 Reply Last reply Reply Quote 0
              • D
                dasch last edited by

                @chewbacca do you have any github repository with some code to try it out?

                1 Reply Last reply Reply Quote 0
                • vladisld
                  vladisld @chewbacca last edited by

                  @chewbacca Personally I'm pretty interested, but there are quite a few BUTs.

                  IMHO the following issues need to be discussed:

                  • What new features the new implementation will bring compared to the current IB store implementation?

                  • Why IBpythonic and not other alternative like: native TWS Python API or ib_insync library?

                  • AFAIU the IBpythonic library is dependent on TWS Python API which supports only Python3. Although the python 2 is deprecated now, it is still officially supported in BT. If you development presents a replacement for the current implementation - this brings the question of future python support in BT itself.

                  • One of the main goals of BT was to provide a framework with as little external dependencies as possible. My hope was that one day we may get rid of IBpy dependency as well once the native TWS Python API was available. Right now you are trading IBpy for IBpythonic + TWS Python API that needs to be installed - effectively increasing the external dependencies.

                  • Testing: how are you going to prove your component will have a better quality compared with the current IB store implementation.?

                    IMHO, the main problem with the current IB store implementation is the lack of proper testing. There are no component level tests that provide a good coverage for all the scenarios involved in live trading nor for backtesting using IB. Without such tests any change to IB store source code is prune to regressions.

                    The only test for IB store that I'm aware of is a sample code provided with BT that needs to be run manually. The coverage of such testing is questionable at best.

                    One may say that given a long time in use the IB store component is slowly getting rid of its bugs - and this may even be partially true. However we are witnessing a different story - number of bugs in IB component is growing even after many years of use - last ones are just few weeks old.

                    I doubt that developing a replacement (or alternative) store using the similar code base without proper discussion of the approach, without clear test plan/infrastructure would make things better.

                  1 Reply Last reply Reply Quote 2
                  • C
                    chewbacca last edited by

                    @vladisld I'm still learning python.

                    I used ibpythonic because it wouldn't require as much changes to the existing code comparing to a complete transition to
                    the official TWS api (which right now is beyond my abilities).

                    Also, I decided to start doing this because.. well.. nobody else did. IbPy is quite old (2015 if memory serves), there's a lot of new functionality we could be using in BT by transitioning to the TWS api. The way I'm implementing this is by keeping the old ibbroker/ibstore/ibdata files and doing everything in a copy of these files. I'll try to rename all functions in the new files so they won't interfere with the old versions, meaning one could use the old or the new, their choice.

                    Let me do some cleanup and I'll place the 3 files here, you can review/change and place in your github repo (backtrader2).

                    1 Reply Last reply Reply Quote 0
                    • vladisld
                      vladisld last edited by

                      @chewbacca said in BT-IBAPI:

                      there's a lot of new functionality we could be using in BT by transitioning to the TWS api

                      It would be great if we could discuss all those new features and see if they worth it - unless the above statement was more of a general assumption of future TWS API capabilities that we will miss.

                      Any plans for test coverage ? How? Are you going to develop IBpythonic or even TWS API (python or even NI) mock ( this actually could help in current implementation testing) ?

                      @chewbacca said in BT-IBAPI:

                      I decided to start doing this because.. well.. nobody else did

                      The above list of issues to discuss is pretty tough to address - may explain why nobody has tried :-)

                      1 Reply Last reply Reply Quote 2
                      • C
                        chewbacca last edited by

                        Well, I tried, I'll provide everything here and its up to the community to use/develop it or not.

                        1 Reply Last reply Reply Quote 1
                        • 1 / 1
                        • First post
                          Last post
                        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors