Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Historical Data Request on IB question

    General Code/Help
    2
    4
    2439
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ricpruss
      ricpruss last edited by

      So I am trying download the historical data from IB.

      So I try run getdata on the store like this:

          stockkwargs = dict(
              timeframe = bt.TimeFrame.Days,
              rtbar = False,  # use RealTime 5 seconds bars
              historical = True,  # only historical download
              what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
              useRTH = False,  # historical - download only Regular Trading Hours
              qcheck = 0.5,  # timeout in seconds (float) to check for events
              backfill_start = True,  # do backfilling at the start
              backfill = True,  # do backfilling when reconnecting
              backfill_from = datetime.datetime(2017,1,2,tzinfo=timezone('US/Eastern')),  # additional data source to do backfill from
              latethrough = False,  # let late samples through
              tradename = None  # use a different asset as order target
              )
          data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
          cerebro.adddata(data0)
      

      But I get an error:

      Server Version: 76
      TWS Time at connection:20170503 10:13:24 AEST
      Traceback (most recent call last):
        File "tutorial3.py", line 129, in <module>
          runstrat()
        File "tutorial3.py", line 122, in runstrat
          cerebro.run()
        File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1070, in run
          runstrat = self.runstrategies(iterstrat)
        File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1144, in runstrategies
          data._start()
        File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feed.py", line 203, in _start
          self.start()
        File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feeds/ibdata.py", line 361, in start
          self.p.backfill_from._start()
      AttributeError: 'datetime.datetime' object has no attribute '_start'
      

      Now as far as I can see backfill_from should be a datetime object.
      Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.

      Ensure initialization of backfill_from feeds takes place

      Now if I simply remove that section of ibdata.py and set

            self._state = self._ST_START 
      

      The historical download works and it disconnects and then plots beautifully.

      So my question? Bug in ibdata.py or I am doing it wrong?
      Thanks,
      Ric

      B 1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators @ricpruss last edited by

        @ricpruss said in Historical Data Request on IB question:

        Now as far as I can see backfill_from should be a datetime object

        You see it wrong. See: Docs - Data Feeds Reference

        backfill_from is another data source meant to backfill from. The intent is to overcome the inherent backfilling limitations for the data feed provided by IB.

        Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.
        Ensure initialization of backfill_from feeds takes place
        Now if I simply remove that section of ibdata.py and set

           self._state = self._ST_START 
        

        The historical download works and it disconnects and then plots beautifully.

        Yes, because you completely remove backfilling from another data source. The datetime.datetime instance you pass as parameter is as valid as None. Nothing to blame in line 361.

        I am doing it wrong?

        Yes you are.

        ricpruss 1 Reply Last reply Reply Quote 1
        • ricpruss
          ricpruss @backtrader last edited by

          @backtrader Thanks, substituted fromdate instead of backfill_from and it works well.

          For the next person searching for how do download ib historical data here is the small snippet that worked.

              stockkwargs = dict(
                  timeframe = bt.TimeFrame.Days,
                  rtbar = False,  # use RealTime 5 seconds bars
                  historical = True,  # only historical download
                  what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
                  useRTH = False,  # historical - download only Regular Trading Hours
                  qcheck = 0.5,  # timeout in seconds (float) to check for events
                  backfill_start = True,  # do backfilling at the start
                  backfill = True,  # do backfilling when reconnecting
                  fromdate = datetime.datetime(2017,1,2),  # get data from..
                  latethrough = False,  # let late samples through
                  tradename = None  # use a different asset as order target
                  )
              data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
              cerebro.adddata(data0)
          
          B 1 Reply Last reply Reply Quote 2
          • B
            backtrader administrators @ricpruss last edited by backtrader

            @ricpruss said in Historical Data Request on IB question:

            stockkwargs = dict(
                    timeframe = bt.TimeFrame.Days,
                    rtbar = False,  # use RealTime 5 seconds bars
                    historical = True,  # only historical download
                    what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
                    useRTH = False,  # historical - download only Regular Trading Hours
                    qcheck = 0.5,  # timeout in seconds (float) to check for events
                    backfill_start = True,  # do backfilling at the start
                    backfill = True,  # do backfilling when reconnecting
                    fromdate = datetime.datetime(2017,1,2),  # get data from..
                    latethrough = False,  # let late samples through
                    tradename = None  # use a different asset as order target
                    )
            

            That combination of parameters seems overly complex. See Docs - Interactive Brokers (or the data feeds reference quoted above, which uses the same source)

            • timeframe defaults already to Days (all data feeds default actually to this)
            • rtbar defaults already to False as has no role in historical downloads
            • what defaults already to TRADES for an asset like *-STK-*
            • useRTH -> default already to False
            • qcheck defaults already to 0.5 and plays no role in historical downloads
            • backfill_start defaults already to True and plays no roles in historical downloads
            • backfill defaults already to True and plays no role in historical downloads
            • latethrough defaults already to Falseand plays no role in historical downloads
            • tradename defaults already to None and plays no role in data fetching at all

            Missing and could play a role:

            • todate which defaults to None to indicate until now and can specified to indicate what the ending data of the historical download has to be

            With that in mind and for a daily historical download like the above the parameters can be simplified to:

                stockkwargs = dict(
                    historical=True,
                    fromdate=datetime.datetime(2017, 1, 2),  # get data from
                )
            

            From the docs quoted above:

                  - ``historical`` (default: ``False``)
            
                    If set to ``True`` the data feed will stop after doing the first
                    download of data.
            
                    The standard data feed parameters ``fromdate`` and ``todate`` will be
                    used as reference.
            
                    The data feed will make multiple requests if the requested duration is
                    larger than the one allowed by IB given the timeframe/compression
                    chosen for the data.
            
            1 Reply Last reply Reply Quote 2
            • 1 / 1
            • First post
              Last post
            Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
            $(document).ready(function () { app.coldLoad(); }); }