Historical Data Request on IB question



  • So I am trying download the historical data from IB.

    So I try run getdata on the store like this:

        stockkwargs = dict(
            timeframe = bt.TimeFrame.Days,
            rtbar = False,  # use RealTime 5 seconds bars
            historical = True,  # only historical download
            what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
            useRTH = False,  # historical - download only Regular Trading Hours
            qcheck = 0.5,  # timeout in seconds (float) to check for events
            backfill_start = True,  # do backfilling at the start
            backfill = True,  # do backfilling when reconnecting
            backfill_from = datetime.datetime(2017,1,2,tzinfo=timezone('US/Eastern')),  # additional data source to do backfill from
            latethrough = False,  # let late samples through
            tradename = None  # use a different asset as order target
            )
        data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
        cerebro.adddata(data0)
    

    But I get an error:

    Server Version: 76
    TWS Time at connection:20170503 10:13:24 AEST
    Traceback (most recent call last):
      File "tutorial3.py", line 129, in <module>
        runstrat()
      File "tutorial3.py", line 122, in runstrat
        cerebro.run()
      File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1070, in run
        runstrat = self.runstrategies(iterstrat)
      File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1144, in runstrategies
        data._start()
      File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feed.py", line 203, in _start
        self.start()
      File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feeds/ibdata.py", line 361, in start
        self.p.backfill_from._start()
    AttributeError: 'datetime.datetime' object has no attribute '_start'
    

    Now as far as I can see backfill_from should be a datetime object.
    Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.

    Ensure initialization of backfill_from feeds takes place

    Now if I simply remove that section of ibdata.py and set

          self._state = self._ST_START 
    

    The historical download works and it disconnects and then plots beautifully.

    So my question? Bug in ibdata.py or I am doing it wrong?
    Thanks,
    Ric


  • administrators

    @ricpruss said in Historical Data Request on IB question:

    Now as far as I can see backfill_from should be a datetime object

    You see it wrong. See: Docs - Data Feeds Reference

    backfill_from is another data source meant to backfill from. The intent is to overcome the inherent backfilling limitations for the data feed provided by IB.

    Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.
    Ensure initialization of backfill_from feeds takes place
    Now if I simply remove that section of ibdata.py and set

       self._state = self._ST_START 
    

    The historical download works and it disconnects and then plots beautifully.

    Yes, because you completely remove backfilling from another data source. The datetime.datetime instance you pass as parameter is as valid as None. Nothing to blame in line 361.

    I am doing it wrong?

    Yes you are.



  • @backtrader Thanks, substituted fromdate instead of backfill_from and it works well.

    For the next person searching for how do download ib historical data here is the small snippet that worked.

        stockkwargs = dict(
            timeframe = bt.TimeFrame.Days,
            rtbar = False,  # use RealTime 5 seconds bars
            historical = True,  # only historical download
            what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
            useRTH = False,  # historical - download only Regular Trading Hours
            qcheck = 0.5,  # timeout in seconds (float) to check for events
            backfill_start = True,  # do backfilling at the start
            backfill = True,  # do backfilling when reconnecting
            fromdate = datetime.datetime(2017,1,2),  # get data from..
            latethrough = False,  # let late samples through
            tradename = None  # use a different asset as order target
            )
        data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
        cerebro.adddata(data0)
    

  • administrators

    @ricpruss said in Historical Data Request on IB question:

    stockkwargs = dict(
            timeframe = bt.TimeFrame.Days,
            rtbar = False,  # use RealTime 5 seconds bars
            historical = True,  # only historical download
            what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
            useRTH = False,  # historical - download only Regular Trading Hours
            qcheck = 0.5,  # timeout in seconds (float) to check for events
            backfill_start = True,  # do backfilling at the start
            backfill = True,  # do backfilling when reconnecting
            fromdate = datetime.datetime(2017,1,2),  # get data from..
            latethrough = False,  # let late samples through
            tradename = None  # use a different asset as order target
            )
    

    That combination of parameters seems overly complex. See Docs - Interactive Brokers (or the data feeds reference quoted above, which uses the same source)

    • timeframe defaults already to Days (all data feeds default actually to this)
    • rtbar defaults already to False as has no role in historical downloads
    • what defaults already to TRADES for an asset like *-STK-*
    • useRTH -> default already to False
    • qcheck defaults already to 0.5 and plays no role in historical downloads
    • backfill_start defaults already to True and plays no roles in historical downloads
    • backfill defaults already to True and plays no role in historical downloads
    • latethrough defaults already to Falseand plays no role in historical downloads
    • tradename defaults already to None and plays no role in data fetching at all

    Missing and could play a role:

    • todate which defaults to None to indicate until now and can specified to indicate what the ending data of the historical download has to be

    With that in mind and for a daily historical download like the above the parameters can be simplified to:

        stockkwargs = dict(
            historical=True,
            fromdate=datetime.datetime(2017, 1, 2),  # get data from
        )
    

    From the docs quoted above:

          - ``historical`` (default: ``False``)
    
            If set to ``True`` the data feed will stop after doing the first
            download of data.
    
            The standard data feed parameters ``fromdate`` and ``todate`` will be
            used as reference.
    
            The data feed will make multiple requests if the requested duration is
            larger than the one allowed by IB given the timeframe/compression
            chosen for the data.
    

Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.