Historical Data Request on IB question

ricpruss

So I am trying download the historical data from IB.

So I try run getdata on the store like this:

    stockkwargs = dict(
        timeframe = bt.TimeFrame.Days,
        rtbar = False,  # use RealTime 5 seconds bars
        historical = True,  # only historical download
        what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
        useRTH = False,  # historical - download only Regular Trading Hours
        qcheck = 0.5,  # timeout in seconds (float) to check for events
        backfill_start = True,  # do backfilling at the start
        backfill = True,  # do backfilling when reconnecting
        backfill_from = datetime.datetime(2017,1,2,tzinfo=timezone('US/Eastern')),  # additional data source to do backfill from
        latethrough = False,  # let late samples through
        tradename = None  # use a different asset as order target
        )
    data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
    cerebro.adddata(data0)

But I get an error:

Server Version: 76
TWS Time at connection:20170503 10:13:24 AEST
Traceback (most recent call last):
  File "tutorial3.py", line 129, in <module>
    runstrat()
  File "tutorial3.py", line 122, in runstrat
    cerebro.run()
  File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1070, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/cerebro.py", line 1144, in runstrategies
    data._start()
  File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feed.py", line 203, in _start
    self.start()
  File "/home/ric/miniconda3/envs/bt34/lib/python3.5/site-packages/backtrader/feeds/ibdata.py", line 361, in start
    self.p.backfill_from._start()
AttributeError: 'datetime.datetime' object has no attribute '_start'

Now as far as I can see backfill_from should be a datetime object.
Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.

Ensure initialization of backfill_from feeds takes place

Now if I simply remove that section of ibdata.py and set

      self._state = self._ST_START

The historical download works and it disconnects and then plots beautifully.

So my question? Bug in ibdata.py or I am doing it wrong?
Thanks,
Ric

backtrader

@ricpruss said in Historical Data Request on IB question:

Now as far as I can see backfill_from should be a datetime object

You see it wrong. See: Docs - Data Feeds Reference

backfill_from is another data source meant to backfill from. The intent is to overcome the inherent backfilling limitations for the data feed provided by IB.

Now looking at git blame I can see that line 361 of ibdata.py was added a few months ago.
Ensure initialization of backfill_from feeds takes place
Now if I simply remove that section of ibdata.py and set

   self._state = self._ST_START

The historical download works and it disconnects and then plots beautifully.

Yes, because you completely remove backfilling from another data source. The datetime.datetime instance you pass as parameter is as valid as None. Nothing to blame in line 361.

I am doing it wrong?

Yes you are.

ricpruss

@backtrader Thanks, substituted fromdate instead of backfill_from and it works well.

For the next person searching for how do download ib historical data here is the small snippet that worked.

    stockkwargs = dict(
        timeframe = bt.TimeFrame.Days,
        rtbar = False,  # use RealTime 5 seconds bars
        historical = True,  # only historical download
        what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
        useRTH = False,  # historical - download only Regular Trading Hours
        qcheck = 0.5,  # timeout in seconds (float) to check for events
        backfill_start = True,  # do backfilling at the start
        backfill = True,  # do backfilling when reconnecting
        fromdate = datetime.datetime(2017,1,2),  # get data from..
        latethrough = False,  # let late samples through
        tradename = None  # use a different asset as order target
        )
    data0 = ibstore.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
    cerebro.adddata(data0)

backtrader

@ricpruss said in Historical Data Request on IB question:

stockkwargs = dict(
        timeframe = bt.TimeFrame.Days,
        rtbar = False,  # use RealTime 5 seconds bars
        historical = True,  # only historical download
        what = 'TRADES',  # historical - what to show TRADES MIDPOINT BID ASK BID_ASK HISTORICAL_VOLATILITY OPTION_IMPLIED_VOLATILITY
        useRTH = False,  # historical - download only Regular Trading Hours
        qcheck = 0.5,  # timeout in seconds (float) to check for events
        backfill_start = True,  # do backfilling at the start
        backfill = True,  # do backfilling when reconnecting
        fromdate = datetime.datetime(2017,1,2),  # get data from..
        latethrough = False,  # let late samples through
        tradename = None  # use a different asset as order target
        )

That combination of parameters seems overly complex. See Docs - Interactive Brokers (or the data feeds reference quoted above, which uses the same source)

timeframe defaults already to Days (all data feeds default actually to this)
rtbar defaults already to False as has no role in historical downloads
what defaults already to TRADES for an asset like *-STK-*
useRTH -> default already to False
qcheck defaults already to 0.5 and plays no role in historical downloads
backfill_start defaults already to True and plays no roles in historical downloads
backfill defaults already to True and plays no role in historical downloads
latethrough defaults already to Falseand plays no role in historical downloads
tradename defaults already to None and plays no role in data fetching at all

Missing and could play a role:

todate which defaults to None to indicate until now and can specified to indicate what the ending data of the historical download has to be

With that in mind and for a daily historical download like the above the parameters can be simplified to:

    stockkwargs = dict(
        historical=True,
        fromdate=datetime.datetime(2017, 1, 2),  # get data from
    )

From the docs quoted above:

      - ``historical`` (default: ``False``)

        If set to ``True`` the data feed will stop after doing the first
        download of data.

        The standard data feed parameters ``fromdate`` and ``todate`` will be
        used as reference.

        The data feed will make multiple requests if the requested duration is
        larger than the one allowed by IB given the timeframe/compression
        chosen for the data.