Starting with Interactive Brokers (IB) - no data feed

rluc99

Hi
I am following the blog in Medium:
https://medium.com/@danjrod/interactive-brokers-in-python-with-backtrader-23dea376b2fc
So far I can get connection notification from TWS (ie the message: "TWS Time at connection...." but there is no data received. There are no run time errors or other messages.
I am using the code (without modification) in the blog section "Getting a data stream".
I am using a demo (paper trading) account on IB
Thanks

rluc99

I think I might have found the problem
The relevant code in the blog (which was written in May of last year) is

    data = store.getdata(dataname='TWTR',
                         timeframe=bt.TimeFrame.Ticks)

However I think that IB may now require more info in the dataname string than just the ticker (eg also exchange information)

backtrader

The complete documentation for working with Interactive Brokers: https://www.backtrader.com/docu/live/ib/ib/

As pointed out there, some tickers require extra information (a lot more tokens in the string) and some require less. If TWTR only required one token initially and now you say it may require more (you provide no data as to what the actual problem is) it may simply be because it now trades in more exchanges (i.e.: internationally) and probably under different currencies.

The only thing you can do is query the contracts and see how many come back under the ticker TWTR.

rluc99

@backtrader thanks for you reply. I will do as you suggest, and do some more investigation using TWS directly to find out what the problem is.

rluc99

I am at a loss! I have managed to get data from IB for forex pairs but I have failed with stocks, and futures. This applies for both backfill and realtime data

I have used the example contracts in
https://interactivebrokers.github.io/tws-api/basic_contracts.html
as a guide
I am using the demo account on IB
I have TWS running with no errors
Any suggestions would be welcome as I am running out of ideas!

Here is my code for GBP.USD which works:

def run(args=None):

    cerebro = bt.Cerebro(stdstats=False)
    cerebro.addstrategy(St)
    store = bt.stores.IBStore(port=7497)
    data = store.getdata(dataname='GBP.USD',
                         sectype='CASH',
                         exchange='IDEALPRO',
                         fromdate=datetime.datetime(2019, 9, 11),
                         todate=datetime.datetime(2019, 9, 12),
                         timeframe=bt.TimeFrame.Minutes,
                         compression=5)
    cerebro.adddata(data)
    cerebro.run()

if __name__ == '__main__':
    run()

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB.py
Server Version: 76
TWS Time at connection:20190924 21:18:34 BST
***** DATA NOTIF: DELAYED
1,2019-09-11T21:15:00,1.23210,1.23269,1.23210,1.23269,-1.00000
2,2019-09-11T21:20:00,1.23269,1.23315,1.23260,1.23273,-1.00000
.
.
.
.
33,2019-09-11T23:55:00,1.23341,1.23345,1.23330,1.23342,-1.00000
34,2019-09-12T00:00:00,1.23342,1.23342,1.23286,1.23307,-1.00000

Process finished with exit code 0

and here is my code for AAPL which doesn't work

    data = store.getdata(dataname='AAPL',
                         sectype='STK',
                         exchange='SMART',
                         currency='USD',
                         fromdate=datetime.datetime(2019, 9, 11),
                         todate=datetime.datetime(2019, 9, 12),
                         timeframe=bt.TimeFrame.Minutes,
                         compression=5)

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB.py
Server Version: 76
TWS Time at connection:20190924 20:41:11 BST
***** DATA NOTIF: DELAYED

nothing appears even after waiting 30 mins with the program running!

backtrader

@rluc99 said in Starting with Interactive Brokers (IB) - no data feed:

                         fromdate=datetime.datetime(2019, 9, 11),
                         todate=datetime.datetime(2019, 9, 12),

You seem to be willing to do a historical download, but there is no historical parameter.

https://www.backtrader.com/docu/live/ib/ib/

rluc99

@backtrader Thanks. I will give historical=True a try.
I thought it wasn't needed if a todate was specified (otherwise why does the first case work?)

backtrader

Because Forex trades 24 hours a day.

rluc99

@backtrader I see. So in that case I presume I will only get historical stock data from IB when the market is open (and with historical=True) even though it is available from TWS at any time.

rluc99

Still no luck accessing data. Is there some way of debugging the communication between IB and backtrader?

Code:

    data = store.getdata(dataname='AAPL',
                         secType='STK',
                         exchange='SMART',
                         currency='USD',
                         historical=True,
                         timeframe=bt.TimeFrame.Minutes,
                         compression=5)

output:

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB.py
Server Version: 76
TWS Time at connection:20190926 19:15:55 British Summer Time

Process finished with exit code 0

code

    data = store.getdata(dataname='AAPL',
                         secType='STK',
                         exchange='SMART',
                         currency='USD',
                         historical=False,
                         timeframe=bt.TimeFrame.Minutes,
                         compression=5)

output:

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB.py
Server Version: 76
TWS Time at connection:20190926 19:17:03 British Summer Time
***** DATA NOTIF: DELAYED

run-out

First let's get some historical data. Try doing it like this. Set up your stock key word arguments as follows:

stockkwargs = dict(
       timeframe=bt.TimeFrame.Minutes,
       rtbar=False,  # use RealTime 5 seconds bars
       historical=True,  # only historical download
       qcheck=0.5,  # timeout in seconds (float) to check for events
       fromdate=datetime(2019, 9, 24),  # get data from..
       todate=datetime(2019, 9, 25),  # get data from..
       latethrough=False,  # let late samples through
       tradename=None  # use a different asset as order target
   )

Then get your data using:

data0 = store.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)

Then add your data using:

cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=5)

If you print from next() in your strategy, or call a log function from there, you should get some results. Make sure TWS is open and connected.

This whole code works for historical. Once you have that working, switch to live and try again when the market is open.

#!/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-e

from __future__ import absolute_import, division, print_function, unicode_literals
import backtrader as bt
import datetime

class St(bt.Strategy):

    def logdata(self):
            txt = []
            txt.append("{}".format(self.data._name))
            txt.append("{}".format(len(self)))
            txt.append("{}".format(self.data.datetime.date(0)))
            txt.append("{:.2f}".format(self.data.open[0]))
            txt.append("{:.2f}".format(self.data.high[0]))
            txt.append("{:.2f}".format(self.data.low[0]))
            txt.append("{:.2f}".format(self.data.close[0]))
            txt.append("{:.2f}".format(self.data.volume[0]))
            print(", ".join(txt))


    def next(self):
        self.logdata()


def run():

    cerebro = bt.Cerebro(stdstats=False)

    store = bt.stores.IBStore(host="127.0.0.1", port=7497, clientId= ###insert_your_number_here### )
    cerebro.broker = store.getbroker()

    stockkwargs = dict(
        timeframe=bt.TimeFrame.Minutes,
        rtbar=False,  # use RealTime 5 seconds bars
        historical=True,  # only historical download
        qcheck=0.5,  # timeout in seconds (float) to check for events
        fromdate=datetime.datetime(2019, 9, 24),  # get data from..
        todate=datetime.datetime(2019, 9, 25),  # get data from..
        latethrough=False,  # let late samples through
        tradename=None  # use a different asset as order target
    )
    data0 = store.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
    cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=5)

    cerebro.addstrategy(St)
    cerebro.run()




if __name__ == "__main__":

    run()

The output starts like this...

TWS Time at connection:20190926 17:50:20 EST
AAPL-STK-SMART-USD, 1, 2019-09-24, 219.90, 219.90, 219.80, 219.80, 12.00
AAPL-STK-SMART-USD, 2, 2019-09-24, 219.80, 219.80, 219.80, 219.80, 0.00
AAPL-STK-SMART-USD, 3, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 1.00
AAPL-STK-SMART-USD, 4, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 0.00
AAPL-STK-SMART-USD, 5, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 0.00
AAPL-STK-SMART-USD, 6, 2019-09-24, 219.96, 220.30, 219.96, 220.30, 12.00
AAPL-STK-SMART-USD, 7, 2019-09-24, 220.30, 220.67, 220.30, 220.39, 35.00
AAPL-STK-SMART-USD, 8, 2019-09-24, 220.21, 220.21, 220.16, 220.16, 16.00
AAPL-STK-SMART-USD, 9, 2019-09-24, 220.05, 220.06, 220.03, 220.06, 4.00

rluc99

@run-out . Thanks so much for taking the time to put together your comprehensive response!

I copied your code exactly, apart from the clientId= parameter which I did not include since I am using the IB demo account without a client ID. See below:

Unfortunately, I get the same result as I was getting before. IE I can receive forex bars but not stocks.
In other words this line:

 data0 = store.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)

produces:

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB_connection.py
Server Version: 76
TWS Time at connection:20190927 16:41:10 British Summer Time

Process finished with exit code 0

Whereas replacing it with this line:

    data0 = store.getdata(dataname="GBP.USD-CASH-IDEALPRO", **stockkwargs)

produces:

C:\Users\Owner\Miniconda3\envs\projectv36_2\python.exe C:/Users/Owner/TestTA-LIB/test_IB_connection.py
Server Version: 76
TWS Time at connection:20190927 16:56:40 British Summer Time
GBP.USD-CASH-IDEALPRO, 1, 2019-09-24, 1.25, 1.25, 1.25, 1.25, -1.00
GBP.USD-CASH-IDEALPRO, 2, 2019-09-24, 1.25, 1.25, 1.25, 1.25, -1.00
.
.
.
GBP.USD-CASH-IDEALPRO, 32, 2019-09-24, 1.25, 1.25, 1.25, 1.25, -1.00
GBP.USD-CASH-IDEALPRO, 33, 2019-09-24, 1.25, 1.25, 1.25, 1.25, -1.00

Process finished with exit code 0

TWS is open and connected and the market is open. AAPL bars are showing up on the chart (after a refresh). See below:

I am using the latest TWS version:

I feel it should be something very obvious but I have no idea what!

run-out

Sorry I haven't used the demo account. You might want to search https://groups.io/g/twsapi .

rluc99

@run-out Thanks - I'll give it a try

rluc99

I have discovered the problem. It is not related to Backtrader.

I did some digging around on the IB website. It seems that the IB demo account does not allow historical bars through the API for anything other than FOREX. The same applies to paper trading or live accounts unless there is a subscription to the relevant market data feed. If there is a subscription in place then the historical bars are available at any time, including week-ends.
(As an aside, historical bars are constructed by TWS regardless of whether there is a subscription in place or not.)
Delayed data is available during market hours for accounts without market data subscription, but to obtain this data a flag has to be set in the API. I don't think Backtrader allows the user to set this flag but I have not investigated this in detail.

Most of the above information comes from the following two links:
http://interactivebrokers.github.io/tws-api/market_data.html
http://interactivebrokers.github.io/tws-api/historical_data.html

run-out

@rluc99 Thanks for the update!

Squirrel Singing

One comment that I see.

I believe that we can remove the parameter on timeframe in stockkwargs so that it becomes as follows since we will be resampling the data anyway later on

stockkwargs = dict(
rtbar=False, # use RealTime 5 seconds bars
historical=True, # only historical download
qcheck=0.5, # timeout in seconds (float) to check for events
fromdate=datetime.datetime(2019, 9, 24), # get data from..
todate=datetime.datetime(2019, 9, 25), # get data from..
latethrough=False, # let late samples through
tradename=None # use a different asset as order target