Error while retrieving IBStore historic data with an IB paper account

[kriku]

Hello

I connect to an existing paper account as following:

ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid)
data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                     timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.adddata(data)

I get the following error:

14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
    self.broker.push_orderstate(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstate'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
    self.broker.push_orderstatus(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
    self.broker.push_orderstate(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstate'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
    self.broker.push_orderstatus(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
    self.broker.push_orderstate(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstate'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
    self.broker.push_orderstatus(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
    self.broker.push_orderstate(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstate'
14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
Traceback (most recent call last):
  File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
    results.append(listener(message))
  File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
    self.broker.push_orderstatus(msg)
AttributeError: 'NoneType' object has no attribute 'push_orderstatus'

It seems to be that the server starts pushing order status etc to the client and the BackBroker does not know what to do with them. Quite logical. When I add an IBBroker as following:

ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid)
cerebro.broker = ibstore.getbroker()
data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                     timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.adddata(data)

that error is gone but I will soon get another one:

Traceback (most recent call last):
  File "C:/Users/ksander/correlation/bt_katse.py", line 66, in <module>
    cerebro.broker.setcash(100000.0)
AttributeError: 'IBBroker' object has no attribute 'setcash'

Also logical - the live broker cannot let you set your cash indeed :)

The question is, what is the proper way to download the historical data from the IB if I cannot use an existing paper account with standing orders etc because of that issue?

Does the BackBroker need some exception handling?

Thanks in advance.

[kriku]

It seems to be that it was a mistake to query with a real client ID. That works:

ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=666)    
data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                     timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.adddata(data)

# Set our desired cash start
cerebro.broker.setcash(100000.0)

# Print out the starting conditions
print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

# Run over everything
cerebro.run()

[kriku]

I was too optimistic. It does not raise an exception but it neither retrieves any data. It would be surprising if it did with a fictional client id.

Please help.

[backtrader]

clientId identifiers are not fictional. They simply let you identify the client issuing operations against TWS. When you don't specify one, you see everything which is what you obviously suffered at the beginning. (although you seem to use the Gateway)

It has nothing to do with data retrieval. Data retrieval is governed by data permissions which are the same for the paper trading account and the real trading account. You have to explicitly share the data permissions and only use them from one (this policy may have been subject to change overtime, check with IB)

As the documentation already points out, aapl (which should be written as AAPL) is a bad example for data download, given the fact it is listed in multiple exchanges and currencies, which means you have to be specific about what you actually want (call IB and let them know you simply want aapl if you don't like that policy)

Please read: Docs - Live Trading - Interactive Brokers and check the actual data permissions your paper trading account.

[kriku]

In the beginning, I used my real account ID which I would not like to disclose here. It is a 7-digit number: 105xxxx. That resulted in the Exception from the BackBroker. I am sorry that I failed to indicate it in my first post.

Later, I tried a fictional client id such as 666 to see what happens.

It is not an issue of permissions since I do get the historic data with a test client that I wrote from scratch with the same clientId, including AAPL. Yes, I am using the GW.

The exception is raised regardless I use "aapl" or "AAPL". I suspect that it happens before the historical data download.

[backtrader]

Sorry but you are confused. Your account number has NOTHING to do with a clientId. The clientId is an identifier for YOU. If you have multiple clients connecting to TWS/IBGateway, you can distinguish the clients using the clientId.

[kriku]

Yes, I know I am confused. Nothing to be sorry about. :) Thanks for the clarification about the clientId.

However, I am getting even more confused. The outcomes of using my account ID for the clientId and an arbitrary number such as 666 differ. How could THAT be???

This results in the exception described in my first post:

ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=105****)  # my account number, digits replaced with asterisks
data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
     timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.adddata(data)

This does not raise an exception but does not retrieve any data either:

ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666)
data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
    timeframe=bt.TimeFrame.Minutes, compression=5)
cerebro.adddata(data)

[backtrader]

Check what the setting for master Client Id is ... https://www.interactivebrokers.com/en/software/tws/usersguidebook/configuretws/apisettings.htm

[kriku]

Thanks, I had completely forgotten about this setting.

The field was empty. However, I filled it with 6666666 and now BOTH of the code snippets in my last post raise the same Exception while a similar request with clientId = 7777777 does not. That's a bit odd but goes beyond the scope of the Backtrader discussion.

However, I still cannot retrieve the historical data and I cannot understand what I am doing wrong. Here is the whole code (based on the First Strategy tutorial):

from _datetime import datetime
import backtrader as bt

# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('Close, %.2f' % self.dataclose[0])


if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=7777777)
    #ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666)
    data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
            timeframe=bt.TimeFrame.Minutes, compression=5)
    cerebro.adddata(data)

    cerebro.addstrategy(TestStrategy)

    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

The output is:

Starting Portfolio Value: 100000.00
Server Version: 76
TWS Time at connection:20190817 12:16:29 
Final Portfolio Value: 100000.00

The breakpoint in the next() is never reached, no Exceptions are raised.

To be sure about the permissions, I just downloaded the 5 minute AAPL bars for the last 2 years with my own client.

Many thanks for your time.

[backtrader]

@kriku said in Error while retrieving IBStore historic data with an IB paper account:

The breakpoint in the next() is never reached, no Exceptions are raised.

You are ignoring the data and store notifications. You are using AAPL, which, as stated above, is a non-deterministic ticker for Interactive Brokers. This is in the documentation. Your other client must already fill information for a specific target.

Again.

@backtrader said in Error while retrieving IBStore historic data with an IB paper account:

As the documentation already points out, aapl (which should be written as AAPL) is a bad example for data download, given the fact it is listed in multiple exchanges and currencies, which means you have to be specific about what you actually want (call IB and let them know you simply want aapl if you don't like that policy)

Please read: Docs - Live Trading - Interactive Brokers

The AAPL case is clearly explained there.

The sample code for ib does also show how to use data and store notifications to understand that the platform is telling you it cannot fulfill your request.

[kriku]

That's right. This works:

data = ibstore.getdata(dataname="AAPL-STK-SMART-USD", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5)

I am sorry for having been a nuisance. BT is an impressive project and the forums show that you provide prompt and straight-to-the-point support even to stupid questions. Keep up the good work!

[bazar guru]

@kriku I am trying to replicate the same scenario for Indian stock . I have a paper account with IB.

from _datetime import datetime
import backtrader as bt

# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('Close, %.2f' % self.dataclose[0])


if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=7777777)
    #ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666)
    data = ibstore.getdata(dataname="RELIANCE-STK-SMART-IND", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5)

    cerebro.adddata(data)

    cerebro.addstrategy(TestStrategy)

    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

But this is not giving me any data. Getting below output :

Starting Portfolio Value: 100000.00
Final Portfolio Value: 100000.00

I have the permission enabled in IB to share the live account data with paper account. Am i missing something ?