Error while retrieving IBStore historic data with an IB paper account
-
Hello
I connect to an existing paper account as following:
ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid) data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data)
I get the following error:
14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'openOrder' for 'openOrder' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder self.broker.push_orderstate(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstate' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'orderStatus' for 'orderStatus' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus self.broker.push_orderstatus(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstatus' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'openOrder' for 'openOrder' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder self.broker.push_orderstate(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstate' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'orderStatus' for 'orderStatus' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus self.broker.push_orderstatus(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstatus' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'openOrder' for 'openOrder' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder self.broker.push_orderstate(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstate' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'orderStatus' for 'orderStatus' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus self.broker.push_orderstatus(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstatus' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'openOrder' for 'openOrder' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder self.broker.push_orderstate(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstate' 14-Aug-19 12:24:24 ERROR Exception in message dispatch. Handler 'orderStatus' for 'orderStatus' Traceback (most recent call last): File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__ results.append(listener(message)) File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus self.broker.push_orderstatus(msg) AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
It seems to be that the server starts pushing order status etc to the client and the BackBroker does not know what to do with them. Quite logical. When I add an IBBroker as following:
ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid) cerebro.broker = ibstore.getbroker() data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data)
that error is gone but I will soon get another one:
Traceback (most recent call last): File "C:/Users/ksander/correlation/bt_katse.py", line 66, in <module> cerebro.broker.setcash(100000.0) AttributeError: 'IBBroker' object has no attribute 'setcash'
Also logical - the live broker cannot let you set your cash indeed :)
The question is, what is the proper way to download the historical data from the IB if I cannot use an existing paper account with standing orders etc because of that issue?
Does the BackBroker need some exception handling?
Thanks in advance.
-
It seems to be that it was a mistake to query with a real client ID. That works:
ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=666) data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run()
-
I was too optimistic. It does not raise an exception but it neither retrieves any data. It would be surprising if it did with a fictional client id.
Please help.
-
clientId
identifiers are not fictional. They simply let you identify the client issuing operations againstTWS
. When you don't specify one, you see everything which is what you obviously suffered at the beginning. (although you seem to use theGateway
)It has nothing to do with data retrieval. Data retrieval is governed by data permissions which are the same for the paper trading account and the real trading account. You have to explicitly share the data permissions and only use them from one (this policy may have been subject to change overtime, check with
IB
)As the documentation already points out,
aapl
(which should be written asAAPL
) is a bad example for data download, given the fact it is listed in multiple exchanges and currencies, which means you have to be specific about what you actually want (callIB
and let them know you simply wantaapl
if you don't like that policy)Please read: Docs - Live Trading - Interactive Brokers and check the actual data permissions your paper trading account.
-
In the beginning, I used my real account ID which I would not like to disclose here. It is a 7-digit number: 105xxxx. That resulted in the Exception from the BackBroker. I am sorry that I failed to indicate it in my first post.
Later, I tried a fictional client id such as 666 to see what happens.
It is not an issue of permissions since I do get the historic data with a test client that I wrote from scratch with the same clientId, including AAPL. Yes, I am using the GW.
The exception is raised regardless I use "aapl" or "AAPL". I suspect that it happens before the historical data download.
-
Sorry but you are confused. Your account number has NOTHING to do with a
clientId
. TheclientId
is an identifier for YOU. If you have multiple clients connecting toTWS/IBGateway
, you can distinguish the clients using theclientId
. -
Yes, I know I am confused. Nothing to be sorry about. :) Thanks for the clarification about the clientId.
However, I am getting even more confused. The outcomes of using my account ID for the clientId and an arbitrary number such as 666 differ. How could THAT be???
This results in the exception described in my first post:
ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=105****) # my account number, digits replaced with asterisks data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data)
This does not raise an exception but does not retrieve any data either:
ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666) data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data)
-
Check what the setting for master Client Id is ... https://www.interactivebrokers.com/en/software/tws/usersguidebook/configuretws/apisettings.htm
-
Thanks, I had completely forgotten about this setting.
The field was empty. However, I filled it with 6666666 and now BOTH of the code snippets in my last post raise the same Exception while a similar request with clientId = 7777777 does not. That's a bit odd but goes beyond the scope of the Backtrader discussion.
However, I still cannot retrieve the historical data and I cannot understand what I am doing wrong. Here is the whole code (based on the First Strategy tutorial):
from _datetime import datetime import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=7777777) #ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666) data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data) cerebro.addstrategy(TestStrategy) cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
The output is:
Starting Portfolio Value: 100000.00 Server Version: 76 TWS Time at connection:20190817 12:16:29 Final Portfolio Value: 100000.00
The breakpoint in the next() is never reached, no Exceptions are raised.
To be sure about the permissions, I just downloaded the 5 minute AAPL bars for the last 2 years with my own client.
Many thanks for your time.
-
@kriku said in Error while retrieving IBStore historic data with an IB paper account:
The breakpoint in the next() is never reached, no Exceptions are raised.
You are ignoring the data and store notifications. You are using
AAPL
, which, as stated above, is a non-deterministic ticker for Interactive Brokers. This is in the documentation. Your other client must already fill information for a specific target.Again.
@backtrader said in Error while retrieving IBStore historic data with an IB paper account:
As the documentation already points out, aapl (which should be written as AAPL) is a bad example for data download, given the fact it is listed in multiple exchanges and currencies, which means you have to be specific about what you actually want (call IB and let them know you simply want aapl if you don't like that policy)
Please read: Docs - Live Trading - Interactive Brokers
The
AAPL
case is clearly explained there.The sample code for
ib
does also show how to use data and store notifications to understand that the platform is telling you it cannot fulfill your request. -
That's right. This works:
data = ibstore.getdata(dataname="AAPL-STK-SMART-USD", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5)
I am sorry for having been a nuisance. BT is an impressive project and the forums show that you provide prompt and straight-to-the-point support even to stupid questions. Keep up the good work!
-
@kriku I am trying to replicate the same scenario for Indian stock . I have a paper account with IB.
from _datetime import datetime import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=7777777) #ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666) data = ibstore.getdata(dataname="RELIANCE-STK-SMART-IND", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5) cerebro.adddata(data) cerebro.addstrategy(TestStrategy) cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
But this is not giving me any data. Getting below output :
Starting Portfolio Value: 100000.00 Final Portfolio Value: 100000.00
I have the permission enabled in IB to share the live account data with paper account. Am i missing something ?