More IB bugs in live trading

ard9000

I have a strategy that I'm running with my IB paper trading account. If I'm running three symbols, says AAPL, JNJ, CSCO and the strategy status is:

AAPL long 1200 shares
JNJ flat no position
CSCO long 800 shares

When live trading with IB, if my code looks like this:

data = self.datas[i]
position = getposition(data)

entry_price = position.price
position_size = position.size

Let's say I'm pulling data for AAPL this way. For AAPL, it returns:

position.size = 2000 and
position.entry_price = the correct entry price

The entry price works correctly, but the position.size returns the aggregate of all open positions. However the result gets stranger when using a different method. If I use:

data = getdatabyname('AAPL')
position = getposition(data)

entry_price = position.price
position_size = position.size

The result of entry_price is an average across all symbols and position_size is still the aggregate of all open positions. The same thing happens if I use:

data = self.datas[i]
position = getpositionbyname('AAPL')

entry_price = position.price
position_size = position.size

These all work fine in backtesting. I verified that [i] is the correct position in datas[i]. Is there something I'm missing here?

backtrader

@ard9000 said in More IB bugs in live trading:

data = self.datas[i]
position = getposition(data)

entry_price = position.price
position_size = position.size

Your code cannot for sure look like that. Because it wouldn't work. The platform may have bugs, but they can for sure only be found if something sensible is posted, i.e.: a minimum working implementation, expected input and actual output (a couple of print statetements)

Starting a thread with "More Bugs" when none has been proven to be there, it also not very friendly.

@ard9000 said in More IB bugs in live trading:

data = getdatabyname('AAPL')
position = getposition(data)

entry_price = position.price
position_size = position.size

That's also likely not to work, because unless things have lately changed a lot, the ticker AAPL needs extra modifiers with Interactive Brokers to be resolved (because it is traded in multiple exchanges and currencies) Or else you have assigned a name to the data. But in the absence of how things are done and with it, not knowing if you have assigned names or are referring to the IB ticket, it is impossible to know what fails and what doesn't.

Now, the standard testing code buying 2 assets and printing the positions out, before the 1st order.

0 Position size: 0  - Price: 0.0
1 Position size: 0  - Price: 0.0
-------------------------------------------------- ORDER BEGIN 2018-05-14 13:30:11.418792

0_1526301093364_31830fcd-726f-48ef-a1fb-a5702c22b2d3-image.png

And after the orders:

-------------------------------------------------- TRADE END
Data0, 1442, 736828.5208912037, 2018-05-14T08:30:05.000000, 185.25, 185.25, 185.25, 185.25, 1.0, 0, 180.634
Data1, 1443, 736828.5209490741, 2018-05-14T08:30:10.000000, 31.53, 31.54, 31.51, 31.52, 10.0, 0, nan
0 Position size: 10  - Price: 185.34
1 Position size: 20  - Price: 31.61

which reflects the actual size and prices of the assets in play.

ard9000

I'm sorry for the 'more bugs' headline, I agree it was harsh and not productive to resolving the issue.

I am a licensed professional trader in the US, I'm a partner at a prop firm.I really love the platform and am championing bringing it into our firm as our python platform. We've been using it for the past 18 months to develop algos with and are now implementing it for live trading. We're using IB to test with as we build a FIX broker interface to work with Backtrader and our servers.

My code does indeed look like that. We had Ed Bartosh integrate IQFeed into Backtrader for us. Ed also integrated CCXT into the platform for crypto trading, both of which we are giving to the Backtrader community for (hopefully) future inclusion in the platform. Both are available from his git branch. We've been debugging the IQFeed implementation for months in live trading, I believe it's production worthy.

So we don't have to deal with using the IB symbol in our code 'AAPL-STK-SMART-USD' I run the code like this:


 params = (
        ('stocks', ('AAPL','TWTR','VIX.XO',)),
        ('tradable', ('AAPL','TWTR',)),
        ('timezone', timezone('US/Eastern')),)

for i, stock in enumerate(self.p.stocks):
            if stock in self.p.tradable:
                self.contracts.append(ibstore.makecontract(stock,'STK','SMART','USD'))
                self.datas[i].tradecontract = self.contracts[i]

In main, I setup the symbols this way:

# Set broker
    broker = bt.brokers.IBBroker(host=args.host, port=args.port,
                                 clientId=args.clientId, reconnect=args.reconnect,
                                 timeout=args.timeout, _debug=args.debug)
    cerebro.setbroker(broker)

    # Create store
    store = bt.stores.IQFeedStore(reconnect=args.reconnect,
                                  timeout=args.timeout,
                                  debug=args.debug)
    
    # Add data feeds
    
    for stock in Strategy.params.stocks:
        ticker = stock.split('-')[0]

        hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime()
        data = bt.feeds.IQFeed(dataname=ticker, name=ticker,
                               tz=Strategy.params.timezone,
                               timeframe=bt.TimeFrame.Minutes,
                               sessionstart=datetime.time(hour=9, minute=30),
                               sessionend=datetime.time(hour=16, minute=00, second=59),
                               fromdate=hist_start_date)
        cerebro.adddata(data)

Doing it this way, you can see we have the ticker = stock.split('-')[0] left over from when we were using the long IB symbols in self.p.stocks.

We've made a test script to verify the bugs in the position.size and position.price when multiple positions are open in the same script. The other problem is the stop loss orders, they are intermittent.

 stop_order = self.close(data_min, price=stop_price, exectype=bt.Order.Stop)
 self.stop_orders[stock] = stop_order
 self.orders.append(stop_order)

If the script is run 10 times, sometimes the stop orders will appear in IB TWS, other times they won't. There seems to be no reproducible pattern as to why the sometimes do and don't appear. All other orders work fine.

Since this platform is free (and I really like it) community driven and to encourage growth of the platform, we're going to fix these now and give them to the community. Even though we will never trade live with IB. If you can give Ed some help when he needs it to expedite fixing these, it would be very helpful for everybody.

ard9000

The test script that demonstrates this with IQFeed with the long IB symbols. If I'm doing something incorrectly here, I'd appreciate what is wrong.


#!/usr/bin/env python
"""
IQFeed live data

IB Brokerage 

"""

import pandas as pd

import argparse
import datetime
import os
import sys

from pytz import timezone
from pandas_market_calendars import get_calendar
from pandas import Series, DataFrame, bdate_range

import backtrader as bt

class Strategy(bt.Strategy):
    """Trading strategy class."""

    params = (
        ('stocks', ('AAPL-STK-SMART-USD','TWTR-STK-SMART-USD','CAT-STK-SMART-USD',)),
        ('timezone', timezone('US/Eastern')),
        ('rsize', 1000),
        ('stdperiod', 30),
        ('debug', True),
        ('info', True),
    )

    def __init__(self):
        bt.Strategy.__init__(self)

        # create IB contract objects

        ibstore = bt.stores.ibstore.IBStore() # to be able to call makecontract API
        self.contracts = [ibstore.makecontract(*stock.split('-')) for stock in self.p.stocks]
        for i, stock in enumerate(self.p.stocks):
            self.datas[i].tradecontract = self.contracts[i]
        
        # initialize parameters

        self.position_sizes = { 'AAPL-STK-SMART-USD': 1200,
                                'TWTR-STK-SMART-USD': 0,
                                'CAT-STK-SMART-USD': 300}
                                
        
        self.stop_orders = {}
        self.df_min = {}
        self.daily = {}
        self.orders = []
        self.entry_price = {}
              
        # set early close days
        calendar = get_calendar('NYSE')
        now = datetime.datetime.now()
        schedule = calendar.schedule(start_date=now - datetime.timedelta(weeks=1),
                                     end_date=now + datetime.timedelta(weeks=200))
        self.early_closes = calendar.early_closes(schedule).index

        self.p.debug = True

    def debug(self, msg):
        if self.p.debug:
            dtime = self.datas[0].datetime
            print('debug: %s %s, %s' % (dtime.date(0, tz=self.p.timezone),
                                        dtime.time(0, tz=self.p.timezone), msg))

    def info(self, msg):
        if self.p.info:
            dtime = self.datas[0].datetime
            print('info: %s %s, %s' % (dtime.date(0, tz=self.p.timezone),
                                       dtime.time(0, tz=self.p.timezone), msg))

    def notify_order(self, order):
        if order.status == order.Completed:
            # calculate cost basis
            stock = order.p.data._name
            value = order.executed.value + order.executed.comm
            entry_price = abs(value / order.executed.size)
            self.info('%s: %s order executed: price: %.2f, value: %.2f, commm %.2f, '
                      'cost basis: %.2f, stock price: %s' %
                      (stock, order.ref, order.executed.price, order.executed.value,
                       order.executed.comm, entry_price, order.p.data[0]))
            self.orders.remove(order)
            
           

    def notify_trade(self, trade):
        if trade.isclosed:

            self.info('Trade value: %.2f, P/L %.2f, P/L comm: %.2f' %
                      (trade.value, trade.pnl, trade.pnlcomm ))
           
    def eod(self):
        """
        Calculate time of eod actions.
        """
        if self.data0.datetime.date() in self.early_closes:
            return datetime.time(12, 30)
        else:
            return datetime.time(15, 54)


    def next(self):
        """Main processing is happening here. This method is called for every bar."""
    
       # check to make sure all data has been accumulated before calling next

        ctime = self.data0.datetime.time(0, tz=self.p.timezone)
        cdate = self.data0.datetime.date(0, tz=self.p.timezone)
        cdt = self.data0.datetime.datetime(0, tz=self.p.timezone)
        tz = self.p.timezone

        # skip bars from history
        if datetime.datetime.now(tz=tz) - tz.localize(self.data0.datetime.datetime()) >= datetime.timedelta(minutes=1):
            return

        # check to make sure all data has been accumulated before calling next
        
        if len(set(data.datetime.datetime() for data in self.datas )) > 1:                                                                             
            # skip processing if all minute bars not arrived yet                                                                                                                     
            return

        # don't process bars after 16:00 or before 9:30
        
        for data in self.datas:
            if data.datetime.time(0, tz=self.p.timezone) < datetime.time(9,30) or self.data.datetime.time(0, tz=self.p.timezone) > datetime.time(16,0,0):
                return

                    
        for i, stock in enumerate(self.p.stocks):
            
           # get position data - first method using getdatabyname(stock)
             
            data_min = self.datas[i]
            position = self.getposition(data_min)

           # if there is no position and it's not set to zero, create a short order for it

            if not position.size and self.position_sizes[stock] != 0:
               order = self.sell(data_min, size=self.position_sizes[stock])
               self.orders.append(order)
               self.stop_orders[stock] = ''

           # second position method

            data_min2 = self.getdatabyname(stock.split("-")[0])
            position2 = self.getpositionbyname(stock.split("-")[0])

            position_size2 = position2.size
            entry_price2 = position2.price

            self.info(" \n getpositionbyname method %s position size: %s entry price: %s" %(stock, position_size2, entry_price2))

           # demonstrate stop orders not being set

            if position.size != 0 and position.price:

                self.entry_price[stock] = position.price

                self.info("%s position size: %s entry price: %s current close: %s" %(stock, position.size, position.price, data_min.close[0]))

                if not self.stop_orders.get(stock) and self.entry_price[stock]:
                    stop_price = self.entry_price[stock] + .3
                    stop_order = self.close(data_min, price=stop_price, exectype=bt.Order.Stop)
                    self.stop_orders[stock] = stop_order
                    self.orders.append(stop_order)
                    self.info(" %s: stop order sent at %.2f" %(stock,stop_price))

                

def parse_args():
    parser = argparse.ArgumentParser(
                 formatter_class=argparse.ArgumentDefaultsHelpFormatter,
                 description='Live strategy run on Interactive Brokers')

    parser.add_argument('--host', default='127.0.0.1',
                        required=False, action='store',
                        help='Host for the Interactive Brokers TWS Connection')

    parser.add_argument('--qcheck', default=0.5, type=float,
                        required=False, action='store',
                        help=('Timeout for periodic '
                              'notification/resampling/replaying check'))

    parser.add_argument('--port', default=7497, type=int,
                        required=False, action='store',
                        help='Port for the Interactive Brokers TWS Connection')

    parser.add_argument('--clientId', default=None, type=int,
                        required=False, action='store',
                        help='Client Id to connect to TWS (default: random)')

    parser.add_argument('--reconnect', default=3, type=int,
                        required=False, action='store',
                        help='Number of recconnection attempts to TWS')

    parser.add_argument('--timeout', default=180, type=float,
                        required=False, action='store',
                        help='Timeout between reconnection attempts to TWS')

    parser.add_argument('--debug',
                        required=False, action='store_true',
                        help='Display all info received form IB')

    return parser.parse_args()
                
def main(argv):
    """Entry point."""
    args = parse_args()

    # Create a cerebro entity
    cerebro = bt.Cerebro(maxcpus=1, oldsync=False)

    # Set broker
    broker = bt.brokers.IBBroker(host=args.host, port=args.port,
                                 clientId=args.clientId, reconnect=args.reconnect,
                                 timeout=args.timeout, _debug=args.debug)
    cerebro.setbroker(broker)

    # Create store
    store = bt.stores.IQFeedStore(reconnect=args.reconnect,
                                  timeout=args.timeout,
                                  debug=args.debug)
    
    # Add data feeds
    
    for stock in Strategy.params.stocks:
        ticker = stock.split('-')[0]

        hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime()
        data = bt.feeds.IQFeed(dataname=ticker, name=ticker,
                               tz=Strategy.params.timezone,
                               timeframe=bt.TimeFrame.Minutes,
                               sessionstart=datetime.time(hour=9, minute=30),
                               sessionend=datetime.time(hour=16, minute=00, second=59),
                               fromdate=hist_start_date)
        cerebro.adddata(data)

    # Print out the starting conditions
    print 'Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()

    # Add a strategy
    cerebro.addstrategy(Strategy)

    # Run over everything
    cerebro.run()

if __name__ == '__main__':
    sys.exit(main(sys.argv))

bigdavediode

@ard9000 Without putting in an hour on your code I can tell you two things that I've found in my experience have been difficult for me. The first is that stop losses are tricky to implement properly and often they won't show any trades at all, for example, if a position is already opened and the code, hypothetically, only allows one open position at a time. The stop trail would not update when, as one example, there was an issue being called during next at a bar, and thus no trades would subsequently open as there was already a position open.

Another issue was capturing the errors from the data provider, in my case, IB. When I used the standard method of retrieving data, returned data provider error codes were not broken out in debug mode. For example, in your case this could be because you haven't sufficiently specified the exchange/primaryexchange. In that event this could mask the problem, or perhaps obtain varying data from different exchanges on the same instrument due to smart routing.

In my case (with IB) I cache data directly from an IBDataCache object to a pandas dataframe (just search for IB data cache on here if you want to look at this) which facilitated monitoring data provider errors/return codes, such as exchange, and then adjust the primaryexchange or other required data provider api arguments appropriately to specify a particular exchange.