For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

More IB bugs in live trading



  • I have a strategy that I'm running with my IB paper trading account. If I'm running three symbols, says AAPL, JNJ, CSCO and the strategy status is:

    AAPL long 1200 shares
    JNJ flat no position
    CSCO long 800 shares

    When live trading with IB, if my code looks like this:

    data = self.datas[i]
    position = getposition(data)
    
    entry_price = position.price
    position_size = position.size
    

    Let's say I'm pulling data for AAPL this way. For AAPL, it returns:

    position.size = 2000 and
    position.entry_price = the correct entry price

    The entry price works correctly, but the position.size returns the aggregate of all open positions. However the result gets stranger when using a different method. If I use:

    data = getdatabyname('AAPL')
    position = getposition(data)
    
    entry_price = position.price
    position_size = position.size
    

    The result of entry_price is an average across all symbols and position_size is still the aggregate of all open positions. The same thing happens if I use:

    data = self.datas[i]
    position = getpositionbyname('AAPL')
    
    entry_price = position.price
    position_size = position.size
    

    These all work fine in backtesting. I verified that [i] is the correct position in datas[i]. Is there something I'm missing here?


  • administrators

    @ard9000 said in More IB bugs in live trading:

    data = self.datas[i]
    position = getposition(data)
    
    entry_price = position.price
    position_size = position.size
    

    Your code cannot for sure look like that. Because it wouldn't work. The platform may have bugs, but they can for sure only be found if something sensible is posted, i.e.: a minimum working implementation, expected input and actual output (a couple of print statetements)

    Starting a thread with "More Bugs" when none has been proven to be there, it also not very friendly.

    @ard9000 said in More IB bugs in live trading:

    data = getdatabyname('AAPL')
    position = getposition(data)
    
    entry_price = position.price
    position_size = position.size
    

    That's also likely not to work, because unless things have lately changed a lot, the ticker AAPL needs extra modifiers with Interactive Brokers to be resolved (because it is traded in multiple exchanges and currencies) Or else you have assigned a name to the data. But in the absence of how things are done and with it, not knowing if you have assigned names or are referring to the IB ticket, it is impossible to know what fails and what doesn't.

    Now, the standard testing code buying 2 assets and printing the positions out, before the 1st order.

    0 Position size: 0  - Price: 0.0
    1 Position size: 0  - Price: 0.0
    -------------------------------------------------- ORDER BEGIN 2018-05-14 13:30:11.418792
    

    0_1526301093364_31830fcd-726f-48ef-a1fb-a5702c22b2d3-image.png

    And after the orders:

    -------------------------------------------------- TRADE END
    Data0, 1442, 736828.5208912037, 2018-05-14T08:30:05.000000, 185.25, 185.25, 185.25, 185.25, 1.0, 0, 180.634
    Data1, 1443, 736828.5209490741, 2018-05-14T08:30:10.000000, 31.53, 31.54, 31.51, 31.52, 10.0, 0, nan
    0 Position size: 10  - Price: 185.34
    1 Position size: 20  - Price: 31.61
    

    which reflects the actual size and prices of the assets in play.



  • I'm sorry for the 'more bugs' headline, I agree it was harsh and not productive to resolving the issue.

    I am a licensed professional trader in the US, I'm a partner at a prop firm.I really love the platform and am championing bringing it into our firm as our python platform. We've been using it for the past 18 months to develop algos with and are now implementing it for live trading. We're using IB to test with as we build a FIX broker interface to work with Backtrader and our servers.

    My code does indeed look like that. We had Ed Bartosh integrate IQFeed into Backtrader for us. Ed also integrated CCXT into the platform for crypto trading, both of which we are giving to the Backtrader community for (hopefully) future inclusion in the platform. Both are available from his git branch. We've been debugging the IQFeed implementation for months in live trading, I believe it's production worthy.

    So we don't have to deal with using the IB symbol in our code 'AAPL-STK-SMART-USD' I run the code like this:

    
     params = (
            ('stocks', ('AAPL','TWTR','VIX.XO',)),
            ('tradable', ('AAPL','TWTR',)),
            ('timezone', timezone('US/Eastern')),)
    
    for i, stock in enumerate(self.p.stocks):
                if stock in self.p.tradable:
                    self.contracts.append(ibstore.makecontract(stock,'STK','SMART','USD'))
                    self.datas[i].tradecontract = self.contracts[i]
    

    In main, I setup the symbols this way:

    # Set broker
        broker = bt.brokers.IBBroker(host=args.host, port=args.port,
                                     clientId=args.clientId, reconnect=args.reconnect,
                                     timeout=args.timeout, _debug=args.debug)
        cerebro.setbroker(broker)
    
        # Create store
        store = bt.stores.IQFeedStore(reconnect=args.reconnect,
                                      timeout=args.timeout,
                                      debug=args.debug)
        
        # Add data feeds
        
        for stock in Strategy.params.stocks:
            ticker = stock.split('-')[0]
    
            hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime()
            data = bt.feeds.IQFeed(dataname=ticker, name=ticker,
                                   tz=Strategy.params.timezone,
                                   timeframe=bt.TimeFrame.Minutes,
                                   sessionstart=datetime.time(hour=9, minute=30),
                                   sessionend=datetime.time(hour=16, minute=00, second=59),
                                   fromdate=hist_start_date)
            cerebro.adddata(data)
    

    Doing it this way, you can see we have the ticker = stock.split('-')[0] left over from when we were using the long IB symbols in self.p.stocks.

    We've made a test script to verify the bugs in the position.size and position.price when multiple positions are open in the same script. The other problem is the stop loss orders, they are intermittent.

     stop_order = self.close(data_min, price=stop_price, exectype=bt.Order.Stop)
     self.stop_orders[stock] = stop_order
     self.orders.append(stop_order)
    

    If the script is run 10 times, sometimes the stop orders will appear in IB TWS, other times they won't. There seems to be no reproducible pattern as to why the sometimes do and don't appear. All other orders work fine.

    Since this platform is free (and I really like it) community driven and to encourage growth of the platform, we're going to fix these now and give them to the community. Even though we will never trade live with IB. If you can give Ed some help when he needs it to expedite fixing these, it would be very helpful for everybody.



  • The test script that demonstrates this with IQFeed with the long IB symbols. If I'm doing something incorrectly here, I'd appreciate what is wrong.

    
    #!/usr/bin/env python
    """
    IQFeed live data
    
    IB Brokerage 
    
    """
    
    import pandas as pd
    
    import argparse
    import datetime
    import os
    import sys
    
    from pytz import timezone
    from pandas_market_calendars import get_calendar
    from pandas import Series, DataFrame, bdate_range
    
    import backtrader as bt
    
    class Strategy(bt.Strategy):
        """Trading strategy class."""
    
        params = (
            ('stocks', ('AAPL-STK-SMART-USD','TWTR-STK-SMART-USD','CAT-STK-SMART-USD',)),
            ('timezone', timezone('US/Eastern')),
            ('rsize', 1000),
            ('stdperiod', 30),
            ('debug', True),
            ('info', True),
        )
    
        def __init__(self):
            bt.Strategy.__init__(self)
    
            # create IB contract objects
    
            ibstore = bt.stores.ibstore.IBStore() # to be able to call makecontract API
            self.contracts = [ibstore.makecontract(*stock.split('-')) for stock in self.p.stocks]
            for i, stock in enumerate(self.p.stocks):
                self.datas[i].tradecontract = self.contracts[i]
            
            # initialize parameters
    
            self.position_sizes = { 'AAPL-STK-SMART-USD': 1200,
                                    'TWTR-STK-SMART-USD': 0,
                                    'CAT-STK-SMART-USD': 300}
                                    
            
            self.stop_orders = {}
            self.df_min = {}
            self.daily = {}
            self.orders = []
            self.entry_price = {}
                  
            # set early close days
            calendar = get_calendar('NYSE')
            now = datetime.datetime.now()
            schedule = calendar.schedule(start_date=now - datetime.timedelta(weeks=1),
                                         end_date=now + datetime.timedelta(weeks=200))
            self.early_closes = calendar.early_closes(schedule).index
    
            self.p.debug = True
    
        def debug(self, msg):
            if self.p.debug:
                dtime = self.datas[0].datetime
                print('debug: %s %s, %s' % (dtime.date(0, tz=self.p.timezone),
                                            dtime.time(0, tz=self.p.timezone), msg))
    
        def info(self, msg):
            if self.p.info:
                dtime = self.datas[0].datetime
                print('info: %s %s, %s' % (dtime.date(0, tz=self.p.timezone),
                                           dtime.time(0, tz=self.p.timezone), msg))
    
        def notify_order(self, order):
            if order.status == order.Completed:
                # calculate cost basis
                stock = order.p.data._name
                value = order.executed.value + order.executed.comm
                entry_price = abs(value / order.executed.size)
                self.info('%s: %s order executed: price: %.2f, value: %.2f, commm %.2f, '
                          'cost basis: %.2f, stock price: %s' %
                          (stock, order.ref, order.executed.price, order.executed.value,
                           order.executed.comm, entry_price, order.p.data[0]))
                self.orders.remove(order)
                
               
    
        def notify_trade(self, trade):
            if trade.isclosed:
    
                self.info('Trade value: %.2f, P/L %.2f, P/L comm: %.2f' %
                          (trade.value, trade.pnl, trade.pnlcomm ))
               
        def eod(self):
            """
            Calculate time of eod actions.
            """
            if self.data0.datetime.date() in self.early_closes:
                return datetime.time(12, 30)
            else:
                return datetime.time(15, 54)
    
    
        def next(self):
            """Main processing is happening here. This method is called for every bar."""
        
           # check to make sure all data has been accumulated before calling next
    
            ctime = self.data0.datetime.time(0, tz=self.p.timezone)
            cdate = self.data0.datetime.date(0, tz=self.p.timezone)
            cdt = self.data0.datetime.datetime(0, tz=self.p.timezone)
            tz = self.p.timezone
    
            # skip bars from history
            if datetime.datetime.now(tz=tz) - tz.localize(self.data0.datetime.datetime()) >= datetime.timedelta(minutes=1):
                return
    
            # check to make sure all data has been accumulated before calling next
            
            if len(set(data.datetime.datetime() for data in self.datas )) > 1:                                                                             
                # skip processing if all minute bars not arrived yet                                                                                                                     
                return
    
            # don't process bars after 16:00 or before 9:30
            
            for data in self.datas:
                if data.datetime.time(0, tz=self.p.timezone) < datetime.time(9,30) or self.data.datetime.time(0, tz=self.p.timezone) > datetime.time(16,0,0):
                    return
    
                        
            for i, stock in enumerate(self.p.stocks):
                
               # get position data - first method using getdatabyname(stock)
                 
                data_min = self.datas[i]
                position = self.getposition(data_min)
    
               # if there is no position and it's not set to zero, create a short order for it
    
                if not position.size and self.position_sizes[stock] != 0:
                   order = self.sell(data_min, size=self.position_sizes[stock])
                   self.orders.append(order)
                   self.stop_orders[stock] = ''
    
               # second position method
    
                data_min2 = self.getdatabyname(stock.split("-")[0])
                position2 = self.getpositionbyname(stock.split("-")[0])
    
                position_size2 = position2.size
                entry_price2 = position2.price
    
                self.info(" \n getpositionbyname method %s position size: %s entry price: %s" %(stock, position_size2, entry_price2))
    
               # demonstrate stop orders not being set
    
                if position.size != 0 and position.price:
    
                    self.entry_price[stock] = position.price
    
                    self.info("%s position size: %s entry price: %s current close: %s" %(stock, position.size, position.price, data_min.close[0]))
    
                    if not self.stop_orders.get(stock) and self.entry_price[stock]:
                        stop_price = self.entry_price[stock] + .3
                        stop_order = self.close(data_min, price=stop_price, exectype=bt.Order.Stop)
                        self.stop_orders[stock] = stop_order
                        self.orders.append(stop_order)
                        self.info(" %s: stop order sent at %.2f" %(stock,stop_price))
    
                    
    
    def parse_args():
        parser = argparse.ArgumentParser(
                     formatter_class=argparse.ArgumentDefaultsHelpFormatter,
                     description='Live strategy run on Interactive Brokers')
    
        parser.add_argument('--host', default='127.0.0.1',
                            required=False, action='store',
                            help='Host for the Interactive Brokers TWS Connection')
    
        parser.add_argument('--qcheck', default=0.5, type=float,
                            required=False, action='store',
                            help=('Timeout for periodic '
                                  'notification/resampling/replaying check'))
    
        parser.add_argument('--port', default=7497, type=int,
                            required=False, action='store',
                            help='Port for the Interactive Brokers TWS Connection')
    
        parser.add_argument('--clientId', default=None, type=int,
                            required=False, action='store',
                            help='Client Id to connect to TWS (default: random)')
    
        parser.add_argument('--reconnect', default=3, type=int,
                            required=False, action='store',
                            help='Number of recconnection attempts to TWS')
    
        parser.add_argument('--timeout', default=180, type=float,
                            required=False, action='store',
                            help='Timeout between reconnection attempts to TWS')
    
        parser.add_argument('--debug',
                            required=False, action='store_true',
                            help='Display all info received form IB')
    
        return parser.parse_args()
                    
    def main(argv):
        """Entry point."""
        args = parse_args()
    
        # Create a cerebro entity
        cerebro = bt.Cerebro(maxcpus=1, oldsync=False)
    
        # Set broker
        broker = bt.brokers.IBBroker(host=args.host, port=args.port,
                                     clientId=args.clientId, reconnect=args.reconnect,
                                     timeout=args.timeout, _debug=args.debug)
        cerebro.setbroker(broker)
    
        # Create store
        store = bt.stores.IQFeedStore(reconnect=args.reconnect,
                                      timeout=args.timeout,
                                      debug=args.debug)
        
        # Add data feeds
        
        for stock in Strategy.params.stocks:
            ticker = stock.split('-')[0]
    
            hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime()
            data = bt.feeds.IQFeed(dataname=ticker, name=ticker,
                                   tz=Strategy.params.timezone,
                                   timeframe=bt.TimeFrame.Minutes,
                                   sessionstart=datetime.time(hour=9, minute=30),
                                   sessionend=datetime.time(hour=16, minute=00, second=59),
                                   fromdate=hist_start_date)
            cerebro.adddata(data)
    
        # Print out the starting conditions
        print 'Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()
    
        # Add a strategy
        cerebro.addstrategy(Strategy)
    
        # Run over everything
        cerebro.run()
    
    if __name__ == '__main__':
        sys.exit(main(sys.argv))