For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

multi asset, time-frame complications (backfilling with IB?)



  • I hope this can help others that wish to implement a multi time-frame and asset strategy.

    I am having problems with the strategy actually running for what seems to be higher timeframe/compression combinations.

    When I run the code below with arguments --compression0 2 --timeframe1 minute --compression1 5 the strategy works as intended and I am receiving the signal output during the backfill process.

    Server Version: 76
    TWS Time at connection:20190104 14:37:53 AEST
    ***** EUR.USD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    ***** GBP.USD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    ***** GBP.AUD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    2019-01-03T10:00:00 EUR.USD-CASH-IDEALPRO-1d signal: none
    2019-01-03T10:00:00 GBP.USD-CASH-IDEALPRO-1d signal: none
    2019-01-03T10:00:00 GBP.AUD-CASH-IDEALPRO-1d signal: none
    2019-01-03T10:02:00 EUR.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T10:02:00 GBP.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T10:02:00 GBP.AUD-CASH-IDEALPRO-1d signal: sell
    ...
    2019-01-03T22:36:00 EUR.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T22:36:00 GBP.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T22:36:00 GBP.AUD-CASH-IDEALPRO-1d signal: sell
    ***** EUR.USD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    ***** GBP.USD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    ***** GBP.AUD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    2019-01-03T22:38:00 EUR.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T22:38:00 EUR.USD-CASH-IDEALPRO-60m signal: buy order id: 1
    2019-01-03T22:38:00 GBP.USD-CASH-IDEALPRO-1d signal: buy
    2019-01-03T22:38:00 GBP.USD-CASH-IDEALPRO-60m signal: buy order id: 2
    2019-01-03T22:38:00 GBP.AUD-CASH-IDEALPRO-1d signal: sell
    2019-01-03T22:38:00 GBP.AUD-CASH-IDEALPRO-60m signal: sell order id: 3
    

    However, when using the same code with arguments --compression0 60 --timeframe1 daily --compression1 1, backtrader connects to the broker but fails to produce any of the signals. The output below is all that is received and the strategy fails to execute (I have kept it running for multiple hours with no additional output). Running with debug shows that messages are being received from the broker, but nothing happens.

    Server Version: 76
    TWS Time at connection:20190104 14:37:15 AEST
    ***** EUR.USD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    ***** GBP.USD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    ***** GBP.AUD-CASH-IDEALPRO-60m DATA NOTIF: DELAYED
    ***** EUR.USD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    ***** GBP.USD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    ***** GBP.AUD-CASH-IDEALPRO-60m DATA NOTIF: LIVE
    

    The above example works when using 1hr/1day historical data saved locally. I feel like this has to do with IB but my understanding is that 1 year of data is within the historical download limits (needed for 200 day moving average). Potentially there is a problem with using multiple assets?

    I feel a bit stuck as I have been working on this for some time to no avail. I would appreciate if anyone has had any luck with the problems I am facing.

    Thank you for your time!

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import argparse
    import datetime
    import math
    
    import pandas as pd
    import backtrader as bt
    
    class ParcelSizer(bt.Sizer):
    
        params = dict(stake=10000)
    
        def _getsizing(self, comminfo, cash, data, isbuy):
            sz = math.floor(self.p.stake / data.close[0])
            return int(sz)
    
    class CFDSt(bt.Strategy):
        params = dict(
            smaperiod=200,
            rsiperiod=5,
            atrperiod=10,
        )
    
        def notify_order(self, order):
            if order.status == order.Submitted:
                return
    
            dt, dn = self.data.datetime.datetime(0).isoformat(), order.data._name
            print('{} {} Order {} Status {}'.format(
                dt, dn, order.ref, order.getstatusname())
            )
    
            if order.status == order.Completed:
                return
    
            whichord = ['main', 'stop', 'limit', 'close']
            if not order.alive():  # not alive - nullify
                dorders = self.o[order.data]
                idx = dorders.index(order)
                dorders[idx] = None
                print('{} {} Order no longer alive {} Ref'.format(dt, dn,
                                                                  whichord[idx]))
    
                if all(x is None for x in dorders):
                    dorders[:] = []  # empty list - New orders allowed
    
        def notify_trade(self, trade):
            dt, dn = self.data.datetime.datetime(0).isoformat(), trade.data._name
            if trade.isclosed:
                print('{} {} Gross Profit {} Net Profit {}' \
                .format(dt, dn, round(trade.pnl,2), round(trade.pnlcomm,2)))
    
    
        def __init__(self):
            self.o = dict()  # orders per data (main, stop, limit, manual-close)
            self.lendata = dict()
            self.inds = dict()
    
            for i, d in enumerate(self.datas):
                self.inds[d._name] = dict()
                self.inds[d._name]['sma'] = bt.indicators.SMA(d,
                                                period=self.p.smaperiod)
                self.inds[d._name]['signal'] = 'none'
    
        data_live = False
    
        def notify_data(self, data, status, *args, **kwargs):
            print('*' * 5, data._name ,'DATA NOTIF:', data._getstatusname(status),
                  *args)
            if status == data.LIVE:
                self.data_live = True
    
        def nextstart(self):
            for i, d in enumerate(self.datas):
                self.lendata[d] = len(d)
    
        def next(self):
            for i, d in enumerate(self.datas):
    
                dt, dn = self.data.datetime.datetime(0).isoformat(), d._name
                if (i + 1) % 2 != 0:
                    # every odd data feed (shorter timeframe first)
                    pos = self.getposition(d).size # get position
                    dnht = self.datas[i+1]._name # higher timeframe dataname
    
                    print('{} {} signal: {}'.format(dt, dnht,
                        self.inds[dnht]['signal'])) # print the signal from the higher timeframe
    
                    if self.inds[dnht]['signal'] == 'buy' and self.data_live:
                        self.o[d] = [self.buy(data=d)]
                        print('{} {} signal: {} order id: {}'.format(dt, dn,
                            self.inds[dnht]['signal'], self.o[d][0].ref))
                        self.inds[dnht]['signal'] = 'none' # reset signal
    
                    elif self.inds[dnht]['signal'] == 'sell' and self.data_live:
                        self.o[d] = [self.sell(data=d)]
                        print('{} {} signal: {} order id: {}'.format(dt, dn,
                            self.inds[dnht]['signal'], self.o[d][0].ref))
                        self.inds[dnht]['signal'] = 'none'
    
                    else:
                        pass
    
                else:
                    # every even data feed (higher timeframe second)
                    pos = self.getposition(self.datas[i-1]).size
                    # position on smaller time frame
                    dnlt = self.datas[i-1]._name # lower timeframe dataname
    
                    if self.lendata[d] < len(d):
                        # new higher timeframe bar
                        self.lendata[d] += 1
    
                        if not pos and not self.o.get(self.datas[i-1], None):
                            # no orders/position
                            if d.close[0] < self.inds[dn]['sma'][0]:
                                self.inds[dn]['signal'] = 'sell'
    
                            elif d.close[0] > self.inds[dn]['sma'][0]:
                                self.inds[dn]['signal'] = 'buy'
    
                            else:
                                self.inds[dn]['signal'] = 'none'
    
                        else:
                            pass
    
    def runstrat(args=None):
    
        args = parse_args(args)
        cerebro = bt.Cerebro()
    
        # Store
        store = bt.stores.IBStore(port=7497, host='127.0.0.1', _debug=args.debug,
                                  notifyall=args.notifyall, reconnect=-1,
                                  timeout=60)
    
        securities = ['EUR.USD-CASH-IDEALPRO','GBP.USD-CASH-IDEALPRO',
                      'GBP.AUD-CASH-IDEALPRO']
    
        tframes = dict(seconds=bt.TimeFrame.Seconds,
                       minute=bt.TimeFrame.Minutes, daily=bt.TimeFrame.Days,
                       weekly=bt.TimeFrame.Weeks, monthly=bt.TimeFrame.Months)
    
        for security, contract in zip(securities,contracts):
            data = store.getdata(dataname=security, tradename=contract,
                                 timeframe=tframes[args.timeframe],
                                 compression=int(args.compression))
    
            cerebro.resampledata(data, timeframe=tframes[args.timeframe0],
                                 compression=int(args.compression0),
                                 name=str(security+args.name0))
    
            cerebro.resampledata(data, timeframe=tframes[args.timeframe1],
                                 compression=int(args.compression1),
                                 name=str(security+args.name1))
    
        # Broker
        cerebro.broker = store.getbroker()
    
        # Strategy
        cerebro.addstrategy(CFDSt)
    
        # Sizer
        cerebro.addsizer(ParcelSizer)
    
        # Execute
        cerebro.run()
    
    def parse_args(pargs=None):
        parser = argparse.ArgumentParser(
            formatter_class=argparse.ArgumentDefaultsHelpFormatter,
            description=(
                'Multiple Values'
            )
        )
    
        parser.add_argument('--cerebro', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--debug', required=False,
                            default=False, action='store',
                            help='Debug parameter')
    
        parser.add_argument('--notifyall', required=False,
                            default=False, action='store',
                            help='If True, any error message sent by IB is relayed')
    
        parser.add_argument('--broker', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--sizer', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--strat', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--timeframe', required=False,
                            default='minute', action='store',
                            help='datafeed timeframe parameter')
    
        parser.add_argument('--compression', required=False,
                            default=5, action='store',
                            help='datafeed compression parameter')
    
        parser.add_argument('--timeframe0', required=False,
                            default='minute', action='store',
                            help='datafeed0 timeframe parameter (resampled)')
    
        parser.add_argument('--compression0', required=False,
                            default=60, action='store',
                            help='datafeed0 compression parameter (resampled)')
    
        parser.add_argument('--name0', required=False,
                            default='-60m', action='store',
                            help='datafeed0 name parameter (resampled)')
    
        parser.add_argument('--timeframe1', required=False,
                            default='daily', action='store',
                            help='datafeed1 timeframe parameter (resampled)')
    
        parser.add_argument('--compression1', required=False,
                            default=1, action='store',
                            help='datafeed1 compression parameter (resampled)')
    
        parser.add_argument('--name1', required=False,
                            default='-1d', action='store',
                            help='datafeed1 name parameter (resampled)')
    
        parser.add_argument('--asx', required=False,
                            default='20', action='store',
                            help='asx group parameter')
    
        return parser.parse_args(pargs)
    
    
    if __name__ == '__main__':
        runstrat()
    

  • administrators

    Get independent data feeds (even if for the same underlying asset) for each resampling and give it a try. The timeframes are too far apart and (without calculating it) it can't probably not get enough historical data in 60-minutes format to fulfill the 1-day needs.

    Do it in general, even if the timeframes are not so different, given the constraints IB puts in place for data downloading.



  • Thank you very much! This appears to have solved the issue.