Multi Example


  • administrators

    Use the link below to go the original post

    Click here to see the full blog post



  • Thank you for this valuable example!



  • Thank you, this is a great example.

    I have a question regarding "Margin" order statuses here.
    What if your logic results in an order placement when there isn't enough cash? I have this when I run similar logic across many more stocks.

    The bracket causes the creation of the stop-loss and take-profit, even though the original 'main' order could not execute... so eventually you end up in short positions when the bracket orders execute at their stop/limit prices. How should this be managed?

    I tried doing order.cancels when a order.Margin status occurs, however sometimes the stoploss/takeprofit executes that day of creation, and the cancellation doesnt occur in time.


  • administrators

    @cwse said in Multi Example:

    The bracket causes the creation of the stop-loss and take-profit, even though the original 'main' order could not execute... so eventually you end up in short positions when the bracket orders execute at their stop/limit prices. How should this be managed?

    The stop-loss and take-profit orders are created but

    • Are canceled if the parent order cannot be accepted or is canceled

    • Are created inactive and will only become active (available for execution) if the parent is Completed

      This is actually acts as a safeguard which prevents the execution of stop-loss or take-profit sell orders even if they are accidentally accepted.

    If any of those 2 things is not working as described there for the implemented bracket order functionality, it would be a bug.

    If the orders are issued manually with no relationship (parent/children or through the buy_bracket / sell_bracket), there is no way to avoid execution of one order, because the other was canceled.



  • @backtrader, then it appears there is a bug..


  • administrators

    A use case to reproduce the bug is very much appreciated.



  • @backtrader, exactly per my script provided in this post: "How is Getvalue() Calculated?".

    I added a print when position goes negative as you suggested,and it only occurs after a bracket was submitted, the 'buy' goes to margin and then the stop-loss and take-profit subsequently (undesirably) execute.

    Alternatively, you may get the same functionality when you run this multi example with higher stakes (such that many executions will result in a margin) or with many more stocks.

    Thanks,
    CWE


  • administrators

    If you have run the multi sample and produced the bug, may you share the execution command?



  • Just tried running your code per below (minor edits under the def runstrat() sections so it would run:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    
    import argparse
    import datetime
    
    import backtrader as bt
    
    
    class Sizer(bt.Sizer):
        params = dict(stake=1)
    
        def _getsizing(self, comminfo, cash, data, isbuy):
            dt, i = self.strategy.datetime.date(), data._id
            s = self.p.stake * (1 + (not isbuy))
            print('{} Data {} OType {} Sizing to {}'.format(
                dt, data._name, ('buy' * isbuy) or 'sell', s))
    
            return s
    
    
    class St(bt.Strategy):
        params = dict(
            enter=[1, 3, 4],  # data ids are 1 based
            hold=[7, 10, 15],  # data ids are 1 based
            usebracket=True,
            rawbracket=True,
            pentry=0.015,
            plimits=0.03,
            valid=10,
        )
    
        def notify_order(self, order):
            if order.status == order.Submitted:
                return
    
            dt, dn = self.datetime.date(), order.data._name
            print('{} {} Order {} Status {}'.format(
                dt, dn, order.ref, order.getstatusname())
            )
    
            whichord = ['main', 'stop', 'limit', 'close']
            if not order.alive():  # not alive - nullify
                dorders = self.o[order.data]
                idx = dorders.index(order)
                dorders[idx] = None
                print('-- No longer alive {} Ref'.format(whichord[idx]))
    
                if all(x is None for x in dorders):
                    dorders[:] = []  # empty list - New orders allowed
    
        def __init__(self):
            self.o = dict()  # orders per data (main, stop, limit, manual-close)
            self.holding = dict()  # holding periods per data
    
        def next(self):
            for i, d in enumerate(self.datas):
                dt, dn = self.datetime.date(), d._name
                pos = self.getposition(d).size
                print('{} {} Position {}'.format(dt, dn, pos))
    
                if not pos and not self.o.get(d, None):  # no market / no orders
                    if dt.weekday() == self.p.enter[i]:
                        if not self.p.usebracket:
                            self.o[d] = [self.buy(data=d)]
                            print('{} {} Buy {}'.format(dt, dn, self.o[d][0].ref))
    
                        else:
                            p = d.close[0] * (1.0 - self.p.pentry)
                            pstp = p * (1.0 - self.p.plimits)
                            plmt = p * (1.0 + self.p.plimits)
                            valid = datetime.timedelta(self.p.valid)
    
                            if self.p.rawbracket:
                                o1 = self.buy(data=d, exectype=bt.Order.Limit,
                                              price=p, valid=valid, transmit=False)
    
                                o2 = self.sell(data=d, exectype=bt.Order.Stop,
                                               price=pstp, size=o1.size,
                                               transmit=False, parent=o1)
    
                                o3 = self.sell(data=d, exectype=bt.Order.Limit,
                                               price=plmt, size=o1.size,
                                               transmit=True, parent=o1)
    
                                self.o[d] = [o1, o2, o3]
    
                            else:
                                self.o[d] = self.buy_bracket(
                                    data=d, price=p, stopprice=pstp,
                                    limitprice=plmt, oargs=dict(valid=valid))
    
                            print('{} {} Main {} Stp {} Lmt {}'.format(
                                dt, dn, *(x.ref for x in self.o[d])))
    
                        self.holding[d] = 0
    
                elif pos:  # exiting can also happen after a number of days
                    self.holding[d] += 1
                    if self.holding[d] >= self.p.hold[i]:
                        o = self.close(data=d)
                        self.o[d].append(o)  # manual order to list of orders
                        print('{} {} Manual Close {}'.format(dt, dn, o.ref))
                        if self.p.usebracket:
                            self.cancel(self.o[d][1])  # cancel stop side
                            print('{} {} Cancel {}'.format(dt, dn, self.o[d][1]))
    
    
    def runstrat(args=None):
        args = parse_args(args)
    
        cerebro = bt.Cerebro()
    
    
        # Data feed
        data0 = bt.feeds.YahooFinanceCSVData(dataname=args.data0)
        cerebro.adddata(data0, name='d0')
    
        data1 = bt.feeds.YahooFinanceCSVData(dataname=args.data1)
        data1.plotinfo.plotmaster = data0
        cerebro.adddata(data1, name='d1')
    
        data2 = bt.feeds.YahooFinanceCSVData(dataname=args.data2)
        data2.plotinfo.plotmaster = data0
        cerebro.adddata(data2, name='d2')
    
        # Broker
    
        cerebro.broker.setcommission(commission=0.001)
    
        # Sizer
        # cerebro.addsizer(bt.sizers.FixedSize, **eval('dict(' + args.sizer + ')'))
        cerebro.addsizer(Sizer)
    
        # Strategy
        cerebro.addstrategy(St)
    
        # Execute
        cerebro.run(runonce=False, writer=True)
    
        cerebro.plot()
    
    
    def parse_args(pargs=None):
        parser = argparse.ArgumentParser(
            formatter_class=argparse.ArgumentDefaultsHelpFormatter,
            description=(
                'Multiple Values and Brackets'
            )
        )
    
        parser.add_argument('--data0', default='C:/Users/cwse8/Desktop/New folder/NVDA.txt',
                            required=False, help='Data0 to read in')
    
        parser.add_argument('--data1', default='C:/Users/cwse8/Desktop/New folder/YHOO.txt',
                            required=False, help='Data1 to read in')
    
        parser.add_argument('--data2', default='C:/Users/cwse8/Desktop/New folder/ORCL.txt',
                            required=False, help='Data1 to read in')
    
        # Defaults for dates
        parser.add_argument('--fromdate', required=False, default='2001-01-01',
                            help='Date[time] in YYYY-MM-DD[THH:MM:SS] format')
    
        parser.add_argument('--todate', required=False, default='2007-01-01',
                            help='Date[time] in YYYY-MM-DD[THH:MM:SS] format')
    
        parser.add_argument('--cerebro', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--broker', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--sizer', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--strat', required=False, default='',
                            metavar='kwargs', help='kwargs in key=value format')
    
        parser.add_argument('--plot', required=False, default=True,
                            nargs='?', const='{}',
                            metavar='kwargs', help='kwargs in key=value format')
    
        return parser.parse_args(pargs)
    
    
    if __name__ == '__main__':
        runstrat()
    

    I downloaded the three files from Yahoo Fnance, eg: https://finance.yahoo.com/quote/ORCL/history?period1=510930000&period2=1492351200&interval=1d&filter=history&frequency=1d

    Heres the log output:

    2017-04-13 d0 Position 0
    2017-04-13 d1 Position 0
    2017-04-13 Data d1 OType buy Sizing to 1
    2017-04-13 d1 Main 1 Stp 2 Lmt 3
    2017-04-13 d2 Position 0
    2017-04-12 d1 Order 1 Status Accepted
    2017-04-12 d1 Order 2 Status Accepted
    2017-04-12 d1 Order 3 Status Accepted
    2017-04-12 d0 Position 0
    2017-04-12 d1 Position 0
    2017-04-12 d2 Position 0
    2017-04-11 d0 Position 0
    2017-04-11 Data d0 OType buy Sizing to 1
    2017-04-11 d0 Main 4 Stp 5 Lmt 6
    2017-04-11 d1 Position 0
    2017-04-11 d2 Position 0
    2017-04-10 d0 Order 4 Status Accepted
    2017-04-10 d0 Order 5 Status Accepted
    2017-04-10 d0 Order 6 Status Accepted
    2017-04-10 d0 Order 6 Status Completed
    -- No longer alive limit Ref
    2017-04-10 d0 Position -1
    2017-04-10 d1 Position 0
    2017-04-10 d2 Position 0
    2017-04-07 d1 Order 1 Status Completed
    -- No longer alive main Ref
    2017-04-07 d0 Position -1
    2017-04-07 d1 Position 1
    2017-04-07 d2 Position 0
    2017-04-07 Data d2 OType buy Sizing to 1
    2017-04-07 d2 Main 7 Stp 8 Lmt 9
    2017-04-06 d2 Order 7 Status Accepted
    2017-04-06 d2 Order 8 Status Accepted
    2017-04-06 d2 Order 9 Status Accepted
    2017-04-06 d0 Position -1
    2017-04-06 d1 Position 1
    2017-04-06 d2 Position 0
    2017-04-05 d2 Order 9 Status Completed
    -- No longer alive limit Ref
    2017-04-05 d0 Position -1
    2017-04-05 d1 Position 1
    2017-04-05 d2 Position -1
    2017-04-04 d0 Position -1
    2017-04-04 d1 Position 1
    2017-04-04 d2 Position -1
    2017-04-03 d0 Position -1
    2017-04-03 d1 Position 1
    2017-04-03 d2 Position -1
    2017-03-31 d0 Position -1
    Traceback (most recent call last):
      File "T:/Google Drive/PyCharm/Hello.py", line 189, in <module>
    2017-03-31 d0 Manual Close 10
    2017-03-31 d0 Cancel Ref: 5
    OrdType: 1
    OrdType: Sell
    Status: 5
    Status: Canceled
    Size: -1
    Price: 93.748754
    Price Limit: None
    TrailAmount: None
    TrailPercent: None
    ExecType: 3
    ExecType: Stop
    CommInfo: None
    End of Session: 736430.9999999999
    Info: AutoOrderedDict([('transmit', False), ('parent', <backtrader.order.BuyOrder object at 0x0000019342F95940>)])
    Broker: None
    Alive: False
    2017-03-31 d1 Position 1
    2017-03-31 d2 Position -1
    2017-03-30 d0 Order 5 Status Canceled
    -- No longer alive stop Ref
    2017-03-30 d0 Order 10 Status Accepted
    2017-03-30 d0 Position -1
    2017-03-30 d0 Manual Close 11
        runstrat()
      File "T:/Google Drive/PyCharm/Hello.py", line 140, in runstrat
        cerebro.run(runonce=False, writer=True)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 794, in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 924, in runstrategies
        self._runnext(runstrats)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1240, in _runnext
        strat._next()
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 296, in _next
        super(Strategy, self)._next()
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 252, in _next
        self.next()
      File "T:/Google Drive/PyCharm/Hello.py", line 106, in next
        self.cancel(self.o[d][1])  # cancel stop side
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 557, in cancel
        self.broker.cancel(order)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\brokers\bbroker.py", line 320, in cancel
        self.pending.remove(order)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__
        return self.ref == other.ref
    AttributeError: 'NoneType' object has no attribute 'ref'
    
    Process finished with exit code 1
    

  • administrators

    The output gives a very good indication as to what's wrong with the input. Trading backwards isn't yet supported.



  • @backtrader good point, but its your script!


  • administrators

    The script cannot know that the data will be fed in the wrong order. And the script (or the underlying platform for the sake of it) won't also fight it, because it makes not sense, to start with, to feed data in the wrong order. This is python and being it a dynamic language, with duck typing and a higher level of introspection something is always true: if you want to break it, you will break it.

    In any case the start of the output generated by the test run in the blog post:

    2001-01-02 d0 Position 0
    2001-01-02 Data d0 OType buy Sizing to 1
    2001-01-02 d0 Main 1 Stp 2 Lmt 3
    2001-01-02 d1 Position 0
    2001-01-02 d2 Position 0
    2001-01-03 d0 Order 1 Status Accepted
    2001-01-03 d0 Order 2 Status Accepted
    2001-01-03 d0 Order 3 Status Accepted
    2001-01-03 d0 Order 1 Status Completed
    ...
    

    Output in which the timestamps move forward.

    Your expectation may be that the platform fixes everything and then finds out that the data is in the wrong order. The platform could buffer the entire data stream, examine it, come to the conclusion that the order is wrong and then sort it. This would have several consequences:

    • Streams may originate for non-fixed length sources which may deliver in steps and it is not known it the stream will have an end (in practical terms it will always have an end)

    • Some people would complain about the extra time taken to do the check

    • Memory consumption would increase

    For streams originating from Yahoo which have not been reversed by the user, one can always apply the reverse parameter. Described in Docs - Data Feeds Reference. Set it to True.

    To simplify things, backtrader includes a yahoodownload.py tool which automatically does the job. The usage

    $ ./yahoodownload.py --help
    usage: yahoodownload.py [-h] --ticker TICKER [--notreverse]
                            [--timeframe TIMEFRAME] --fromdate FROMDATE --todate
                            TODATE --outfile OUTFILE
    
    Download Yahoo CSV Finance Data
    
    optional arguments:
      -h, --help            show this help message and exit
      --ticker TICKER       Ticker to be downloaded
      --notreverse          Do not reverse the downloaded files
      --timeframe TIMEFRAME
                            Timeframe: d -> day, w -> week, m -> month
      --fromdate FROMDATE   Starting date in YYYY-MM-DD format
      --todate TODATE       Ending date in YYYY-MM-DD format
      --outfile OUTFILE     Output file name
    


  • @backtrader, it may be more productive to work off the code I have written.
    You have seen this code before, but I have adapted it to your latest 'multi example' order management logic (full code & log copied at end)

    I now get errors when the margin order status occurs, I get the following error once the main order goes to margin:

      File "T:/Google Drive/PyCharm/Backtrader.py", line 205, in notify_order
        idx = dorders.index(order)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__
        return self.ref == other.ref
    AttributeError: 'NoneType' object has no attribute 'ref'
    

    This is how I have interpreted the sequence of events, please feel free to correct me:

    1. Bracket order created
    2. Main, stop and limit order Submitted and then Accepted
    3. Main order 'Margin' because not enough cash
    4. doreders[0] set to None (because Margin is not alive)
    5. Stop-loss tries to execute (see log where the order type SELL is printed immediately before the error)

    At this stage the stock holding would go short, however the order maangement logic then errors when trying to index(dorders).
    It appears to fall over when the index is matching the order.ref against the None object at the start of the dorders list (error log also shows this).

    If I can be of further assistance to resolve this issue I would be happy to help. I love your platform and I want to see this functionality running!

    FULL CODE:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import argparse
    import pandas as pd
    import numpy as np
    import datetime
    from scipy.stats import norm
    import math
    import backtrader as bt
    import backtrader.indicators as btind
    import backtrader.feeds as btfeeds
    import glob
    import ntpath
    
    
    def parse_args():
        parser = argparse.ArgumentParser(description='MultiData Strategy')
    
        parser.add_argument('--Custom_Alg',
                            default=False, # True OR False... NOT 'True' OR 'False'
                            help='True = Use custom alg')
    
        parser.add_argument('--SLTP_On',
                            default=True, # True OR False... NOT 'True' OR 'False'
                            help='True = Use Stop-Loss & Take-Profit Orders, False = do NOT use SL & TP orders')
    
        parser.add_argument('--stoploss',
                            action='store',
                            default=0.10, type=float,
                            help=('sell a long position if loss exceeds'))
    
        parser.add_argument('--takeprofit',
                            action='store',
                            default=2.00, type=float,
                            help=('Exit a long position if profit exceeds'))
    
        parser.add_argument('--data0', '-d0',
                            default=r'T:\PD_Stock_Data\DBLOAD',
                            help='Directory of CSV data source')
    
        parser.add_argument('--betaperiod',
                            default=1.4, type=float,
                            help='Per "Inside the Black Box" Mean. 1.4 also outperforms old model of 3months.')
    
        parser.add_argument('--fromdate', '-f',
                            default='2012-01-01',
                            help='Starting date in YYYY-MM-DD format')
    
        parser.add_argument('--todate', '-t',
                            default='2013-12-31',
                            help='Ending date in YYYY-MM-DD format')
    
        parser.add_argument('--limitpct',
                            action='store',
                            default=0.005, type=float,
                            help=('For buying at LIMIT, this will only purchase if the price is less than (1-limitpct)*Closing price'))
    
        parser.add_argument('--validdays',
                            action='store',
                            default=30, type=int,
                            help=('The number of days which a buy order remains valid'))
    
        parser.add_argument('--sellscore',
                            action='store',
                            default=-0.91, type=float,
                            help=('Max score for a sell'))
    
        parser.add_argument('--marketindex',
                            default='XJO',
                            help='XAO = All Ords, XJO = ASX200')
    
        parser.add_argument('--startingcash',
                            default=100000, type=int,
                            help='Starting Cash')
    
        parser.add_argument('--minholddays',
                            default=3, type=int,
                            help='Dont exit a market position until have held stock for at least this many days (excl. Stop-Loss and TP). May assist stopping exiting/cancelling orders when they are still being accepted by broker (i.e. day after entering mkt).')
    
        parser.add_argument('--pctperstock',
                            action='store', #0.083 = 1/12... i.e. a portfolio of up to 12 stocks
                            default=0.083, type=float, #i.e. 10% portfolio value in each stock
                            help=('Pct of portfolio starting cash to invest in each stock purchase'))
    
        parser.add_argument('--maxpctperstock',
                            action='store',
                            default=0.20, type=float,
                            help=('Max pct portfolio to invest in any porticular stock'))
    
        parser.add_argument('--mintrade',
                            default=1000, type=float,
                            help='Smallest dollar value to invest in a stock (if cash level below amount required for pctperstock)')
    
        parser.add_argument('--tradefee',
                            default=10.0, type=float,
                            help='CMC Markets Fee per stock trade (BUY OR SELL)')
    
        parser.add_argument('--alg_buyscore', #only used if Custom_Alg ==True
                            action='store',  # 0.91884558
                            default=0.91, type=float,
                            help=('Min score for a buy'))
    
        return parser.parse_args()
    
    #Excel sheet with ASX200 index constituents (used for chosing stocks to analyse in Backtrader)
    def LoadIndicies(Excel_Path, Excel_Sheet):
        # Load ASX200 Excel File
        ASX200 = pd.read_excel(Excel_Path, sheetname=Excel_Sheet)
        Index_Constituents = ASX200.to_dict(orient='list')
        for key, value in Index_Constituents.items():
            Index_Constituents[key] = [x for x in value if str(x) != 'nan']  # drop any "blank" (NaN) tickers from the index constituents table
        IndexDates = sorted(Index_Constituents.keys())
        IndexDates.append(datetime.datetime.now().strftime("%Y-%m-%d"))  # ordered list of the Index constituent Dates, with todays date at the end
        return Index_Constituents, IndexDates
    
    def LoadStockData(CSV_path=None):
        args = parse_args()
        if CSV_path is None:
            raise RuntimeError("no stock folder directory specifed.")
        allFiles = glob.glob(CSV_path + "/*.csv")
        Stocks = {}  # Create a DICTIONARY object to store the entire contents of all dataframes, allows for easy reference to / looping through dataframes by a string of their name, i.e. : 'CSL'
        for file_ in allFiles:
            name = ntpath.basename(file_[:-4])  # Set DF name = basename (not path) of the CSV.  [:-4] gets rid of the '.CSV' extention.
            Stocks[name] = pd.read_csv(file_, index_col='Date', parse_dates=True, header=0)
        return Stocks
    
    
    class StockLoader(btfeeds.PandasData):
        args = parse_args()
        params = (
            ('openinterest', None),     # None= column not present
            ('TOTAL_SCORE', -1))        # -1 = autodetect position or case-wise equal name
        if args.Custom_Alg == True:
            lines = ('TOTAL_SCORE',)
        if args.Custom_Alg == True:
            datafields = btfeeds.PandasData.datafields + (['TOTAL_SCORE'])
        else:
            datafields = btfeeds.PandasData.datafields
    
    class st(bt.Strategy):
        args = parse_args()
        params = ( #NB: self.p = self.params
            ('printlog', True),
        )
    
        def log(self, txt, dt=None, doprint=False):
            if self.p.printlog or doprint:
                dt = dt or self.datas[0].datetime.date(0)
                print('%s - %s' % (dt.isoformat(), txt))
    
        def __init__(self):
    
            self.o = {}  # orders per data (main, stop, limit, manual-close)
            self.holding = {}  # holding periods per data
    
            self.sma_short = {}
            self.sma_long = {}
            for i, d in enumerate(d for d in self.datas):
                self.sma_short[d] = bt.indicators.SimpleMovingAverage(d, period=42) #np.round(pd.rolling_mean(d.Close, window=42),2)
                self.sma_long[d] = bt.indicators.SimpleMovingAverage(d, period=252) #np.round(pd.rolling_mean(d.Close, window=252),2)
    
            # Plot Indicators if plot function called
            # bt.indicators.ExponentialMovingAverage(self.datas[0], period=25)
            # bt.indicators.WeightedMovingAverage(self.datas[0], period=25, subplot=True)
            # bt.indicators.StochasticSlow(self.datas[0])
            # bt.indicators.MACDHisto(self.datas[0])
            # rsi = bt.indicators.RSI(self.datas[0])
            # bt.indicators.SmoothedMovingAverage(rsi, period=10)
            # bt.indicators.ATR(self.datas[0], plot=False)
    
        def notify_trade(self, trade): #NB: "print(Trade.__dict__)"
            if trade.isclosed: #Market Position exited
                self.log('OPERATION PROFIT: %s, Gross: %2f, Net: %2f' %(trade.data._name,
                                                                        trade.pnl,
                                                                        trade.pnlcomm))
    
        def notify_order(self, order):
            '''Notes:
            SUBMITTED: Marks an order as submitted and stores the broker to which it was submitted.
            ACCEPTED: Marks an order as accepted
            COMPLETED: Order executed, completely filled.
            MARGIN: Not enough cash to execute the order
            REJECTED: broker could reject order if not enough cash
            CANCELLED: Marks an order as cancelled. Status.canceled occurs when: a "self.broker.cancel()" has occured by user
            #NB: "print(order.executed.__dict__)"
            '''
    
            if order.status in [order.Submitted, order.Accepted]:
                return #do nothing
    
            #if order.status ==order.Margin:
                #self.cancel(self.o[order.data][1]) # cancel stop-loss
                #self.cancel(self.o[order.data][2]) # cancel take-profit
    
    
            if order.isbuy():
                buysell = 'BUY'
            elif order.issell():
                buysell = 'SELL'
                print('{} {}'.format(order.data._name, order.getstatusname()))
            print(buysell)
    
            #Order Type identification
            whichord = ['main','stop','limit','close']
            dorders = self.o[order.data]
            print(dorders)
            idx = dorders.index(order)
    
            self.log('%s %s: %s, Type: %s, Ref: %s, Price: %.2f, Cost: %.2f, Size: %.2f, Comm %.2f' %(buysell,
                                                                                                      order.getstatusname(),
                                                                                                      order.data._name,
                                                                                                      whichord[idx], #Order Type (main, stop, limit or close)
                                                                                                      order.ref,
                                                                                                      order.executed.price,
                                                                                                      order.executed.value,
                                                                                                      order.executed.size,
                                                                                                      order.executed.comm))
    
            if not order.alive():# indicate no order is pending, allows new orders. Alive = if order is in status Partial or Accepted
                dorders[idx] = None #nullify the specific order (main, stop, limit or close)
                print('-- No longer alive {} Ref'.format(whichord[idx]))
                if all(x is None for x in dorders):
                    dorders[:] = []  # empty list - New orders allowed
    
        #def prenext(self): #overrides PRENEXT() so that the "NEXT()" calculations runs regardless of when each stock data date range starts.
            #self.next()
    
        def next(self):
            today = self.datetime.date(0)
            weekday = today.isoweekday() #Monday = 1, Sunday = 7
            if weekday in range(1,8): # analyse on all weekdays (MONDAY to SUNDAY)
    
                num_long = 0 #number long stocks
                #IdealLongPortf = pd.DataFrame(columns=('Stock', 'Score','Close','Current Position', 'Ideal Position', 'Pos Delta Value', 'Go NoGo')) #ideal stock positions at end of each next() iteration
    
                for i, d in enumerate(d for d in self.datas):  # Loop through Universe of Stocks. "If Len(d)" is used to check that all datafeeds have delivered values. as if using minute data, some may have had many minutes, 500, and another may not have 1 record yet (if its still on daily)
                    if d._name != args.marketindex:
                        position = self.broker.getposition(d)
                        positiondol = float(self.broker.getposition(d).size*d.close[0])
                        cash = self.broker.getcash() #total available cash
    
                        if not position.size\
                            and not self.o.get(d, None)\
                            and d.close[0] > 0 \
                            and self.sma_short[d][0] > self.sma_long[d][0]\
                            and cash > args.mintrade:
                            #and d.lines.TOTAL_SCORE[0] >= args.alg_buyscore:
    
                            #IdealLongPortf.append([d._name, d.lines.TOTAL_SCORE[0], d.close[0], position.size, np.NaN, np.NaN,np.NaN])
                            buylimit = d.close[0]*(1.0-args.limitpct)
    
                            if args.SLTP_On == True:
                                stop_loss = d.close[0]*(1.0 - args.stoploss)
                                take_profit = d.close[0]*(1.0 + args.takeprofit)
    
                                o1 = self.buy(data = d,
                                              exectype=bt.Order.Limit,
                                              price=buylimit,
                                              valid=today + datetime.timedelta(days=args.validdays),
                                              transmit=False)
    
                                o2 = self.sell(data = d,
                                               size = o1.size,         # could be an issue with re-balancing!!!
                                               exectype=bt.Order.Stop,
                                               price=stop_loss,
                                               parent=o1,
                                               transmit=False)
    
                                o3 = self.sell(data = d,
                                               size = o1.size,
                                               exectype=bt.Order.Limit,
                                               price=take_profit,
                                               parent=o1,
                                               transmit=True)
    
                                self.o[d] = [o1, o2, o3]
    
                                self.log('CREATE BUY: %s, Main: %2f, Stop: %2f, Limit: %2f, Close: %2f, Score: %2f' %(d._name,
                                                                                                                      buylimit,
                                                                                                                      stop_loss,
                                                                                                                      take_profit,
                                                                                                                      d.close[0],
                                                                                                                      1)) #d.lines.TOTAL_SCORE[0]))
    
                            else:
                                o1 = self.buy(data = d,
                                              exectype=bt.Order.Limit,
                                              price=buylimit,
                                              valid=today + datetime.timedelta(days=args.validdays))
                                self.log('CREATE BUY: %s, Close: %2f, Buy @: %2f, Score: %2f' %(d._name,
                                                                                                          d.close[0],
                                                                                                          buylimit,
                                                                                                          1)) #d.lines.TOTAL_SCORE[0]))
                                self.o[d] = [o1]
    
                            self.holding[d] = 0
    
                        elif position.size: # Currently LONG
                            self.holding[d] += 1
                            num_long += 1
                            self.log('Stock Held: %s, Close: %2f, Posn: %i, Posn($): %2f, Days Held: %i, Score: %2f, Score Yest: %2f' %(d._name,
                                                                                                                                       d.close[0],
                                                                                                                                       position.size,
                                                                                                                                       positiondol,
                                                                                                                                       self.holding[d],
                                                                                                                                       1, #d.lines.TOTAL_SCORE[0],
                                                                                                                                       1)) #d.lines.TOTAL_SCORE[-1]))
    
    
                            if position.size > 0\
                                and not self.o.get(d, None) \
                                and self.holding[d] >= args.minholddays \
                                and self.sma_short[d][0] < self.sma_long[d][0]:
                                #and d.lines.TOTAL_SCORE[0] < args.alg_buyscore:
    
                                self.log('CLOSING LONG POSITION: %s, Close: %2f, Score: %2f' %(d._name,
                                                                                                d.close[0],
                                                                                             1)) #d.lines.TOTAL_SCORE[0]))
    
                                if self.o[d][1]:
                                    self.cancel(self.o[d][1]) # cancel stop side, this automatically cancels the TP too
                                    self.log('CANCELLING SL & TP for: %s' %(d._name))
    
                                o = self.close(data=d)
                                self.o[d].append(o)  # manual order to list of orders
    
                            elif position.size < 0:
                                print('WTFMATE:{}, size:{}, Value$: {}, daysheld: ()'.format(d._name, position.size, positiondol))
    
                totalwealth = self.broker.getvalue()
                cash = self.broker.getcash()
                invested = totalwealth - cash
    
                self.log("Stocks Held: %s, Total Wealth: %i, Invested: %i, Cash-On-Hand: %i" %(str(num_long),
                                                                                               totalwealth,
                                                                                               invested,
                                                                                               cash))
    
        def stop(self):
            pass
    
    class PortfolioSizer(bt.Sizer):
        def _getsizing(self, comminfo, cash, data, isbuy):
            args = parse_args()
            position = self.broker.getposition(data)
            price = data.close[0]
            investment = args.startingcash * args.pctperstock
            if cash < investment:
                investment = max(cash,args.mintrade) # i.e. never invest less than the "mintrade" $value
            qty = math.floor(investment/price)
    
            # This method returns the desired size for the buy/sell operation
            if isbuy:  # if buying
                if position.size < 0:  # if currently short, buy the amount which are short to close out trade.
                     return -position.size
                elif position.size > 0:
                    return 0  # dont buy if already hold
                else:
                    return qty  # num. stocks to LONG
    
            if not isbuy:  # if selling..
                if position.size < 0:
                    return 0  # dont sell if already SHORT
                elif position.size > 0:
                    return position.size  # currently Long... sell what hold
                else:
                    return qty  # num. stocks to SHORT
    
    
    def RunStrategy():
        args = parse_args()
        cerebro = bt.Cerebro()
        cerebro.addstrategy(st)
        # strats = cerebro.optstrategy(st,maperiod=range(10, 31))
    
        #date range to backtest
        tradingdates = Stocks[args.marketindex].loc[
            (Stocks[args.marketindex].index>=datetime.datetime.strptime(args.fromdate, "%Y-%m-%d")) &
            (Stocks[args.marketindex].index<datetime.datetime.strptime(args.todate, "%Y-%m-%d"))
        ]
    
        #Load 200 stocks into Backtrader (specified in the Index_constituents list)
        for ticker in Index_Constituents[IndexDates[3]]:
            datarange = Stocks[ticker].loc[
                (Stocks[ticker].index>=datetime.datetime.strptime(args.fromdate, "%Y-%m-%d")) &
                (Stocks[ticker].index<datetime.datetime.strptime(args.todate, "%Y-%m-%d"))
            ]
            #REINDEX to make sure the stock has the exact same trading days as the MARKET INDEX. Reindex ffill doesn't fill GAPS. Therefore also apply FILLNA
            datarange.reindex(tradingdates.index, method='ffill').fillna(method='ffill',inplace=True)
            data = StockLoader(dataname=datarange)
            data.plotinfo.plot=False
            cerebro.adddata(data, name=ticker)
    
        data = btfeeds.PandasData(dataname=tradingdates, openinterest=None) #load market index (for date referencing)
        cerebro.adddata(data, name=args.marketindex)
    
        #cerebro.addanalyzer(CurrentBuysAnalyzer, )
        cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, ) #length of holds etc
        cerebro.addanalyzer(bt.analyzers.TimeReturn, timeframe=bt.TimeFrame.Years)
        cerebro.addanalyzer(bt.analyzers.SharpeRatio, timeframe=bt.TimeFrame.Years, riskfreerate=0.03, annualize=True)
        cerebro.addanalyzer(bt.analyzers.SQN)
        cerebro.addanalyzer(bt.analyzers.DrawDown, )
    
        cerebro.broker.setcash(args.startingcash) # set starting cash
        cerebro.addsizer(PortfolioSizer)
    
        commission = float(args.tradefee/(args.pctperstock*args.startingcash))
        print("The Commission rate is: %0.5f" % (commission))
        cerebro.broker.setcommission(commission=commission)
    
        cerebro.run(runonce=False, writer=True)
        cerebro.plot(volume=False, stdstats=False)
        '''
        zdown=True: Rotation of the date labes on the x axis
        stdstats=False: Disable the standard plotted observers
        numfigs=1: Plot on one chart
        '''
    
    if __name__ == '__main__':
        # if python is running this script (module) as the main program, then __name__ == __main__, and this block of code will run.
        #  However, if another script (module) is IMPORTING this script (module), this block of code WILL NOT RUN, but the above functions can be called.
        args = parse_args()
        t1 = datetime.datetime.now()
        print('Processing Commenced: {}'.format(str(t1)))
    
        #Load stocks from local drive for analysis
        Stocks = LoadStockData(args.data0)
        t2 = datetime.datetime.now()
    
        # Dictionary of Index Constituents (and their stock dataframes)
        Index_Constituents, IndexDates = LoadIndicies(
            Excel_Path='T:\Google Drive\Capriole\CAPRIOLEPROCESSOR\TickerUpdater.xlsm',
            Excel_Sheet='ASX200_Const_Updated')
        print('ASX200 constituent list date: {}'.format(IndexDates[3]))
    
        if args.Custom_Alg == True:
            initiate()
            t3 = datetime.datetime.now()
    
        RunStrategy()
        t4 = datetime.datetime.now()
    
        #TIMER
        print('Run-time - TOTAL: {0}'.format(datetime.datetime.now() - t1))
        print('Run-time - Load Data: {0}'.format(t2 - t1))
        if 't3' in locals():
            print('Run-time - Algorithm: {0}'.format(t3 - t2))
            print('Run-time - Strategy Back-test: {0}'.format(t4 - t3))
        else:
            print('Run-time - Strategy Back-test: {0}'.format(t4 - t2))
    
    
    
        #Current_Buys() #...if using to BUY today!
    
    
    
    

    LOG (could only keep key sections due to message limit. All references to EWC maintained. Note the "SELL" word only prints once and in relation to EWC):

    2013-11-25 - CREATE BUY: GWA, Main: 3.048692, Stop: 2.757611, Limit: 9.192036, Close: 3.064012, Score: 1.000000
    2013-11-25 - CREATE BUY: DL_MTU, Main: 6.059550, Stop: 5.481000, Limit: 18.270000, Close: 6.090000, Score: 1.000000
    2013-11-25 - CREATE BUY: EWC, Main: 0.398000, Stop: 0.360000, Limit: 1.200000, Close: 0.400000, Score: 1.000000
    2013-11-25 - CREATE BUY: DL_SKE, Main: 3.402900, Stop: 3.078000, Limit: 10.260000, Close: 3.420000, Score: 1.000000
    
    
    ...
    
    
    
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C16C0F0>, <backtrader.order.SellOrder object at 0x000002204C1B39E8>, <backtrader.order.SellOrder object at 0x000002204C1B3B38>]
    2013-11-26 - BUY Margin: DL_MTU, Type: main, Ref: 319, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C1965F8>, <backtrader.order.SellOrder object at 0x000002204C42EFD0>, <backtrader.order.SellOrder object at 0x000002204C42E0F0>]
    2013-11-26 - BUY Margin: EWC, Type: main, Ref: 322, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C1B37B8>, <backtrader.order.SellOrder object at 0x000002204C1632E8>, <backtrader.order.SellOrder object at 0x000002204C1634E0>]
    2013-11-26 - BUY Margin: DL_SKE, Type: main, Ref: 325, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C16AA58>, <backtrader.order.SellOrder object at 0x000002204C1B9240>, <backtrader.order.SellOrder object at 0x000002204C1B9048>]
    
    
    ...
    
    
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C16FAC8>, <backtrader.order.SellOrder object at 0x000002204C16A828>, <backtrader.order.SellOrder object at 0x000002204C16A278>]
    2013-11-27 - BUY Margin: SAI, Type: main, Ref: 292, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C16A7F0>, <backtrader.order.SellOrder object at 0x000002204C13D550>, <backtrader.order.SellOrder object at 0x000002204C154908>]
    2013-11-27 - BUY Margin: BDR, Type: main, Ref: 307, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    BUY
    [<backtrader.order.BuyOrder object at 0x000002204C154F28>, <backtrader.order.SellOrder object at 0x000002204C154A58>, <backtrader.order.SellOrder object at 0x000002204C154F98>]
    2013-11-27 - BUY Margin: SXL, Type: main, Ref: 310, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00
    -- No longer alive main Ref
    EWC Completed
    SELL
    [None, <backtrader.order.SellOrder object at 0x000002204C42EFD0>, <backtrader.order.SellOrder object at 0x000002204C42E0F0>]
    Traceback (most recent call last):
      File "T:/Google Drive/PyCharm/Backtrader.py", line 784, in <module>
        RunStrategy()
      File "T:/Google Drive/PyCharm/Backtrader.py", line 755, in RunStrategy
        cerebro.run(runonce=False, writer=True)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 794, in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 924, in runstrategies
        self._runnext(runstrats)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1240, in _runnext
        strat._next()
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 296, in _next
        super(Strategy, self)._next()
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 245, in _next
        self._notify()
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 499, in _notify
        self.notify_order(order)
      File "T:/Google Drive/PyCharm/Backtrader.py", line 211, in notify_order
        idx = dorders.index(order)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__
        return self.ref == other.ref
    AttributeError: 'NoneType' object has no attribute 'ref'
    
    Process finished with exit code 1
    
    
    

    Thank you,
    CWE


  • administrators

    This is unfortunately the typical case of a non-bug report. backtrader may be full of bugs, but when reporting one, there are some minimums:

    • Version
    • Some input data
    • A simple use case in the code

    Your report is a moving target.

    • In your Get Value thread it was set in question how the platform calculated the value

    • The diagnostic that you were entering short positions was not accepted because no short positions were entered because they were not shown in the log (which was only printing positive positions)

    • After some left and right, that thread is abandoned and a quick mention above shows that your code was entering short positions

    • The next thing is reporting that the sample in the code in the blog post produces a bug

    • The new sample data passed to the (modified) code is in the wrong order (timewise) which obviously produces errors and the answer is: "it's your script"

    • Instead of correcting the new sample data the answer is to throw in again a monster (in size) code and ask again for debugging.

    • Again, things like version, input data and simplicity are missing.

    Is the sample from the blog post above (re)producing any bug? Because that was stated, but has not yet been confirmed.

    In the meantime, your error.

     File "T:/Google Drive/PyCharm/Backtrader.py", line 205, in notify_order
        idx = dorders.index(order)
      File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__
        return self.ref == other.ref
    AttributeError: 'NoneType' object has no attribute 'ref'
    

    To support that comparison the following commit was added 16 days ago:

    Bracket order support was added after that with this other commit:

    And announced even later with this other commit tagged as version 1.9.37.116:

    The original Blog - Bracket orders post announcing bracket order support says:

    Release 1.9.37.116 adds bracket orders giving a very broad spectrum of orders which are supported by the backtesting broker (Market, Limit, Close, Stop, StopLimit, StopTrail, StopTrailLimit, OCO)

    It is therefore really difficult to imagine that you are using any version of backtrader which supports bracket orders.



  • Hi @backtrader,

    Thank you for this detailed response.

    My humble apologies for how this issue has been approached.
    If I ever have a bug report in the future I will be sure to include that information.

    As it turns out, you are quite right... I stuffed up on the version front and wasn't using the lastest. I have since upgraded and the script runs completely! Wish I knew this yesterday before wasting several hours trying to debug a problem which didnt exist (my mistake!)!!

    I assure you I haven't abandoned the getvalue thread, I am still working on addressing your suggested solutions and will get back to you to confirm if I have resolved the issue regarding short positions as soon as I have time to do some more thorough testing.

    As a suggestion, with those example codes, would it be possible to include all elements for users such as myself to do a full run without edits? Perhaps inclusion of the necessary files etc too would be valuable and save some time. I spent some time yesterday to try and get your blog post to run and didn't consider that I would have to incorporate additional code to re-order data etc. I understand that it would never have worked for me though given my version issues..! But this may be useful for others going forward.

    Thank you for your time and patience in assisting me as I learn this platform and process.

    CWE


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.