Multi Example
-
Use the link below to go the original post
Click here to see the full blog post
-
Thank you for this valuable example!
-
Thank you, this is a great example.
I have a question regarding "Margin" order statuses here.
What if your logic results in an order placement when there isn't enough cash? I have this when I run similar logic across many more stocks.The bracket causes the creation of the stop-loss and take-profit, even though the original 'main' order could not execute... so eventually you end up in short positions when the bracket orders execute at their stop/limit prices. How should this be managed?
I tried doing order.cancels when a order.Margin status occurs, however sometimes the stoploss/takeprofit executes that day of creation, and the cancellation doesnt occur in time.
-
@cwse said in Multi Example:
The bracket causes the creation of the stop-loss and take-profit, even though the original 'main' order could not execute... so eventually you end up in short positions when the bracket orders execute at their stop/limit prices. How should this be managed?
The
stop-lossandtake-profitorders are created but-
Are canceled if the parent order cannot be accepted or is canceled
-
Are created inactive and will only become active (available for execution) if the parent is Completed
This is actually acts as a safeguard which prevents the execution of stop-loss or take-profit
sellorders even if they are accidentally accepted.
If any of those 2 things is not working as described there for the implemented bracket order functionality, it would be a bug.
If the orders are issued manually with no relationship (parent/children or through the
buy_bracket/sell_bracket), there is no way to avoid execution of one order, because the other was canceled. -
-
@backtrader, then it appears there is a bug..
-
A use case to reproduce the bug is very much appreciated.
-
@backtrader, exactly per my script provided in this post: "How is Getvalue() Calculated?".
I added a print when position goes negative as you suggested,and it only occurs after a bracket was submitted, the 'buy' goes to margin and then the stop-loss and take-profit subsequently (undesirably) execute.
Alternatively, you may get the same functionality when you run this
multiexample with higher stakes (such that many executions will result in a margin) or with many more stocks.Thanks,
CWE -
If you have run the
multisample and produced the bug, may you share the execution command? -
Just tried running your code per below (minor edits under the
def runstrat()sections so it would run:from __future__ import (absolute_import, division, print_function, unicode_literals) import argparse import datetime import backtrader as bt class Sizer(bt.Sizer): params = dict(stake=1) def _getsizing(self, comminfo, cash, data, isbuy): dt, i = self.strategy.datetime.date(), data._id s = self.p.stake * (1 + (not isbuy)) print('{} Data {} OType {} Sizing to {}'.format( dt, data._name, ('buy' * isbuy) or 'sell', s)) return s class St(bt.Strategy): params = dict( enter=[1, 3, 4], # data ids are 1 based hold=[7, 10, 15], # data ids are 1 based usebracket=True, rawbracket=True, pentry=0.015, plimits=0.03, valid=10, ) def notify_order(self, order): if order.status == order.Submitted: return dt, dn = self.datetime.date(), order.data._name print('{} {} Order {} Status {}'.format( dt, dn, order.ref, order.getstatusname()) ) whichord = ['main', 'stop', 'limit', 'close'] if not order.alive(): # not alive - nullify dorders = self.o[order.data] idx = dorders.index(order) dorders[idx] = None print('-- No longer alive {} Ref'.format(whichord[idx])) if all(x is None for x in dorders): dorders[:] = [] # empty list - New orders allowed def __init__(self): self.o = dict() # orders per data (main, stop, limit, manual-close) self.holding = dict() # holding periods per data def next(self): for i, d in enumerate(self.datas): dt, dn = self.datetime.date(), d._name pos = self.getposition(d).size print('{} {} Position {}'.format(dt, dn, pos)) if not pos and not self.o.get(d, None): # no market / no orders if dt.weekday() == self.p.enter[i]: if not self.p.usebracket: self.o[d] = [self.buy(data=d)] print('{} {} Buy {}'.format(dt, dn, self.o[d][0].ref)) else: p = d.close[0] * (1.0 - self.p.pentry) pstp = p * (1.0 - self.p.plimits) plmt = p * (1.0 + self.p.plimits) valid = datetime.timedelta(self.p.valid) if self.p.rawbracket: o1 = self.buy(data=d, exectype=bt.Order.Limit, price=p, valid=valid, transmit=False) o2 = self.sell(data=d, exectype=bt.Order.Stop, price=pstp, size=o1.size, transmit=False, parent=o1) o3 = self.sell(data=d, exectype=bt.Order.Limit, price=plmt, size=o1.size, transmit=True, parent=o1) self.o[d] = [o1, o2, o3] else: self.o[d] = self.buy_bracket( data=d, price=p, stopprice=pstp, limitprice=plmt, oargs=dict(valid=valid)) print('{} {} Main {} Stp {} Lmt {}'.format( dt, dn, *(x.ref for x in self.o[d]))) self.holding[d] = 0 elif pos: # exiting can also happen after a number of days self.holding[d] += 1 if self.holding[d] >= self.p.hold[i]: o = self.close(data=d) self.o[d].append(o) # manual order to list of orders print('{} {} Manual Close {}'.format(dt, dn, o.ref)) if self.p.usebracket: self.cancel(self.o[d][1]) # cancel stop side print('{} {} Cancel {}'.format(dt, dn, self.o[d][1])) def runstrat(args=None): args = parse_args(args) cerebro = bt.Cerebro() # Data feed data0 = bt.feeds.YahooFinanceCSVData(dataname=args.data0) cerebro.adddata(data0, name='d0') data1 = bt.feeds.YahooFinanceCSVData(dataname=args.data1) data1.plotinfo.plotmaster = data0 cerebro.adddata(data1, name='d1') data2 = bt.feeds.YahooFinanceCSVData(dataname=args.data2) data2.plotinfo.plotmaster = data0 cerebro.adddata(data2, name='d2') # Broker cerebro.broker.setcommission(commission=0.001) # Sizer # cerebro.addsizer(bt.sizers.FixedSize, **eval('dict(' + args.sizer + ')')) cerebro.addsizer(Sizer) # Strategy cerebro.addstrategy(St) # Execute cerebro.run(runonce=False, writer=True) cerebro.plot() def parse_args(pargs=None): parser = argparse.ArgumentParser( formatter_class=argparse.ArgumentDefaultsHelpFormatter, description=( 'Multiple Values and Brackets' ) ) parser.add_argument('--data0', default='C:/Users/cwse8/Desktop/New folder/NVDA.txt', required=False, help='Data0 to read in') parser.add_argument('--data1', default='C:/Users/cwse8/Desktop/New folder/YHOO.txt', required=False, help='Data1 to read in') parser.add_argument('--data2', default='C:/Users/cwse8/Desktop/New folder/ORCL.txt', required=False, help='Data1 to read in') # Defaults for dates parser.add_argument('--fromdate', required=False, default='2001-01-01', help='Date[time] in YYYY-MM-DD[THH:MM:SS] format') parser.add_argument('--todate', required=False, default='2007-01-01', help='Date[time] in YYYY-MM-DD[THH:MM:SS] format') parser.add_argument('--cerebro', required=False, default='', metavar='kwargs', help='kwargs in key=value format') parser.add_argument('--broker', required=False, default='', metavar='kwargs', help='kwargs in key=value format') parser.add_argument('--sizer', required=False, default='', metavar='kwargs', help='kwargs in key=value format') parser.add_argument('--strat', required=False, default='', metavar='kwargs', help='kwargs in key=value format') parser.add_argument('--plot', required=False, default=True, nargs='?', const='{}', metavar='kwargs', help='kwargs in key=value format') return parser.parse_args(pargs) if __name__ == '__main__': runstrat()I downloaded the three files from Yahoo Fnance, eg: https://finance.yahoo.com/quote/ORCL/history?period1=510930000&period2=1492351200&interval=1d&filter=history&frequency=1d
Heres the log output:
2017-04-13 d0 Position 0 2017-04-13 d1 Position 0 2017-04-13 Data d1 OType buy Sizing to 1 2017-04-13 d1 Main 1 Stp 2 Lmt 3 2017-04-13 d2 Position 0 2017-04-12 d1 Order 1 Status Accepted 2017-04-12 d1 Order 2 Status Accepted 2017-04-12 d1 Order 3 Status Accepted 2017-04-12 d0 Position 0 2017-04-12 d1 Position 0 2017-04-12 d2 Position 0 2017-04-11 d0 Position 0 2017-04-11 Data d0 OType buy Sizing to 1 2017-04-11 d0 Main 4 Stp 5 Lmt 6 2017-04-11 d1 Position 0 2017-04-11 d2 Position 0 2017-04-10 d0 Order 4 Status Accepted 2017-04-10 d0 Order 5 Status Accepted 2017-04-10 d0 Order 6 Status Accepted 2017-04-10 d0 Order 6 Status Completed -- No longer alive limit Ref 2017-04-10 d0 Position -1 2017-04-10 d1 Position 0 2017-04-10 d2 Position 0 2017-04-07 d1 Order 1 Status Completed -- No longer alive main Ref 2017-04-07 d0 Position -1 2017-04-07 d1 Position 1 2017-04-07 d2 Position 0 2017-04-07 Data d2 OType buy Sizing to 1 2017-04-07 d2 Main 7 Stp 8 Lmt 9 2017-04-06 d2 Order 7 Status Accepted 2017-04-06 d2 Order 8 Status Accepted 2017-04-06 d2 Order 9 Status Accepted 2017-04-06 d0 Position -1 2017-04-06 d1 Position 1 2017-04-06 d2 Position 0 2017-04-05 d2 Order 9 Status Completed -- No longer alive limit Ref 2017-04-05 d0 Position -1 2017-04-05 d1 Position 1 2017-04-05 d2 Position -1 2017-04-04 d0 Position -1 2017-04-04 d1 Position 1 2017-04-04 d2 Position -1 2017-04-03 d0 Position -1 2017-04-03 d1 Position 1 2017-04-03 d2 Position -1 2017-03-31 d0 Position -1 Traceback (most recent call last): File "T:/Google Drive/PyCharm/Hello.py", line 189, in <module> 2017-03-31 d0 Manual Close 10 2017-03-31 d0 Cancel Ref: 5 OrdType: 1 OrdType: Sell Status: 5 Status: Canceled Size: -1 Price: 93.748754 Price Limit: None TrailAmount: None TrailPercent: None ExecType: 3 ExecType: Stop CommInfo: None End of Session: 736430.9999999999 Info: AutoOrderedDict([('transmit', False), ('parent', <backtrader.order.BuyOrder object at 0x0000019342F95940>)]) Broker: None Alive: False 2017-03-31 d1 Position 1 2017-03-31 d2 Position -1 2017-03-30 d0 Order 5 Status Canceled -- No longer alive stop Ref 2017-03-30 d0 Order 10 Status Accepted 2017-03-30 d0 Position -1 2017-03-30 d0 Manual Close 11 runstrat() File "T:/Google Drive/PyCharm/Hello.py", line 140, in runstrat cerebro.run(runonce=False, writer=True) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 794, in run runstrat = self.runstrategies(iterstrat) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 924, in runstrategies self._runnext(runstrats) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1240, in _runnext strat._next() File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 296, in _next super(Strategy, self)._next() File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 252, in _next self.next() File "T:/Google Drive/PyCharm/Hello.py", line 106, in next self.cancel(self.o[d][1]) # cancel stop side File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 557, in cancel self.broker.cancel(order) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\brokers\bbroker.py", line 320, in cancel self.pending.remove(order) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__ return self.ref == other.ref AttributeError: 'NoneType' object has no attribute 'ref' Process finished with exit code 1 -
The output gives a very good indication as to what's wrong with the input. Trading backwards isn't yet supported.
-
@backtrader good point, but its your script!
-
The script cannot know that the data will be fed in the wrong order. And the script (or the underlying platform for the sake of it) won't also fight it, because it makes not sense, to start with, to feed data in the wrong order. This is python and being it a dynamic language, with duck typing and a higher level of introspection something is always true: if you want to break it, you will break it.
In any case the start of the output generated by the test run in the blog post:
2001-01-02 d0 Position 0 2001-01-02 Data d0 OType buy Sizing to 1 2001-01-02 d0 Main 1 Stp 2 Lmt 3 2001-01-02 d1 Position 0 2001-01-02 d2 Position 0 2001-01-03 d0 Order 1 Status Accepted 2001-01-03 d0 Order 2 Status Accepted 2001-01-03 d0 Order 3 Status Accepted 2001-01-03 d0 Order 1 Status Completed ...Output in which the timestamps move forward.
Your expectation may be that the platform fixes everything and then finds out that the data is in the wrong order. The platform could buffer the entire data stream, examine it, come to the conclusion that the order is wrong and then sort it. This would have several consequences:
-
Streams may originate for non-fixed length sources which may deliver in steps and it is not known it the stream will have an end (in practical terms it will always have an end)
-
Some people would complain about the extra time taken to do the check
-
Memory consumption would increase
For streams originating from
Yahoowhich have not been reversed by the user, one can always apply thereverseparameter. Described in Docs - Data Feeds Reference. Set it toTrue.To simplify things, backtrader includes a
yahoodownload.pytool which automatically does the job. The usage$ ./yahoodownload.py --help usage: yahoodownload.py [-h] --ticker TICKER [--notreverse] [--timeframe TIMEFRAME] --fromdate FROMDATE --todate TODATE --outfile OUTFILE Download Yahoo CSV Finance Data optional arguments: -h, --help show this help message and exit --ticker TICKER Ticker to be downloaded --notreverse Do not reverse the downloaded files --timeframe TIMEFRAME Timeframe: d -> day, w -> week, m -> month --fromdate FROMDATE Starting date in YYYY-MM-DD format --todate TODATE Ending date in YYYY-MM-DD format --outfile OUTFILE Output file name -
-
@backtrader, it may be more productive to work off the code I have written.
You have seen this code before, but I have adapted it to your latest 'multi example' order management logic (full code & log copied at end)I now get errors when the margin order status occurs, I get the following error once the
mainorder goes tomargin:File "T:/Google Drive/PyCharm/Backtrader.py", line 205, in notify_order idx = dorders.index(order) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__ return self.ref == other.ref AttributeError: 'NoneType' object has no attribute 'ref'This is how I have interpreted the sequence of events, please feel free to correct me:
- Bracket order created
- Main, stop and limit order Submitted and then Accepted
- Main order 'Margin' because not enough cash
- doreders[0] set to
None(because Margin is notalive) - Stop-loss tries to execute (see log where the order type
SELLis printed immediately before the error)
At this stage the stock holding would go short, however the order maangement logic then errors when trying to
index(dorders).
It appears to fall over when the index is matching theorder.refagainst theNoneobject at the start of thedorderslist (error log also shows this).If I can be of further assistance to resolve this issue I would be happy to help. I love your platform and I want to see this functionality running!
FULL CODE:
from __future__ import (absolute_import, division, print_function, unicode_literals) import argparse import pandas as pd import numpy as np import datetime from scipy.stats import norm import math import backtrader as bt import backtrader.indicators as btind import backtrader.feeds as btfeeds import glob import ntpath def parse_args(): parser = argparse.ArgumentParser(description='MultiData Strategy') parser.add_argument('--Custom_Alg', default=False, # True OR False... NOT 'True' OR 'False' help='True = Use custom alg') parser.add_argument('--SLTP_On', default=True, # True OR False... NOT 'True' OR 'False' help='True = Use Stop-Loss & Take-Profit Orders, False = do NOT use SL & TP orders') parser.add_argument('--stoploss', action='store', default=0.10, type=float, help=('sell a long position if loss exceeds')) parser.add_argument('--takeprofit', action='store', default=2.00, type=float, help=('Exit a long position if profit exceeds')) parser.add_argument('--data0', '-d0', default=r'T:\PD_Stock_Data\DBLOAD', help='Directory of CSV data source') parser.add_argument('--betaperiod', default=1.4, type=float, help='Per "Inside the Black Box" Mean. 1.4 also outperforms old model of 3months.') parser.add_argument('--fromdate', '-f', default='2012-01-01', help='Starting date in YYYY-MM-DD format') parser.add_argument('--todate', '-t', default='2013-12-31', help='Ending date in YYYY-MM-DD format') parser.add_argument('--limitpct', action='store', default=0.005, type=float, help=('For buying at LIMIT, this will only purchase if the price is less than (1-limitpct)*Closing price')) parser.add_argument('--validdays', action='store', default=30, type=int, help=('The number of days which a buy order remains valid')) parser.add_argument('--sellscore', action='store', default=-0.91, type=float, help=('Max score for a sell')) parser.add_argument('--marketindex', default='XJO', help='XAO = All Ords, XJO = ASX200') parser.add_argument('--startingcash', default=100000, type=int, help='Starting Cash') parser.add_argument('--minholddays', default=3, type=int, help='Dont exit a market position until have held stock for at least this many days (excl. Stop-Loss and TP). May assist stopping exiting/cancelling orders when they are still being accepted by broker (i.e. day after entering mkt).') parser.add_argument('--pctperstock', action='store', #0.083 = 1/12... i.e. a portfolio of up to 12 stocks default=0.083, type=float, #i.e. 10% portfolio value in each stock help=('Pct of portfolio starting cash to invest in each stock purchase')) parser.add_argument('--maxpctperstock', action='store', default=0.20, type=float, help=('Max pct portfolio to invest in any porticular stock')) parser.add_argument('--mintrade', default=1000, type=float, help='Smallest dollar value to invest in a stock (if cash level below amount required for pctperstock)') parser.add_argument('--tradefee', default=10.0, type=float, help='CMC Markets Fee per stock trade (BUY OR SELL)') parser.add_argument('--alg_buyscore', #only used if Custom_Alg ==True action='store', # 0.91884558 default=0.91, type=float, help=('Min score for a buy')) return parser.parse_args() #Excel sheet with ASX200 index constituents (used for chosing stocks to analyse in Backtrader) def LoadIndicies(Excel_Path, Excel_Sheet): # Load ASX200 Excel File ASX200 = pd.read_excel(Excel_Path, sheetname=Excel_Sheet) Index_Constituents = ASX200.to_dict(orient='list') for key, value in Index_Constituents.items(): Index_Constituents[key] = [x for x in value if str(x) != 'nan'] # drop any "blank" (NaN) tickers from the index constituents table IndexDates = sorted(Index_Constituents.keys()) IndexDates.append(datetime.datetime.now().strftime("%Y-%m-%d")) # ordered list of the Index constituent Dates, with todays date at the end return Index_Constituents, IndexDates def LoadStockData(CSV_path=None): args = parse_args() if CSV_path is None: raise RuntimeError("no stock folder directory specifed.") allFiles = glob.glob(CSV_path + "/*.csv") Stocks = {} # Create a DICTIONARY object to store the entire contents of all dataframes, allows for easy reference to / looping through dataframes by a string of their name, i.e. : 'CSL' for file_ in allFiles: name = ntpath.basename(file_[:-4]) # Set DF name = basename (not path) of the CSV. [:-4] gets rid of the '.CSV' extention. Stocks[name] = pd.read_csv(file_, index_col='Date', parse_dates=True, header=0) return Stocks class StockLoader(btfeeds.PandasData): args = parse_args() params = ( ('openinterest', None), # None= column not present ('TOTAL_SCORE', -1)) # -1 = autodetect position or case-wise equal name if args.Custom_Alg == True: lines = ('TOTAL_SCORE',) if args.Custom_Alg == True: datafields = btfeeds.PandasData.datafields + (['TOTAL_SCORE']) else: datafields = btfeeds.PandasData.datafields class st(bt.Strategy): args = parse_args() params = ( #NB: self.p = self.params ('printlog', True), ) def log(self, txt, dt=None, doprint=False): if self.p.printlog or doprint: dt = dt or self.datas[0].datetime.date(0) print('%s - %s' % (dt.isoformat(), txt)) def __init__(self): self.o = {} # orders per data (main, stop, limit, manual-close) self.holding = {} # holding periods per data self.sma_short = {} self.sma_long = {} for i, d in enumerate(d for d in self.datas): self.sma_short[d] = bt.indicators.SimpleMovingAverage(d, period=42) #np.round(pd.rolling_mean(d.Close, window=42),2) self.sma_long[d] = bt.indicators.SimpleMovingAverage(d, period=252) #np.round(pd.rolling_mean(d.Close, window=252),2) # Plot Indicators if plot function called # bt.indicators.ExponentialMovingAverage(self.datas[0], period=25) # bt.indicators.WeightedMovingAverage(self.datas[0], period=25, subplot=True) # bt.indicators.StochasticSlow(self.datas[0]) # bt.indicators.MACDHisto(self.datas[0]) # rsi = bt.indicators.RSI(self.datas[0]) # bt.indicators.SmoothedMovingAverage(rsi, period=10) # bt.indicators.ATR(self.datas[0], plot=False) def notify_trade(self, trade): #NB: "print(Trade.__dict__)" if trade.isclosed: #Market Position exited self.log('OPERATION PROFIT: %s, Gross: %2f, Net: %2f' %(trade.data._name, trade.pnl, trade.pnlcomm)) def notify_order(self, order): '''Notes: SUBMITTED: Marks an order as submitted and stores the broker to which it was submitted. ACCEPTED: Marks an order as accepted COMPLETED: Order executed, completely filled. MARGIN: Not enough cash to execute the order REJECTED: broker could reject order if not enough cash CANCELLED: Marks an order as cancelled. Status.canceled occurs when: a "self.broker.cancel()" has occured by user #NB: "print(order.executed.__dict__)" ''' if order.status in [order.Submitted, order.Accepted]: return #do nothing #if order.status ==order.Margin: #self.cancel(self.o[order.data][1]) # cancel stop-loss #self.cancel(self.o[order.data][2]) # cancel take-profit if order.isbuy(): buysell = 'BUY' elif order.issell(): buysell = 'SELL' print('{} {}'.format(order.data._name, order.getstatusname())) print(buysell) #Order Type identification whichord = ['main','stop','limit','close'] dorders = self.o[order.data] print(dorders) idx = dorders.index(order) self.log('%s %s: %s, Type: %s, Ref: %s, Price: %.2f, Cost: %.2f, Size: %.2f, Comm %.2f' %(buysell, order.getstatusname(), order.data._name, whichord[idx], #Order Type (main, stop, limit or close) order.ref, order.executed.price, order.executed.value, order.executed.size, order.executed.comm)) if not order.alive():# indicate no order is pending, allows new orders. Alive = if order is in status Partial or Accepted dorders[idx] = None #nullify the specific order (main, stop, limit or close) print('-- No longer alive {} Ref'.format(whichord[idx])) if all(x is None for x in dorders): dorders[:] = [] # empty list - New orders allowed #def prenext(self): #overrides PRENEXT() so that the "NEXT()" calculations runs regardless of when each stock data date range starts. #self.next() def next(self): today = self.datetime.date(0) weekday = today.isoweekday() #Monday = 1, Sunday = 7 if weekday in range(1,8): # analyse on all weekdays (MONDAY to SUNDAY) num_long = 0 #number long stocks #IdealLongPortf = pd.DataFrame(columns=('Stock', 'Score','Close','Current Position', 'Ideal Position', 'Pos Delta Value', 'Go NoGo')) #ideal stock positions at end of each next() iteration for i, d in enumerate(d for d in self.datas): # Loop through Universe of Stocks. "If Len(d)" is used to check that all datafeeds have delivered values. as if using minute data, some may have had many minutes, 500, and another may not have 1 record yet (if its still on daily) if d._name != args.marketindex: position = self.broker.getposition(d) positiondol = float(self.broker.getposition(d).size*d.close[0]) cash = self.broker.getcash() #total available cash if not position.size\ and not self.o.get(d, None)\ and d.close[0] > 0 \ and self.sma_short[d][0] > self.sma_long[d][0]\ and cash > args.mintrade: #and d.lines.TOTAL_SCORE[0] >= args.alg_buyscore: #IdealLongPortf.append([d._name, d.lines.TOTAL_SCORE[0], d.close[0], position.size, np.NaN, np.NaN,np.NaN]) buylimit = d.close[0]*(1.0-args.limitpct) if args.SLTP_On == True: stop_loss = d.close[0]*(1.0 - args.stoploss) take_profit = d.close[0]*(1.0 + args.takeprofit) o1 = self.buy(data = d, exectype=bt.Order.Limit, price=buylimit, valid=today + datetime.timedelta(days=args.validdays), transmit=False) o2 = self.sell(data = d, size = o1.size, # could be an issue with re-balancing!!! exectype=bt.Order.Stop, price=stop_loss, parent=o1, transmit=False) o3 = self.sell(data = d, size = o1.size, exectype=bt.Order.Limit, price=take_profit, parent=o1, transmit=True) self.o[d] = [o1, o2, o3] self.log('CREATE BUY: %s, Main: %2f, Stop: %2f, Limit: %2f, Close: %2f, Score: %2f' %(d._name, buylimit, stop_loss, take_profit, d.close[0], 1)) #d.lines.TOTAL_SCORE[0])) else: o1 = self.buy(data = d, exectype=bt.Order.Limit, price=buylimit, valid=today + datetime.timedelta(days=args.validdays)) self.log('CREATE BUY: %s, Close: %2f, Buy @: %2f, Score: %2f' %(d._name, d.close[0], buylimit, 1)) #d.lines.TOTAL_SCORE[0])) self.o[d] = [o1] self.holding[d] = 0 elif position.size: # Currently LONG self.holding[d] += 1 num_long += 1 self.log('Stock Held: %s, Close: %2f, Posn: %i, Posn($): %2f, Days Held: %i, Score: %2f, Score Yest: %2f' %(d._name, d.close[0], position.size, positiondol, self.holding[d], 1, #d.lines.TOTAL_SCORE[0], 1)) #d.lines.TOTAL_SCORE[-1])) if position.size > 0\ and not self.o.get(d, None) \ and self.holding[d] >= args.minholddays \ and self.sma_short[d][0] < self.sma_long[d][0]: #and d.lines.TOTAL_SCORE[0] < args.alg_buyscore: self.log('CLOSING LONG POSITION: %s, Close: %2f, Score: %2f' %(d._name, d.close[0], 1)) #d.lines.TOTAL_SCORE[0])) if self.o[d][1]: self.cancel(self.o[d][1]) # cancel stop side, this automatically cancels the TP too self.log('CANCELLING SL & TP for: %s' %(d._name)) o = self.close(data=d) self.o[d].append(o) # manual order to list of orders elif position.size < 0: print('WTFMATE:{}, size:{}, Value$: {}, daysheld: ()'.format(d._name, position.size, positiondol)) totalwealth = self.broker.getvalue() cash = self.broker.getcash() invested = totalwealth - cash self.log("Stocks Held: %s, Total Wealth: %i, Invested: %i, Cash-On-Hand: %i" %(str(num_long), totalwealth, invested, cash)) def stop(self): pass class PortfolioSizer(bt.Sizer): def _getsizing(self, comminfo, cash, data, isbuy): args = parse_args() position = self.broker.getposition(data) price = data.close[0] investment = args.startingcash * args.pctperstock if cash < investment: investment = max(cash,args.mintrade) # i.e. never invest less than the "mintrade" $value qty = math.floor(investment/price) # This method returns the desired size for the buy/sell operation if isbuy: # if buying if position.size < 0: # if currently short, buy the amount which are short to close out trade. return -position.size elif position.size > 0: return 0 # dont buy if already hold else: return qty # num. stocks to LONG if not isbuy: # if selling.. if position.size < 0: return 0 # dont sell if already SHORT elif position.size > 0: return position.size # currently Long... sell what hold else: return qty # num. stocks to SHORT def RunStrategy(): args = parse_args() cerebro = bt.Cerebro() cerebro.addstrategy(st) # strats = cerebro.optstrategy(st,maperiod=range(10, 31)) #date range to backtest tradingdates = Stocks[args.marketindex].loc[ (Stocks[args.marketindex].index>=datetime.datetime.strptime(args.fromdate, "%Y-%m-%d")) & (Stocks[args.marketindex].index<datetime.datetime.strptime(args.todate, "%Y-%m-%d")) ] #Load 200 stocks into Backtrader (specified in the Index_constituents list) for ticker in Index_Constituents[IndexDates[3]]: datarange = Stocks[ticker].loc[ (Stocks[ticker].index>=datetime.datetime.strptime(args.fromdate, "%Y-%m-%d")) & (Stocks[ticker].index<datetime.datetime.strptime(args.todate, "%Y-%m-%d")) ] #REINDEX to make sure the stock has the exact same trading days as the MARKET INDEX. Reindex ffill doesn't fill GAPS. Therefore also apply FILLNA datarange.reindex(tradingdates.index, method='ffill').fillna(method='ffill',inplace=True) data = StockLoader(dataname=datarange) data.plotinfo.plot=False cerebro.adddata(data, name=ticker) data = btfeeds.PandasData(dataname=tradingdates, openinterest=None) #load market index (for date referencing) cerebro.adddata(data, name=args.marketindex) #cerebro.addanalyzer(CurrentBuysAnalyzer, ) cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, ) #length of holds etc cerebro.addanalyzer(bt.analyzers.TimeReturn, timeframe=bt.TimeFrame.Years) cerebro.addanalyzer(bt.analyzers.SharpeRatio, timeframe=bt.TimeFrame.Years, riskfreerate=0.03, annualize=True) cerebro.addanalyzer(bt.analyzers.SQN) cerebro.addanalyzer(bt.analyzers.DrawDown, ) cerebro.broker.setcash(args.startingcash) # set starting cash cerebro.addsizer(PortfolioSizer) commission = float(args.tradefee/(args.pctperstock*args.startingcash)) print("The Commission rate is: %0.5f" % (commission)) cerebro.broker.setcommission(commission=commission) cerebro.run(runonce=False, writer=True) cerebro.plot(volume=False, stdstats=False) ''' zdown=True: Rotation of the date labes on the x axis stdstats=False: Disable the standard plotted observers numfigs=1: Plot on one chart ''' if __name__ == '__main__': # if python is running this script (module) as the main program, then __name__ == __main__, and this block of code will run. # However, if another script (module) is IMPORTING this script (module), this block of code WILL NOT RUN, but the above functions can be called. args = parse_args() t1 = datetime.datetime.now() print('Processing Commenced: {}'.format(str(t1))) #Load stocks from local drive for analysis Stocks = LoadStockData(args.data0) t2 = datetime.datetime.now() # Dictionary of Index Constituents (and their stock dataframes) Index_Constituents, IndexDates = LoadIndicies( Excel_Path='T:\Google Drive\Capriole\CAPRIOLEPROCESSOR\TickerUpdater.xlsm', Excel_Sheet='ASX200_Const_Updated') print('ASX200 constituent list date: {}'.format(IndexDates[3])) if args.Custom_Alg == True: initiate() t3 = datetime.datetime.now() RunStrategy() t4 = datetime.datetime.now() #TIMER print('Run-time - TOTAL: {0}'.format(datetime.datetime.now() - t1)) print('Run-time - Load Data: {0}'.format(t2 - t1)) if 't3' in locals(): print('Run-time - Algorithm: {0}'.format(t3 - t2)) print('Run-time - Strategy Back-test: {0}'.format(t4 - t3)) else: print('Run-time - Strategy Back-test: {0}'.format(t4 - t2)) #Current_Buys() #...if using to BUY today!LOG (could only keep key sections due to message limit. All references to
EWCmaintained. Note the "SELL" word only prints once and in relation to EWC):2013-11-25 - CREATE BUY: GWA, Main: 3.048692, Stop: 2.757611, Limit: 9.192036, Close: 3.064012, Score: 1.000000 2013-11-25 - CREATE BUY: DL_MTU, Main: 6.059550, Stop: 5.481000, Limit: 18.270000, Close: 6.090000, Score: 1.000000 2013-11-25 - CREATE BUY: EWC, Main: 0.398000, Stop: 0.360000, Limit: 1.200000, Close: 0.400000, Score: 1.000000 2013-11-25 - CREATE BUY: DL_SKE, Main: 3.402900, Stop: 3.078000, Limit: 10.260000, Close: 3.420000, Score: 1.000000 ... BUY [<backtrader.order.BuyOrder object at 0x000002204C16C0F0>, <backtrader.order.SellOrder object at 0x000002204C1B39E8>, <backtrader.order.SellOrder object at 0x000002204C1B3B38>] 2013-11-26 - BUY Margin: DL_MTU, Type: main, Ref: 319, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref BUY [<backtrader.order.BuyOrder object at 0x000002204C1965F8>, <backtrader.order.SellOrder object at 0x000002204C42EFD0>, <backtrader.order.SellOrder object at 0x000002204C42E0F0>] 2013-11-26 - BUY Margin: EWC, Type: main, Ref: 322, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref BUY [<backtrader.order.BuyOrder object at 0x000002204C1B37B8>, <backtrader.order.SellOrder object at 0x000002204C1632E8>, <backtrader.order.SellOrder object at 0x000002204C1634E0>] 2013-11-26 - BUY Margin: DL_SKE, Type: main, Ref: 325, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref BUY [<backtrader.order.BuyOrder object at 0x000002204C16AA58>, <backtrader.order.SellOrder object at 0x000002204C1B9240>, <backtrader.order.SellOrder object at 0x000002204C1B9048>] ... BUY [<backtrader.order.BuyOrder object at 0x000002204C16FAC8>, <backtrader.order.SellOrder object at 0x000002204C16A828>, <backtrader.order.SellOrder object at 0x000002204C16A278>] 2013-11-27 - BUY Margin: SAI, Type: main, Ref: 292, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref BUY [<backtrader.order.BuyOrder object at 0x000002204C16A7F0>, <backtrader.order.SellOrder object at 0x000002204C13D550>, <backtrader.order.SellOrder object at 0x000002204C154908>] 2013-11-27 - BUY Margin: BDR, Type: main, Ref: 307, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref BUY [<backtrader.order.BuyOrder object at 0x000002204C154F28>, <backtrader.order.SellOrder object at 0x000002204C154A58>, <backtrader.order.SellOrder object at 0x000002204C154F98>] 2013-11-27 - BUY Margin: SXL, Type: main, Ref: 310, Price: 0.00, Cost: 0.00, Size: 0.00, Comm 0.00 -- No longer alive main Ref EWC Completed SELL [None, <backtrader.order.SellOrder object at 0x000002204C42EFD0>, <backtrader.order.SellOrder object at 0x000002204C42E0F0>] Traceback (most recent call last): File "T:/Google Drive/PyCharm/Backtrader.py", line 784, in <module> RunStrategy() File "T:/Google Drive/PyCharm/Backtrader.py", line 755, in RunStrategy cerebro.run(runonce=False, writer=True) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 794, in run runstrat = self.runstrategies(iterstrat) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 924, in runstrategies self._runnext(runstrats) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1240, in _runnext strat._next() File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 296, in _next super(Strategy, self)._next() File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 245, in _next self._notify() File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\strategy.py", line 499, in _notify self.notify_order(order) File "T:/Google Drive/PyCharm/Backtrader.py", line 211, in notify_order idx = dorders.index(order) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__ return self.ref == other.ref AttributeError: 'NoneType' object has no attribute 'ref' Process finished with exit code 1Thank you,
CWE -
This is unfortunately the typical case of a non-bug report. backtrader may be full of bugs, but when reporting one, there are some minimums:
- Version
- Some input data
- A simple use case in the code
Your report is a moving target.
-
In your Get Value thread it was set in question how the platform calculated the value
-
The diagnostic that you were entering short positions was not accepted because no short positions were entered because they were not shown in the log (which was only printing positive positions)
-
After some left and right, that thread is abandoned and a quick mention above shows that your code was entering short positions
-
The next thing is reporting that the sample in the code in the blog post produces a bug
-
The new sample data passed to the (modified) code is in the wrong order (timewise) which obviously produces errors and the answer is: "it's your script"
-
Instead of correcting the new sample data the answer is to throw in again a monster (in size) code and ask again for debugging.
-
Again, things like version, input data and simplicity are missing.
Is the sample from the blog post above (re)producing any bug? Because that was stated, but has not yet been confirmed.
In the meantime, your error.
File "T:/Google Drive/PyCharm/Backtrader.py", line 205, in notify_order idx = dorders.index(order) File "C:\Program Files\Anaconda3\lib\site-packages\backtrader\order.py", line 402, in __eq__ return self.ref == other.ref AttributeError: 'NoneType' object has no attribute 'ref'To support that comparison the following commit was added 16 days ago:
Bracket order support was added after that with this other commit:
And announced even later with this other commit tagged as version
1.9.37.116:The original Blog - Bracket orders post announcing bracket order support says:
Release
1.9.37.116adds bracket orders giving a very broad spectrum of orders which are supported by the backtesting broker (Market,Limit,Close,Stop,StopLimit,StopTrail,StopTrailLimit,OCO)It is therefore really difficult to imagine that you are using any version of backtrader which supports bracket orders.
-
Hi @backtrader,
Thank you for this detailed response.
My humble apologies for how this issue has been approached.
If I ever have a bug report in the future I will be sure to include that information.As it turns out, you are quite right... I stuffed up on the version front and wasn't using the lastest. I have since upgraded and the script runs completely! Wish I knew this yesterday before wasting several hours trying to debug a problem which didnt exist (my mistake!)!!
I assure you I haven't abandoned the
getvaluethread, I am still working on addressing your suggested solutions and will get back to you to confirm if I have resolved the issue regarding short positions as soon as I have time to do some more thorough testing.As a suggestion, with those example codes, would it be possible to include all elements for users such as myself to do a full run without edits? Perhaps inclusion of the necessary files etc too would be valuable and save some time. I spent some time yesterday to try and get your blog post to run and didn't consider that I would have to incorporate additional code to re-order data etc. I understand that it would never have worked for me though given my version issues..! But this may be useful for others going forward.
Thank you for your time and patience in assisting me as I learn this platform and process.
CWE
-
Thanks for this valuable example, and why not put it in the document?
-
It's a blog post.