Limit to number of datafeeds?
-
Adding 20 datafeeds and cerebro.run() throws
array assignment index out of range
Is there a limit to the number of datafeeds?
-
@techydoc No limit other than your machine. If you could provide more details of your error we might be able to help.
-
@run-out Thanks you have already been very helpful.
I pulled the latest backtrader code from github to make sure I was on the latest version. backtrader==1.9.76.123
I am using the Momentum strategy code you already helped me with. The first datafeed does not have any indicators (I say because where error located). If I added another 12 stock datafeeds the code runs, but when I try 13 or more, I get an error
File "/home/swaldren/Development/gitrepo/quant/trade-testing/runstrategy.py", line 34, in <module> results = cerebro.run() File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1293, in runstrategies File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1652, in _runonce File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 297, in _once File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 317, in _once File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 327, in oncestart File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/indicators/basicops.py", line 70, in once IndexError: array assignment index out of range
The code for runstrategy.py:
import backtrader as bt from datetime import datetime import glob import os from strategies import MomentumStrategy cerebro = bt.Cerebro(stdstats=False) cerebro.broker.set_coc(True) spy = bt.feeds.BacktraderCSVData(dataname='./data/SPY/daily/SPY_daily_2001-02-05T060000_2021-02-05T060000', fromdate=datetime(2015,10,1), todate=datetime(2021,2,2), plot=True) cerebro.adddata(spy) # add S&P 500 Index tickers = {'AAPL','AMZN','GOOG','TSLA','NVDA','DIS','KO','SQ','UBER','AMD','NIO','GM','PLTR'} #,'TWTR','PLUG','OPEN','CGC','PACB','MSTR'} ftype = 'daily' for ticker in tickers: # Load data list_of_files = glob.glob('./data/{}/{}/*'.format(ticker,ftype)) # get list of all data files for ticker latest_file = max(list_of_files, key=os.path.getctime) cerebro.adddata(bt.feeds.BacktraderCSVData(dataname=latest_file, fromdate=datetime(2016,2,2), todate=datetime(2021,2,2), plot=False)) cerebro.addobserver(bt.observers.Value) cerebro.addanalyzer(bt.analyzers.SharpeRatio, riskfreerate=0.0) cerebro.addanalyzer(bt.analyzers.Returns) cerebro.addanalyzer(bt.analyzers.DrawDown) cerebro.addstrategy(MomentumStrategy.MomentumStrategy) results = cerebro.run() cerebro.plot(iplot=False)[0][0] print(f"Sharpe: {results[0].analyzers.sharperatio.get_analysis()['sharperatio']:.3f}") print(f"Norm. Annual Return: {results[0].analyzers.returns.get_analysis()['rnorm100']:.2f}%") print(f"Max Drawdown: {results[0].analyzers.drawdown.get_analysis()['max']['drawdown']:.2f}%")
The momentum strategy code:
''' Original code and algo from: https://teddykoker.com/2019/05/momentum-strategy-from-stocks-on-the-move-in-python/ Revised code from: https://www.backtrader.com/blog/2019-05-20-momentum-strategy/momentum-strategy/ Changes: - Monmentum func change <this_array> to <period> and added variable for indicator that is passed to function call - In __init__ - fixed typo (add <p>) on call to params <momentum_period> - added: self.stock_under_idx_mav_filter - but this does not work, so reverted back a checks in rebalance and reposition method - In <prenext> added: >= self.p.stock_period - In <prenext> and <nextstart> changed <self.datas> to <self.stocks> to exclude the idx datafeed - Fixed bugs in selling stocks as it checked self.data for positions not d - Added timers for rebalance and reposition - Converted top percentage of stocks to buy to a param - Converted risk parity sizing to a param TODO * Add a cross over indicator for the SPY < SMA check ''' import backtrader as bt import numpy as np from scipy.stats import linregress import collections class RepositionTimer(object): def __init__(self): self.fridays = 0 self.curmonth = -1 def __call__(self, d): _, _, isowkday = d.isocalendar() if d.month != self.curmonth: self.curmonth = d.month self.fridays = 0 # Mon=1 ... Sun=7 if isowkday == 5: self.fridays += 1 if self.fridays == 2: # 2nd Friday return True # timer allowed return False # timer disallowed def momentum_func(ind, period): r = np.log(period) slope, _, rvalue, _, _ = linregress(np.arange(len(r)), r) annualized = (1 + slope) ** 252 return annualized * (rvalue ** 2) class MomentumIndicator(bt.ind.OperationN): lines = ('trend',) params = dict(period=50) func = momentum_func class MomentumStrategy(bt.Strategy): params = dict( momentum=MomentumIndicator, # parametrize the momentum and its period momentum_period=90, movav=bt.ind.SMA, # parametrize the moving average and its periods idx_period=200, stock_period=100, volatr=bt.ind.ATR, # parametrize the volatility and its period vol_period=20, buy_top_perc_stock=0.3, # the fraction of the top momentum stocks to buy risk_parity_size=0.001, # a standard weight to size the position in stocks rebal_weekday=5 # rebalance 5 is Friday ) def __init__(self): #self.i = 0 # See below as to why the counter is commented out self.inds = collections.defaultdict(dict) # avoid per data dct in for # Use "self.data0" (or self.data) in the script to make the naming not # fixed on this being a "spy" strategy. Keep things generic # self.spy = self.datas[0] self.stocks = self.datas[1:] # Again ... remove the name "spy" self.idx_mav = self.p.movav(self.data0, period=self.p.idx_period) for d in self.stocks: self.inds[d]['mom'] = self.p.momentum(d, period=self.p.momentum_period) self.inds[d]['mav'] = self.p.movav(d, period=self.p.stock_period) self.inds[d]['vol'] = self.p.volatr(d, period=self.p.vol_period) #self.stock_under_idx_mav_filter = self.datas[0].open < self.idx_mav # Was unable to make this a filter # Timer to support rebalancing weekcarry over in case of holiday self.add_timer( name = "rebalance", when=bt.Timer.SESSION_START, weekdays=[self.p.rebal_weekday], weekcarry=True, # if a day isn't there, execute on the next ) self.add_timer( name = "reposition", when=bt.Timer.SESSION_START, allow=RepositionTimer(), weekcarry=True, # if a day isn't there, execute on the next ) #List of stocks that have sufficient length (based on indicators) self.d_with_len = [] def notify_timer(self, timer, when, *args, **kwargs): if kwargs['name'] == 'rebalance': # print("Rebalanceing at {}".format(when)) self.rebalance_portfolio() elif kwargs['name'] == 'reposition': # print("Repositioning at {}".format(when)) self.rebalance_positions() else: print("Unknown Timer at {}".format(when)) def prenext(self): # Populate d_with_len self.d_with_len = [d for d in self.stocks if len(d) >= self.p.stock_period] # call next() even when data is not available for all tickers self.next() def nextstart(self): # This is called exactly ONCE, when next is 1st called and defaults to # call `next` self.d_with_len = self.stocks # all data sets fulfill the guarantees now self.next() # delegate the work to next def next(self): # converted code to use timers ''' l = len(self) if l % 5 == 0: self.rebalance_portfolio() if l % 10 == 0: self.rebalance_positions() ''' def rebalance_portfolio(self): # only look at data that we can have indicators for self.rankings = self.d_with_len #if no stocks are ready return - Added but not sure if needed if(len(self.rankings) == 0): return self.rankings.sort(key=lambda d: self.inds[d]["mom"][0]) num_stocks = len(self.rankings) # sell stocks based on criteria for i, d in enumerate(self.rankings): if self.getposition(d).size: # changed self.data to d if i > num_stocks * self.p.buy_top_perc_stock or d < self.inds[d]["mav"]: self.close(d) if self.datas[0].open < self.idx_mav: #self.stock_under_idx_mav_filter: return # buy stocks with remaining cash for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]): cash = self.broker.get_cash() value = self.broker.get_value() if cash <= 0: break if not self.getposition(d).size: # changed self.data to d size = value * self.p.risk_parity_size / self.inds[d]["vol"] self.buy(d, size=size) def rebalance_positions(self): num_stocks = len(self.rankings) if self.datas[0].open < self.idx_mav: #self.stock_under_idx_mav_filter: return # rebalance all stocks for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]): cash = self.broker.get_cash() value = self.broker.get_value() if cash <= 0: break size = value * self.p.risk_parity_size / self.inds[d]["vol"] self.order_target_size(d, size)
-
@techydoc You error is in a base module handling indicator functions. Can you paste your entire code in here and I can look at it?
-
@run-out That is my entire code. The first block is runstrategy.py and the second is momentumstrategy.py.
Those are the only two python files.
-
@techydoc My bad, didn't see the scroll down.
-
@techydoc My bad, didn't see the scroll down.
-
@run-out No worries!! Appreciate all the help.
-
@techydoc I'm not 100% sure what's wrong, but I can get you in the right area. I'm running this code and debugging for this error:
dst[i] = func(src[i - period + 1: i + 1]) IndexError: array assignment index out of range
This is occurring in basicops.py here:
''' Calculates "func" for a given period Serves as a base for classes that work with a period and can express the logic in a callable object Note: Base classes must provide a "func" attribute which is a callable Formula: - line = func(data, period) ''' def next(self): self.line[0] = self.func(self.data.get(size=self.p.period)) def once(self, start, end): dst = self.line.array src = self.data.array period = self.p.period func = self.func for i in range(start, end): dst[i] = func(src[i - period + 1: i + 1])
What's happening is the last loop is bouncing back and forth between your custom momentum function and here, until it runout out of room in the array. I suspect for some reason your np arrays are the wrong length.
I'm also getting this error:
xmin, xmax = axes[-1].get_xlim() IndexError: list index out of range
For another stock set. You should investigate the length of your out put of your custom indicator.
Good luck.
-
@run-out Thanks! I will report back.
-
One of my datafeeds was length 85. If remove it from the code, the error goes away. My original error is back now though...lol!
I had to learn about python logging (Java programmer here) so now I at least can work to figure out the issue (I hope)
Thanks!