Limit to number of datafeeds?

[techydoc]

Adding 20 datafeeds and cerebro.run() throws

array assignment index out of range

Is there a limit to the number of datafeeds?

[run-out]

@techydoc No limit other than your machine. If you could provide more details of your error we might be able to help.

[techydoc]

@run-out Thanks you have already been very helpful.

I pulled the latest backtrader code from github to make sure I was on the latest version. backtrader==1.9.76.123

I am using the Momentum strategy code you already helped me with. The first datafeed does not have any indicators (I say because where error located). If I added another 12 stock datafeeds the code runs, but when I try 13 or more, I get an error

  File "/home/swaldren/Development/gitrepo/quant/trade-testing/runstrategy.py", line 34, in <module>
    results = cerebro.run()
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1293, in runstrategies
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1652, in _runonce
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 297, in _once
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 317, in _once
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 327, in oncestart
  File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/indicators/basicops.py", line 70, in once
IndexError: array assignment index out of range

The code for runstrategy.py:

import backtrader as bt
from datetime import datetime
import glob
import os
from strategies import MomentumStrategy


cerebro = bt.Cerebro(stdstats=False)
cerebro.broker.set_coc(True)

spy = bt.feeds.BacktraderCSVData(dataname='./data/SPY/daily/SPY_daily_2001-02-05T060000_2021-02-05T060000',
                                 fromdate=datetime(2015,10,1),
                                 todate=datetime(2021,2,2),
                                 plot=True)
cerebro.adddata(spy)  # add S&P 500 Index
tickers = {'AAPL','AMZN','GOOG','TSLA','NVDA','DIS','KO','SQ','UBER','AMD','NIO','GM','PLTR'} #,'TWTR','PLUG','OPEN','CGC','PACB','MSTR'}


ftype = 'daily'
for ticker in tickers:
    # Load data
    list_of_files = glob.glob('./data/{}/{}/*'.format(ticker,ftype)) # get list of all data files for ticker
    latest_file = max(list_of_files, key=os.path.getctime)
    cerebro.adddata(bt.feeds.BacktraderCSVData(dataname=latest_file,
                                    fromdate=datetime(2016,2,2),
                                    todate=datetime(2021,2,2),
                                    plot=False))

cerebro.addobserver(bt.observers.Value)
cerebro.addanalyzer(bt.analyzers.SharpeRatio, riskfreerate=0.0)
cerebro.addanalyzer(bt.analyzers.Returns)
cerebro.addanalyzer(bt.analyzers.DrawDown)
cerebro.addstrategy(MomentumStrategy.MomentumStrategy)
results = cerebro.run()

cerebro.plot(iplot=False)[0][0]

print(f"Sharpe: {results[0].analyzers.sharperatio.get_analysis()['sharperatio']:.3f}")
print(f"Norm. Annual Return: {results[0].analyzers.returns.get_analysis()['rnorm100']:.2f}%")
print(f"Max Drawdown: {results[0].analyzers.drawdown.get_analysis()['max']['drawdown']:.2f}%")

The momentum strategy code:

'''
Original code and algo from: https://teddykoker.com/2019/05/momentum-strategy-from-stocks-on-the-move-in-python/
Revised code from: https://www.backtrader.com/blog/2019-05-20-momentum-strategy/momentum-strategy/

Changes:
 - Monmentum func change <this_array> to <period>  and added variable for indicator 
    that is passed to function call
 - In __init__ 
    - fixed typo (add <p>) on call to params <momentum_period>
    - added:  self.stock_under_idx_mav_filter
        - but this does not work, so reverted back a checks in rebalance and reposition method
 - In <prenext> added: >= self.p.stock_period 
 - In <prenext> and <nextstart> changed <self.datas> to <self.stocks> to exclude the idx datafeed
 - Fixed bugs in selling stocks as it checked self.data for positions not d
 - Added timers for rebalance and reposition
 - Converted top percentage of stocks to buy to a param
 - Converted risk parity sizing to a param

TODO 
    * Add a cross over indicator for the SPY < SMA check

'''
import backtrader as bt
import numpy as np
from scipy.stats import linregress
import collections

class RepositionTimer(object):
    def __init__(self):
        self.fridays = 0
        self.curmonth = -1

    def __call__(self, d):
        _, _, isowkday = d.isocalendar()

        if d.month != self.curmonth:
            self.curmonth = d.month
            self.fridays = 0

        # Mon=1 ... Sun=7
        if isowkday == 5:
            self.fridays += 1

            if self.fridays == 2:  # 2nd Friday
                return True  # timer allowed

        return False  # timer disallowed

def momentum_func(ind, period):
    r = np.log(period)
    slope, _, rvalue, _, _ = linregress(np.arange(len(r)), r)
    annualized = (1 + slope) ** 252
    return annualized * (rvalue ** 2)


class MomentumIndicator(bt.ind.OperationN):
    lines = ('trend',)
    params = dict(period=50)
    func = momentum_func

class MomentumStrategy(bt.Strategy):
    params = dict(
        momentum=MomentumIndicator,  # parametrize the momentum and its period
        momentum_period=90,

        movav=bt.ind.SMA,  # parametrize the moving average and its periods
        idx_period=200,
        stock_period=100,

        volatr=bt.ind.ATR,  # parametrize the volatility and its period
        vol_period=20,

        buy_top_perc_stock=0.3,  # the fraction of the top momentum stocks to buy
        risk_parity_size=0.001,  # a standard weight to size the position in stocks
        rebal_weekday=5  # rebalance 5 is Friday
    )


    def __init__(self):
        #self.i = 0  # See below as to why the counter is commented out
        self.inds = collections.defaultdict(dict)  # avoid per data dct in for

        # Use "self.data0" (or self.data) in the script to make the naming not
        # fixed on this being a "spy" strategy. Keep things generic
        # self.spy = self.datas[0]
        self.stocks = self.datas[1:]

        # Again ... remove the name "spy"
        self.idx_mav = self.p.movav(self.data0, period=self.p.idx_period)
        for d in self.stocks:
            self.inds[d]['mom'] = self.p.momentum(d, period=self.p.momentum_period)
            self.inds[d]['mav'] = self.p.movav(d, period=self.p.stock_period)
            self.inds[d]['vol'] = self.p.volatr(d, period=self.p.vol_period)
        
        #self.stock_under_idx_mav_filter = self.datas[0].open < self.idx_mav  # Was unable to make this a filter

        # Timer to support rebalancing weekcarry over in case of holiday
        self.add_timer(
            name = "rebalance",
            when=bt.Timer.SESSION_START,
            weekdays=[self.p.rebal_weekday],
            weekcarry=True,  # if a day isn't there, execute on the next
        )
        self.add_timer(
            name = "reposition",
            when=bt.Timer.SESSION_START,
            allow=RepositionTimer(),
            weekcarry=True,  # if a day isn't there, execute on the next
        )
        #List of stocks that have sufficient length (based on indicators)
        self.d_with_len = []
    
    def notify_timer(self, timer, when, *args, **kwargs):
        if kwargs['name'] == 'rebalance':
            # print("Rebalanceing at {}".format(when))
            self.rebalance_portfolio()
        elif kwargs['name'] == 'reposition':
            # print("Repositioning at {}".format(when))
            self.rebalance_positions()
        else:
            print("Unknown Timer at {}".format(when))

    def prenext(self):
        # Populate d_with_len
        self.d_with_len = [d for d in self.stocks if len(d) >= self.p.stock_period]
        # call next() even when data is not available for all tickers
        self.next()

    def nextstart(self):
        # This is called exactly ONCE, when next is 1st called and defaults to
        # call `next`
        self.d_with_len = self.stocks  # all data sets fulfill the guarantees now

        self.next()  # delegate the work to next

    def next(self):
        # converted code to use timers
        '''
        l = len(self)
         if l % 5 == 0:
             self.rebalance_portfolio()
        if l % 10 == 0:
            self.rebalance_positions()
        '''
    
    def rebalance_portfolio(self):
        # only look at data that we can have indicators for 
        self.rankings = self.d_with_len

        #if no stocks are ready return   - Added but not sure if needed
        if(len(self.rankings) == 0):
            return

        self.rankings.sort(key=lambda d: self.inds[d]["mom"][0])
        num_stocks = len(self.rankings)
        
        # sell stocks based on criteria
        for i, d in enumerate(self.rankings):
            if self.getposition(d).size:  # changed self.data to d
                if i > num_stocks * self.p.buy_top_perc_stock or d < self.inds[d]["mav"]:
                    self.close(d)
        
        if self.datas[0].open < self.idx_mav:  #self.stock_under_idx_mav_filter:
            return
        
        # buy stocks with remaining cash
        for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]):
            cash = self.broker.get_cash()
            value = self.broker.get_value()
            if cash <= 0:
                break
            if not self.getposition(d).size:   # changed self.data to d
                size = value * self.p.risk_parity_size / self.inds[d]["vol"]
                self.buy(d, size=size)
                
        
    def rebalance_positions(self):
        num_stocks = len(self.rankings)
        
        if self.datas[0].open < self.idx_mav:      #self.stock_under_idx_mav_filter:
            return

        # rebalance all stocks
        for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]):
            cash = self.broker.get_cash()
            value = self.broker.get_value()
            if cash <= 0:
                break
            size = value * self.p.risk_parity_size  / self.inds[d]["vol"]
            self.order_target_size(d, size)

[run-out]

@techydoc You error is in a base module handling indicator functions. Can you paste your entire code in here and I can look at it?

[techydoc]

@run-out That is my entire code. The first block is runstrategy.py and the second is momentumstrategy.py.

Those are the only two python files.

[run-out]

@techydoc My bad, didn't see the scroll down.

[run-out]

@techydoc My bad, didn't see the scroll down.

[techydoc]

@run-out No worries!! Appreciate all the help.

[run-out]

@techydoc I'm not 100% sure what's wrong, but I can get you in the right area. I'm running this code and debugging for this error:

dst[i] = func(src[i - period + 1: i + 1])
IndexError: array assignment index out of range

This is occurring in basicops.py here:

    '''
    Calculates "func" for a given period

    Serves as a base for classes that work with a period and can express the
    logic in a callable object

    Note:
      Base classes must provide a "func" attribute which is a callable

    Formula:
      - line = func(data, period)
    '''
    def next(self):
        self.line[0] = self.func(self.data.get(size=self.p.period))

    def once(self, start, end):
        dst = self.line.array
        src = self.data.array
        period = self.p.period
        func = self.func

        for i in range(start, end):
            dst[i] = func(src[i - period + 1: i + 1])

What's happening is the last loop is bouncing back and forth between your custom momentum function and here, until it runout out of room in the array. I suspect for some reason your np arrays are the wrong length.

I'm also getting this error:

 xmin, xmax = axes[-1].get_xlim()
IndexError: list index out of range

For another stock set. You should investigate the length of your out put of your custom indicator.

Good luck.

[techydoc]

@run-out Thanks! I will report back.

[techydoc]

One of my datafeeds was length 85. If remove it from the code, the error goes away. My original error is back now though...lol!

I had to learn about python logging (Java programmer here) so now I at least can work to figure out the issue (I hope)

Thanks!