Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Limit to number of datafeeds?

    General Code/Help
    2
    11
    236
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      techydoc last edited by

      Adding 20 datafeeds and cerebro.run() throws

      array assignment index out of range
      

      Is there a limit to the number of datafeeds?

      run-out 1 Reply Last reply Reply Quote 0
      • run-out
        run-out @techydoc last edited by

        @techydoc No limit other than your machine. If you could provide more details of your error we might be able to help.

        RunBacktest.com

        T 1 Reply Last reply Reply Quote 0
        • T
          techydoc @run-out last edited by

          @run-out Thanks you have already been very helpful.

          I pulled the latest backtrader code from github to make sure I was on the latest version. backtrader==1.9.76.123

          I am using the Momentum strategy code you already helped me with. The first datafeed does not have any indicators (I say because where error located). If I added another 12 stock datafeeds the code runs, but when I try 13 or more, I get an error

            File "/home/swaldren/Development/gitrepo/quant/trade-testing/runstrategy.py", line 34, in <module>
              results = cerebro.run()
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1293, in runstrategies
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/cerebro.py", line 1652, in _runonce
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 297, in _once
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 317, in _once
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/lineiterator.py", line 327, in oncestart
            File "/home/techydoc/anaconda3/envs/quant/lib/python3.6/site-packages/backtrader-1.9.76.123-py3.6.egg/backtrader/indicators/basicops.py", line 70, in once
          IndexError: array assignment index out of range
          

          The code for runstrategy.py:

          import backtrader as bt
          from datetime import datetime
          import glob
          import os
          from strategies import MomentumStrategy
          
          
          cerebro = bt.Cerebro(stdstats=False)
          cerebro.broker.set_coc(True)
          
          spy = bt.feeds.BacktraderCSVData(dataname='./data/SPY/daily/SPY_daily_2001-02-05T060000_2021-02-05T060000',
                                           fromdate=datetime(2015,10,1),
                                           todate=datetime(2021,2,2),
                                           plot=True)
          cerebro.adddata(spy)  # add S&P 500 Index
          tickers = {'AAPL','AMZN','GOOG','TSLA','NVDA','DIS','KO','SQ','UBER','AMD','NIO','GM','PLTR'} #,'TWTR','PLUG','OPEN','CGC','PACB','MSTR'}
          
          
          ftype = 'daily'
          for ticker in tickers:
              # Load data
              list_of_files = glob.glob('./data/{}/{}/*'.format(ticker,ftype)) # get list of all data files for ticker
              latest_file = max(list_of_files, key=os.path.getctime)
              cerebro.adddata(bt.feeds.BacktraderCSVData(dataname=latest_file,
                                              fromdate=datetime(2016,2,2),
                                              todate=datetime(2021,2,2),
                                              plot=False))
          
          cerebro.addobserver(bt.observers.Value)
          cerebro.addanalyzer(bt.analyzers.SharpeRatio, riskfreerate=0.0)
          cerebro.addanalyzer(bt.analyzers.Returns)
          cerebro.addanalyzer(bt.analyzers.DrawDown)
          cerebro.addstrategy(MomentumStrategy.MomentumStrategy)
          results = cerebro.run()
          
          cerebro.plot(iplot=False)[0][0]
          
          print(f"Sharpe: {results[0].analyzers.sharperatio.get_analysis()['sharperatio']:.3f}")
          print(f"Norm. Annual Return: {results[0].analyzers.returns.get_analysis()['rnorm100']:.2f}%")
          print(f"Max Drawdown: {results[0].analyzers.drawdown.get_analysis()['max']['drawdown']:.2f}%")
          
          

          The momentum strategy code:

          
          '''
          Original code and algo from: https://teddykoker.com/2019/05/momentum-strategy-from-stocks-on-the-move-in-python/
          Revised code from: https://www.backtrader.com/blog/2019-05-20-momentum-strategy/momentum-strategy/
          
          Changes:
           - Monmentum func change <this_array> to <period>  and added variable for indicator 
              that is passed to function call
           - In __init__ 
              - fixed typo (add <p>) on call to params <momentum_period>
              - added:  self.stock_under_idx_mav_filter
                  - but this does not work, so reverted back a checks in rebalance and reposition method
           - In <prenext> added: >= self.p.stock_period 
           - In <prenext> and <nextstart> changed <self.datas> to <self.stocks> to exclude the idx datafeed
           - Fixed bugs in selling stocks as it checked self.data for positions not d
           - Added timers for rebalance and reposition
           - Converted top percentage of stocks to buy to a param
           - Converted risk parity sizing to a param
          
          TODO 
              * Add a cross over indicator for the SPY < SMA check
          
          '''
          import backtrader as bt
          import numpy as np
          from scipy.stats import linregress
          import collections
          
          class RepositionTimer(object):
              def __init__(self):
                  self.fridays = 0
                  self.curmonth = -1
          
              def __call__(self, d):
                  _, _, isowkday = d.isocalendar()
          
                  if d.month != self.curmonth:
                      self.curmonth = d.month
                      self.fridays = 0
          
                  # Mon=1 ... Sun=7
                  if isowkday == 5:
                      self.fridays += 1
          
                      if self.fridays == 2:  # 2nd Friday
                          return True  # timer allowed
          
                  return False  # timer disallowed
          
          def momentum_func(ind, period):
              r = np.log(period)
              slope, _, rvalue, _, _ = linregress(np.arange(len(r)), r)
              annualized = (1 + slope) ** 252
              return annualized * (rvalue ** 2)
          
          
          class MomentumIndicator(bt.ind.OperationN):
              lines = ('trend',)
              params = dict(period=50)
              func = momentum_func
          
          class MomentumStrategy(bt.Strategy):
              params = dict(
                  momentum=MomentumIndicator,  # parametrize the momentum and its period
                  momentum_period=90,
          
                  movav=bt.ind.SMA,  # parametrize the moving average and its periods
                  idx_period=200,
                  stock_period=100,
          
                  volatr=bt.ind.ATR,  # parametrize the volatility and its period
                  vol_period=20,
          
                  buy_top_perc_stock=0.3,  # the fraction of the top momentum stocks to buy
                  risk_parity_size=0.001,  # a standard weight to size the position in stocks
                  rebal_weekday=5  # rebalance 5 is Friday
              )
          
          
              def __init__(self):
                  #self.i = 0  # See below as to why the counter is commented out
                  self.inds = collections.defaultdict(dict)  # avoid per data dct in for
          
                  # Use "self.data0" (or self.data) in the script to make the naming not
                  # fixed on this being a "spy" strategy. Keep things generic
                  # self.spy = self.datas[0]
                  self.stocks = self.datas[1:]
          
                  # Again ... remove the name "spy"
                  self.idx_mav = self.p.movav(self.data0, period=self.p.idx_period)
                  for d in self.stocks:
                      self.inds[d]['mom'] = self.p.momentum(d, period=self.p.momentum_period)
                      self.inds[d]['mav'] = self.p.movav(d, period=self.p.stock_period)
                      self.inds[d]['vol'] = self.p.volatr(d, period=self.p.vol_period)
                  
                  #self.stock_under_idx_mav_filter = self.datas[0].open < self.idx_mav  # Was unable to make this a filter
          
                  # Timer to support rebalancing weekcarry over in case of holiday
                  self.add_timer(
                      name = "rebalance",
                      when=bt.Timer.SESSION_START,
                      weekdays=[self.p.rebal_weekday],
                      weekcarry=True,  # if a day isn't there, execute on the next
                  )
                  self.add_timer(
                      name = "reposition",
                      when=bt.Timer.SESSION_START,
                      allow=RepositionTimer(),
                      weekcarry=True,  # if a day isn't there, execute on the next
                  )
                  #List of stocks that have sufficient length (based on indicators)
                  self.d_with_len = []
              
              def notify_timer(self, timer, when, *args, **kwargs):
                  if kwargs['name'] == 'rebalance':
                      # print("Rebalanceing at {}".format(when))
                      self.rebalance_portfolio()
                  elif kwargs['name'] == 'reposition':
                      # print("Repositioning at {}".format(when))
                      self.rebalance_positions()
                  else:
                      print("Unknown Timer at {}".format(when))
          
              def prenext(self):
                  # Populate d_with_len
                  self.d_with_len = [d for d in self.stocks if len(d) >= self.p.stock_period]
                  # call next() even when data is not available for all tickers
                  self.next()
          
              def nextstart(self):
                  # This is called exactly ONCE, when next is 1st called and defaults to
                  # call `next`
                  self.d_with_len = self.stocks  # all data sets fulfill the guarantees now
          
                  self.next()  # delegate the work to next
          
              def next(self):
                  # converted code to use timers
                  '''
                  l = len(self)
                   if l % 5 == 0:
                       self.rebalance_portfolio()
                  if l % 10 == 0:
                      self.rebalance_positions()
                  '''
              
              def rebalance_portfolio(self):
                  # only look at data that we can have indicators for 
                  self.rankings = self.d_with_len
          
                  #if no stocks are ready return   - Added but not sure if needed
                  if(len(self.rankings) == 0):
                      return
          
                  self.rankings.sort(key=lambda d: self.inds[d]["mom"][0])
                  num_stocks = len(self.rankings)
                  
                  # sell stocks based on criteria
                  for i, d in enumerate(self.rankings):
                      if self.getposition(d).size:  # changed self.data to d
                          if i > num_stocks * self.p.buy_top_perc_stock or d < self.inds[d]["mav"]:
                              self.close(d)
                  
                  if self.datas[0].open < self.idx_mav:  #self.stock_under_idx_mav_filter:
                      return
                  
                  # buy stocks with remaining cash
                  for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]):
                      cash = self.broker.get_cash()
                      value = self.broker.get_value()
                      if cash <= 0:
                          break
                      if not self.getposition(d).size:   # changed self.data to d
                          size = value * self.p.risk_parity_size / self.inds[d]["vol"]
                          self.buy(d, size=size)
                          
                  
              def rebalance_positions(self):
                  num_stocks = len(self.rankings)
                  
                  if self.datas[0].open < self.idx_mav:      #self.stock_under_idx_mav_filter:
                      return
          
                  # rebalance all stocks
                  for i, d in enumerate(self.rankings[:int(num_stocks * self.p.buy_top_perc_stock)]):
                      cash = self.broker.get_cash()
                      value = self.broker.get_value()
                      if cash <= 0:
                          break
                      size = value * self.p.risk_parity_size  / self.inds[d]["vol"]
                      self.order_target_size(d, size)
          
          
          run-out 1 Reply Last reply Reply Quote 0
          • run-out
            run-out @techydoc last edited by

            @techydoc You error is in a base module handling indicator functions. Can you paste your entire code in here and I can look at it?

            RunBacktest.com

            T 1 Reply Last reply Reply Quote 0
            • T
              techydoc @run-out last edited by

              @run-out That is my entire code. The first block is runstrategy.py and the second is momentumstrategy.py.

              Those are the only two python files.

              run-out 2 Replies Last reply Reply Quote 0
              • run-out
                run-out @techydoc last edited by

                @techydoc My bad, didn't see the scroll down.

                RunBacktest.com

                1 Reply Last reply Reply Quote 0
                • run-out
                  run-out @techydoc last edited by

                  @techydoc My bad, didn't see the scroll down.

                  RunBacktest.com

                  T 1 Reply Last reply Reply Quote 0
                  • T
                    techydoc @run-out last edited by

                    @run-out No worries!! Appreciate all the help.

                    run-out 1 Reply Last reply Reply Quote 0
                    • run-out
                      run-out @techydoc last edited by

                      @techydoc I'm not 100% sure what's wrong, but I can get you in the right area. I'm running this code and debugging for this error:

                      dst[i] = func(src[i - period + 1: i + 1])
                      IndexError: array assignment index out of range
                      

                      This is occurring in basicops.py here:

                          '''
                          Calculates "func" for a given period
                      
                          Serves as a base for classes that work with a period and can express the
                          logic in a callable object
                      
                          Note:
                            Base classes must provide a "func" attribute which is a callable
                      
                          Formula:
                            - line = func(data, period)
                          '''
                          def next(self):
                              self.line[0] = self.func(self.data.get(size=self.p.period))
                      
                          def once(self, start, end):
                              dst = self.line.array
                              src = self.data.array
                              period = self.p.period
                              func = self.func
                      
                              for i in range(start, end):
                                  dst[i] = func(src[i - period + 1: i + 1])
                      

                      What's happening is the last loop is bouncing back and forth between your custom momentum function and here, until it runout out of room in the array. I suspect for some reason your np arrays are the wrong length.

                      I'm also getting this error:

                       xmin, xmax = axes[-1].get_xlim()
                      IndexError: list index out of range
                      

                      For another stock set. You should investigate the length of your out put of your custom indicator.

                      Good luck.

                      RunBacktest.com

                      T 1 Reply Last reply Reply Quote 2
                      • T
                        techydoc @run-out last edited by

                        @run-out Thanks! I will report back.

                        T 1 Reply Last reply Reply Quote 1
                        • T
                          techydoc @techydoc last edited by

                          One of my datafeeds was length 85. If remove it from the code, the error goes away. My original error is back now though...lol!

                          I had to learn about python logging (Java programmer here) so now I at least can work to figure out the issue (I hope)

                          Thanks!

                          1 Reply Last reply Reply Quote 1
                          • 1 / 1
                          • First post
                            Last post
                          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors