For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Array index out of range + other idiosyncracies



  • Below I have two custom indicators: ROC and Smooth. ROC is callable as a bt.indicators, however, Smooth is not. When trying to run code with bt.inidcators before Smooth I receive the error " module 'backtrader.indicators' has no attribute 'Smooth'". That is the first odd thing that occurs...

    Here is the full code I am working with:

    from __future__ import (absolute_import, division, print_function,
                                unicode_literals)
    
    import datetime  # For datetime objects
    import os.path  # To manage paths
    import sys  # To find out the script name (in argv[0])
    import math
    from math import cos, sin
    %matplotlib notebook
    
    # Import the backtrader platform
    import backtrader as bt
    
    class ROC(bt.Indicator):
        lines = ('roc','ma',)
    
        def __init__(self):
            self.ma[0] = self.data[0]
    
        def next(self):
            self.roc = (self.ma-self.ma(-1))
    
    
    class Smooth(bt.Indicator):
        lines = ('filty','smooth',)
    
        def __init__(self):
            self.l.smooth[0] = self.data[0]
            f = math.degrees( 1.44 * 3.14159 / 20)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = self.b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
        def next(self):
            self.filty = ((self.c1 * ((self.smooth(0) + self.smooth(-1)) / 2 + (self.c2 * self.filty(-1)) + (self.c3 * self.filty(-2)))))
    
    
    class ShortStrategy(bt.Strategy):
    
        def log(self, txt, dt=None):
            ''' Logging function fot this strategy'''
    
            dt = dt or self.datas[0].datetime.date(0)
            print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
    
            # Keep a reference to the "close" line in the data[0] dataseries
    
            self.dataclose = self.datas[0].close
            self.FiltClose = self.datas[1].close
            self.Filt2Close = self.datas[2].close
    
            #self.ema = bt.indicators.ExponentialMovingAverage(self.FiltClose, period=13)
            #self.eroc = bt.indicators.ROC(self.ema)
            self.rsi = bt.indicators.RSI_SMA(self.Filt2Close, period=15,upperband=80,lowerband=20,safediv=True)
            self.sma = bt.indicators.SimpleMovingAverage(self.dataclose,period=25)
            self.filties = Smooth(self.dataclose)
    
        # To keep track of pending orders and buy price/commission
            self.order = None
            self.buyprice = None
            self.buycomm = None
    
    
        def notify_order(self, order):
    
            if order.status in [order.Submitted, order.Accepted]:
                # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                return
    
            # Check if an order has been completed
            # Attention: broker could reject order if not enough cash
            if order.status in [order.Completed]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
                self.bar_executed = len(self)
    
            elif order.status in [order.Canceled, order.Margin, order.Rejected]:
                self.log('Order Canceled/Margin/Rejected')
    
            self.order = None
    
        def notify_trade(self, trade):
            if not trade.isclosed:
                return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                     (trade.pnl, trade.pnlcomm))
    
        def next(self):
    
            #self.log(self.eroc[0])
            self.log(self.filties[0])
    
            # Simply log the closing price of the series from the reference
            self.log('Close, %.2f' % self.dataclose[0])
            #self.log('Close, %.2f' % self.Filtclose[0])
            # Check if an order is pending ... if yes, we cannot send a 2nd one
            if self.order:
                return
    
            # Check if we are in the market
    
            if not self.position:
    
                # Not yet ... we MIGHT SELL if ...
                if ((self.dataclose[0] < self.sma[0]) & (self.rsi<45)):
    
                    # SELL, SELL, SELL!!! (with all possible default parameters)
                    self.log('SELL CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.sell()
    
            else:
    
                if (self.rsi[0]>45):
                    # BUY, BUY, BUY!!! (with all possible default parameters)
                    self.log('BUY CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.buy()
    
    
    if __name__ == '__main__':
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        cerebro.addstrategy(ShortStrategy)
    
        datapath1 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-AdjMar_Binance.csv')
    
        # Create a Data Feed
    
        data = bt.feeds.GenericCSVData(
            dataname=datapath1,
            fromdate=datetime.datetime(2018, 3, 8),
            todate=datetime.datetime(2018, 3, 25),
            dtformat = '%m/%d/%Y %H:%M',timeframe=bt.TimeFrame.Minutes
            )
    
        datapath2 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-FiltMar_Binance2.csv')
        Filt = bt.feeds.GenericCSVData(
            dataname=datapath2,
            fromdate = datetime.datetime(2018, 3, 8), 
            todate = datetime.datetime(2018,3,25),
            dtformat = '%m/%d/%Y %H:%M',
            datetime=0,
            close=1,
            open=-1,
            high=-1,
            low=-1,
            volume=-1,
            openinterest=-1, timeframe=bt.TimeFrame.Minutes)
    
        datapath3 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-FiltMar_Binance3.csv')
        Filt2 = bt.feeds.GenericCSVData(
            dataname=datapath3,
            fromdate = datetime.datetime(2018, 3, 8), 
            todate = datetime.datetime(2018,3,25),
            dtformat = '%m/%d/%Y %H:%M',
            datetime=0,
            close=1,
            open=-1,
            high=-1,
            low=-1,
            volume=-1,
            openinterest=-1, timeframe=bt.TimeFrame.Minutes)
    
    
        Filt.plotinfo.plotmaster = data
    
        # Add the Data Feed to Cerebro
        cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes,compression=48)
        cerebro.replaydata(Filt, timeframe=bt.TimeFrame.Minutes,compression=48)
        cerebro.replaydata(Filt2, timeframe=bt.TimeFrame.Minutes,compression=48)
    
        # Set our desired cash start
        cerebro.broker.setcash(100000.0)
    
        cerebro.addsizer(bt.sizers.FixedSize, stake=10)
    
        cerebro.broker.setcommission(commission=0.001) # 0.1% ... divide by 100 to remove the %
    
        # Print out the starting conditions
        print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()
        # Run over everything
        cerebro.run()
        # Print out the final result
        print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    
        cerebro.plot(volume=False)
    

    When I run the above code, this is the error I receive....

    IndexError                                Traceback (most recent call last)
    <ipython-input-1-dfc816206e87> in <module>()
    325 
    326     # Run over everything
    --> 327     cerebro.run()
    328 
    329     # Print out the final result
    
    ~\Anaconda3\lib\site-packages\backtrader\cerebro.py in run(self, **kwargs)
        1125             # let's skip process "spawning"
        1126             for iterstrat in iterstrats:
     -> 1127                 runstrat = self.runstrategies(iterstrat)
        1128                 self.runstrats.append(runstrat)
        1129                 if self._dooptimize:
    
     ~\Anaconda3\lib\site-packages\backtrader\cerebro.py in runstrategies(self, iterstrat, predata)
     
        1215             sargs = self.datas + list(sargs)
        1216             try:
     -> 1217                 strat = stratcls(*sargs, **skwargs)
        1218             except bt.errors.StrategySkipError:
        1219                 continue  # do not add strategy to the mix
    
     ~\Anaconda3\lib\site-packages\backtrader\metabase.py in __call__(cls, *args, **kwargs)
     
          86         _obj, args, kwargs = cls.donew(*args, **kwargs)
          87         _obj, args, kwargs = cls.dopreinit(_obj, *args, **kwargs)
     ---> 88         _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
          89         _obj, args, kwargs = cls.dopostinit(_obj, *args, **kwargs)
          90         return _obj
    
     ~\Anaconda3\lib\site-packages\backtrader\metabase.py in doinit(cls, _obj, *args, **kwargs)
     
          76 
          77     def doinit(cls, _obj, *args, **kwargs):
     ---> 78         _obj.__init__(*args, **kwargs)
          79         return _obj, args, kwargs
          80 
    
     <ipython-input-1-dfc816206e87> in __init__(self)
     
         168         self.rsi = bt.indicators.RSI_SMA(self.Filt2Close, period=15,upperband=80,lowerband=20,safediv=True)
         169         self.sma = bt.indicators.SimpleMovingAverage(self.dataclose,period=25)
     --> 170         self.filties = Smooth(self.dataclose)
         171     # To keep track of pending orders and buy price/commission
         172         self.order = None
    
    
     ~\Anaconda3\lib\site-packages\backtrader\indicator.py in __call__(cls, *args, **kwargs)
     
          51     def __call__(cls, *args, **kwargs):
          52         if not cls._icacheuse:
     ---> 53             return super(MetaIndicator, cls).__call__(*args, **kwargs)
          54 
          55         # implement a cache to avoid duplicating lines actions
    
     ~\Anaconda3\lib\site-packages\backtrader\metabase.py in __call__(cls, *args, **kwargs)
     
          86         _obj, args, kwargs = cls.donew(*args, **kwargs)
          87         _obj, args, kwargs = cls.dopreinit(_obj, *args, **kwargs)
     ---> 88         _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
          89         _obj, args, kwargs = cls.dopostinit(_obj, *args, **kwargs)
          90         return _obj
    
     ~\Anaconda3\lib\site-packages\backtrader\metabase.py in doinit(cls, _obj, *args, **kwargs)
     
          76 
          77     def doinit(cls, _obj, *args, **kwargs):
     ---> 78         _obj.__init__(*args, **kwargs)
          79         return _obj, args, kwargs
          80 
    
     <ipython-input-1-dfc816206e87> in __init__(self)
     
          29 
          30     def __init__(self):
         ---> 31         self.l.smooth[0] = self.data[0]
          32         f = math.degrees( 1.44 * 3.14159 / 20)
          33         a1 = math.exp(-f)
    
     ~\Anaconda3\lib\site-packages\backtrader\lineseries.py in __getitem__(self, key)
     
         465 
         466     def __getitem__(self, key):
     --> 467         return self.lines[0][key]
         468
         469     def __setitem__(self, key, value):
    
     ~\Anaconda3\lib\site-packages\backtrader\linebuffer.py in __getitem__(self, ago)
     
         161 
         162     def __getitem__(self, ago):
     --> 163         return self.array[self.idx + ago]
         164 
         165     def get(self, ago=0, size=1):
    
     IndexError: array index out of range
    

    The strange part is that as you can see, the setup for feeding the data into Smooth is the exact same as ROC, however, it doesn't seem to be working. I think it may be rooted in the fact that the program is not recognizing Smooth as an indicator, but that is just a guess.



  • You need to call self.l.filty or self.l.smooth, not just simple self.filty or self.smooth. Your actual self.smooth is not defined, but you tried to use it in final equation.

    Since you didn't use self.l.roc, than I really doubt that you will get what you want in the ROC indicator.


  • administrators

    Both indicators are wrong. You really want to have a look at the recent post:

    and

    The usage of [x] for the lines during __init__ is killing those indicators.



  • Thanks for the advice, I've partially gotten it to work in that it computes what it's supposed to and even plots values on the chart at the end, however, when I try to print out the values by logging it during the next method under strategies, I am given Nan. The issue now is understanding how to get it so that I can work with the data within strategies. Below is the new layout for the indicator:

    class Smooth(bt.Indicator):
    
        lines = ('filty','smooth',)
    
        def __init__(self):
    
            self.l.smooth = self.data
            f = math.degrees(1.44 * 3.14159 / 20)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
            self.l.filty = ((c1 * ((self.l.smooth(0) + self.l.smooth(-1)) / 2 + (c2 * self.l.filty(-1)) + (c3 * self.l.filty(-2)))))

  • administrators

    If you don't show what you do in the Strategy, it will be impossible for anyone to say what's wrong



  • @backtrader, It hadn't changed from the original post, but here is the strategy section:

    class ShortStrategy(bt.Strategy):
    
        def log(self, txt, dt=None):
            ''' Logging function fot this strategy'''
    
            dt = dt or self.datas[0].datetime.date(0)
            print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
    
            # Keep a reference to the "close" line in the data[0] dataseries
    
            self.dataclose = self.datas[0].close
            self.FiltClose = self.datas[1].close
            self.Filt2Close = self.datas[2].close
    
            #self.ema = bt.indicators.ExponentialMovingAverage(self.FiltClose, period=13)
            #self.eroc = bt.indicators.ROC(self.ema)
            self.rsi = bt.indicators.RSI_SMA(self.Filt2Close, period=15,upperband=80,lowerband=20,safediv=True)
            self.sma = bt.indicators.SimpleMovingAverage(self.dataclose,period=25)
            self.filties = Smooth(self.dataclose)
    
        # To keep track of pending orders and buy price/commission
            self.order = None
            self.buyprice = None
            self.buycomm = None
    
    
        def notify_order(self, order):
    
             if order.status in [order.Submitted, order.Accepted]:
                 # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                 return
    
             # Check if an order has been completed
            # Attention: broker could reject order if not enough cash
            if order.status in [order.Completed]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
                self.bar_executed = len(self)
    
            elif order.status in [order.Canceled, order.Margin, order.Rejected]:
                self.log('Order Canceled/Margin/Rejected')
    
            self.order = None
    
        def notify_trade(self, trade):
            if not trade.isclosed:
                return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                     (trade.pnl, trade.pnlcomm))
    
        def next(self):
    
            #self.log(self.eroc[0])
            self.log(self.filties[0])
    
            # Simply log the closing price of the series from the reference
            self.log('Close, %.2f' % self.dataclose[0])
            #self.log('Close, %.2f' % self.Filtclose[0])
            # Check if an order is pending ... if yes, we cannot send a 2nd one
           if self.order:
                return
    
             # Check if we are in the market
    
            if not self.position:
    
               # Not yet ... we MIGHT SELL if ...
                if ((self.dataclose[0] < self.sma[0]) & (self.rsi<45)):
    
                    # SELL, SELL, SELL!!! (with all possible default parameters)
                     self.log('SELL CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.sell()
    
            else:
    
                 if (self.rsi[0]>45):
                    # BUY, BUY, BUY!!! (with all possible default parameters)
                    self.log('BUY CREATE, %.2f' % self.dataclose[0])
    
                     # Keep track of the created order to avoid a 2nd order
                    self.order = self.buy()
    

    I'd also like to add that self.log(self.eroc[0]) would output the values of ROC in the text


  • administrators

    @guwop said in Array index out of range + other idiosyncracies:

    It hadn't changed from the original post,

    That's something the world cannot know.

    @guwop said in Array index out of range + other idiosyncracies:

        def next(self):
    
            #self.log(self.eroc[0])
            self.log(self.filties[0])
    
            # Simply log the closing price of the series from the reference
            self.log('Close, %.2f' % self.dataclose[0])
    

    You probably want to check the difference between the two non-commented self.log calls (and what's being passed) and what the def log method is doing.



  • @backtrader, So I've narrowed the problem down to a single line of code. After messing with the log method I figured that it wasn't the issue. For sake of simplicity I decided to simply print the values in the next method as so:

    print('Filties, %.2f' % self.filties[0])
    

    I've attempted both the declarative and non-declarative approaches as y'all highlighted in the custom indicator blog post. Below is the non (declarative is at bottom):

    class Smooth(bt.Indicator):
    lines = ('smooth','filty',)
    
        def __init__(self):
        
            self.addminperiod(20)
            self.count = 0
        
        def next(self):
            self.l.smooth[0] = sm = self.data[0]
            f = math.degrees(1.44 * 3.14159 / 20)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
            if(self.count==0):
                self.l.filty[0] = ((c1 * ((sm + sm) / 2 + (c2 * sm ) + (c3 * sm))))
            if(self.count==1):
                self.l.filty[0] = ((c1 * ((sm + self.l.smooth[-1]) / 2 + (c2 * sm ) + (c3 * self.l.smooth[0]))))
            if(self.count>=2):
                self.l.filty[0] = ((c1 * ((sm + self.l.smooth[-1]) / 2 + (c2 * self.l.filty[-1] ) + (c3 * self.l.filty[-2]))))
            
            self.count= self.count+1
    

    What I am doing here with the three lines of self.l.filty[0] is trying to initialize the line with values so that in the final line I can run it how it should. When I run the program as is, I am returned Nan values.

    However, in the final line, if I replace self.l.filty[-1] and self.l.filty[-2] with self.l.smooth[-1] and self.l.smooth[-2], then I am returned actual numerical values for the line filty. I don't understand why this is occurring because I'm almost certain that the first two values in the filty line are being initialized, and so I should be able to call for their values. If I try to call self.l.filty[-1] and self.l.smooth[-2], I am again returned Nan values.

    For the sake of being thorough, here is the declarative version I used:

    class Smooth(bt.Indicator):
        lines = ('smooth','filty',)
    
        def __init__(self):
        
            self.l.smooth = sm = self.data
            f = math.degrees(1.44 * 3.14159 / 20)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
            self.l.filty = ((c1 * ((sm + self.l.smooth(-1)) / 2 + (c2 * self.l.filty(-1)) + (c3 * self.l.filty(-2)))))


  • I'm getting a strong sense that maybe lines aren't able to call on past values of themselves to generate new values of the same line.I don't see why that would be an issue though, since calling past lines values is a standard thing, but it's the only explanation I have. It would be nice for it to work because then most other worthwhile filters would become usable.


  • administrators

    @guwop said in Array index out of range + other idiosyncracies:

    I'm getting a strong sense that maybe lines aren't able to call on past values of themselves to generate new values of the same line

    Sorry, but no matter how strong your sense is, it is a a very strong wrong sense.

    I understand you wan't to do it on your own and learn Python, but you keep on adding unneeded complexities to it.

    @guwop said in Array index out of range + other idiosyncracies:

            f = math.degrees(1.44 * 3.14159 / 20)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
    

    What's the point in making calculations if your what you want is to be able to understand how to work with lines? It's only confusing boilerplate.

    @guwop said in Array index out of range + other idiosyncracies:

        def __init__(self):
        
            self.l.smooth = sm = self.data
    

    What's the point in assigning the value of self.data to a line which you have defined in the indicator? Again, confusing boilerplate.

    @guwop said in Array index out of range + other idiosyncracies:

           self.l.filty = ((c1 * ((sm + self.l.smooth(-1)) / 2 + (c2 * self.l.filty(-1)) + (c3 * self.l.filty(-2)))))
    

    Your problem here is not the use of past values and the strong sense you have that you cannot, the problem is called: recursivity. You have not used an initial seed for self.l.filty. With no seed you can only get NaN because all lines are initalized to NaN.

    All lines objects support 2 methods in addition to next:

    • prenext which is called before the minimum warmup period has been reached (and there are no guarantees that all buffers in all objects can actually be accessed)

    • nextstart which is called exactly ONCE when the minimum warmup period has been reached and which defaults to delegate the call to next.

    See: