For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Custom indicator/ understanding lines/list index out of range



  • Hi there, I'm trying to make a custom stochastic oscillator, but I'm running into indexing errors from what I imagine to be my lack of understanding of how lines should be declared and called within an indicator. I know that inside an indicator being developed that the lines which it has must be declared as so:

    class MyStochastic(bt.Indicator):
        lines = ('stoch',)
    

    Currently, this is how I'm calling my indicator from within the strategy:

    self.stoch = MyStochastic(self.datas[1])
    

    What I'm seeming to fail to understand is how the line name knows which data feed to associate with... For example, here is the rest of myindicator code:

    class MyStochastic(bt.Indicator):
        lines = ('filts','stoch',)
    
       def __init__(self):
    
        self.lines.filts = self.datas[1].close #Most recent close of Filt
        HighestC = self.lines.filts
        LowestC = self.lines.filts
        for count in range (len(self.filts)-1,0,-1):
            if (self.filts[-count]>HighestC):
                HighestC=self.filts[-count]
            if (self.filts[-count]<LowestC):
                LowestC=self.filts[-count]
        self.Stoc = (self.filts-LowestC)/(HighestC-LowestC)
        if len(self.Stoc)==1:
             self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2
        if len(self.Stoc)==2:
             self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1]
        if (len(self.Stoc)>2):
             self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2]
    

    I'm trying to do in the for loop (which is where my indexing error is popping up) is to look through the past x datapoints and iterating through them from oldest to most recent.

    So I guess my question is two-fold: How is it that after declaring a line name, that it will be able to associate itself with a datafeed; and why is it that I'm receiving an "index out of range" for the fist line where:

    self.lines.filts = self.datas[1].close #Most recent close of Filt
    

    I'm still not fully confident that I have used lines correctly. I'll continue to update this as I move along, however, if anybody can see anything glaringly wrong please let me know.



  • Here is several things need to be corrected:

    • Strategy's init() is for array-type calculations only. No looping thru the data feed can be there.
    • self.lines.filts and self.filts - are they supposed to be the same or different?
    • why datas[1]? in case of single data feed it will throw the indexing error, cause only datas[0] exists
    • lines of stoch are not defined
    • lines of filts are defined as simple close prices only if there are 2 data feeds minimum
    • c1, c2, c3 are not defined at all
    • self.Stoc and self.MyStoch will not be returned as indicator values and might not be accessible

  • administrators

    @ab_trader has pinpointed most of the problems, you may want to have a look at the docs

    See:

    • Docs - Indicator Development

      • Shows how individual calculations are done in next if needed (it seems pointless to do them during object construction)
    • Docs - Platform Concepts, here you may want to focus on

      • Accessing lines
      • Slicing
      • Delayed Indexing
      • Stage1 and Stage2 (for the declarative calculations with delayed indexing and other indicators or the [] calculations in next)

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    How is it that after declaring a line name, that it will be able to associate itself with a datafeed

    Fail to understand the question. There is no association between the lines (which are the output channels of your indicators) and the data feeds (which is the input)

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    self.lines.filts = self.datas[1].close #Most recent close of Filt
    

    This comment in this line of code says something which is clearly wrong. We don't even know if you are passing more than one data feed to the indicator, because there is no sample usage.



  • @ab_trader ,@backtrader , thank you both for the explanations and directions. I now understand better the characteristics of that which I'm trying to accomplish - very helpful.
    I'd like to address a couple of questions that y'all had:

    Yes, I am using 'datas[1]' because I have a second data feed which I want passed instead of the first; c1,c2,c3 were intentionally left out, they are simply variables; 'self.lines.filts' and 'self.filts' were originally supposed to be the same but thanks to y'all I realized that they are not.

    What I still don't get is why I'm receiving "list index out of range" when trying to pass the datafeed to the indicator. It worked fine when doing so within my strategy. I'm pasting my full code so that there isn't any confusion.

    class MyStochastic(bt.Indicator):
        
        def __init__(self):
            lines = ('filts',)
            #period = 10
            f = math.degrees(1.44 * 3.14159 / 10)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            c1 = 1 - c2 - c3
    
            self.FiltClose = self.datas[1].close #Most recent close of Filt
            HighestC = self.lines.filts
            LowestC = self.lines.filts
    
        def next(self):
            for count in range (len(self.filts)-1,0,-1):
                if (self.filts[-count]>HighestC):
                    HighestC=self.filts[-count]
                if (self.filts[-count]<LowestC):
                    LowestC=self.filts[-count]
             self.Stoc = (self.filts-LowestC)/(HighestC-LowestC)
    
            if len(self.Stoc)==1:
                self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2
            if len(self.Stoc)==2:
                self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1]
            if (len(self.Stoc)>2):
                self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2]
    
    
    # Create a Stratey
    class TestStrategy(bt.Strategy):
    
        def log(self, txt, dt=None):
            ''' Logging function fot this strategy'''
            dt = dt or self.datas[0].datetime.date(0)   
            print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
               # Keep a reference to the "close" line in the data[0] dataseries
            self.dataclose = self.datas[0].close
            self.FiltClose = self.datas[1].close
    
        # To keep track of pending orders and buy price/commission
    
            self.order = None
            self.buyprice = None
            self.buycomm = None
    
            self.rsi = bt.indicators.RSI_SMA(self.FiltClose, period=5,upperband=80,lowerband=20,safediv=True)
    
            self.stoch = MyStochastic(self.FiltClose)
    
    
        def notify_order(self, order):
    
            if order.status in [order.Submitted, order.Accepted]:
                # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                return
    
            # Check if an order has been completed
            # Attention: broker could reject order if not enough cash
            if order.status in [order.Completed]:
                if order.isbuy():
                    self.log(
                           'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,    
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
    
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
                self.bar_executed = len(self)
    
            elif order.status in [order.Canceled, order.Margin, order.Rejected]:
    
                self.log('Order Canceled/Margin/Rejected')
    
            self.order = None
    
        def notify_trade(self, trade):
                if not trade.isclosed:
                   return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                         (trade.pnl, trade.pnlcomm))
    
    
    
        def next(self):
    
            print(self.stoch)
    
            # Simply log the closing price of the series from the reference
            self.log('Close, %.2f' % self.dataclose[0])
    
            # Check if an order is pending ... if yes, we cannot send a 2nd one 
            if self.order:
                return
    
            # Check if we are in the market
            if not self.position:
                # Not yet ... we MIGHT BUY if ...
                if self.rsi[0] <= 20:
    
                    # BUY, BUY, BUY!!! (with all possible default parameters)
    
                    self.log('BUY CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
    
                    self.order = self.buy()
    
            else:
    
                if self.rsi[0] >= 80:
    
                    # SELL, SELL, SELL!!! (with all possible default parameters)
    
                    self.log('SELL CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
    
                    self.order = self.sell()
    
    if __name__ == '__main__':
    
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        cerebro.addstrategy(TestStrategy)
    
        datapath1 = ('C:/Users/Guwop/Desktop/Binance/BTCUSDT-AdjMar_Binance.csv')
    
        # Create a Data Feed
    
        data = bt.feeds.GenericCSVData(
            dataname=datapath1,
            fromdate=datetime.datetime(2018, 3, 8),
            todate=datetime.datetime(2018, 3, 25),
            dtformat = '%m/%d/%Y %H:%M',timeframe=bt.TimeFrame.Minutes
            )
    
    
        datapath2 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-FiltMar_Binance2.csv')
    
        Filt = bt.feeds.GenericCSVData(
            dataname=datapath2,
            fromdate = datetime.datetime(2018, 3, 8), 
            todate = datetime.datetime(2018,3,25),
            dtformat = '%m/%d/%Y %H:%M',
            datetime=0,
            close=1,
            open=-1,
            high=-1,
            low=-1,
            volume=-1,
            openinterest=-1, timeframe=bt.TimeFrame.Minutes)
    
        Filt.plotinfo.plotmaster = data
    
        # Add the Data Feed to Cerebro
        cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes,compression=96)
        cerebro.replaydata(Filt, timeframe=bt.TimeFrame.Minutes,compression=96)
    
        # Set our desired cash start
    
        cerebro.broker.setcash(100000.0)
    
        cerebro.addsizer(bt.sizers.FixedSize, stake=1)
    
        cerebro.broker.setcommission(commission=0.001) # 0.1% ... divide by 100 to remove the %
    
        # Print out the starting conditions
    
        print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
    
        # Run over everything
    
        cerebro.run()
    
        # Print out the final result
    
        print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

  • administrators

    @guwop said in Custom indicator/ understanding lines/list index out of range:

        def __init__(self):
            lines = ('filts',)
    

    This is obviously out of place, so it is difficult to believe it is a working indicator.

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    self.stoch = MyStochastic(self.FiltClose)
    

    Only 1 data feed is being provided to the indicator. It is easy to understand why self.datas[1].close fails to work. The documentation link (Platform Concepts, see above), explains how to work and pass data feeds.

    Futhermore, you have already taken close as the reference you pass to your indicator, so trying to access close again is poised to fail.

    @guwop said in Custom indicator/ understanding lines/list index out of range:

          HighestC = self.lines.filts
          LowestC = self.lines.filts
    

    This seems meaningless.

    Please don't take this as discouraging but it seems

    • You fail to understand what the input/s to the indicators is/are
    • You fail to understand what the output/s of the indicator is/are
    • Why some things are done in a declarative manner in __init__ and step-by-step calculations are done in next

    This is because things are completely out of place and there are calculations all over the place that end up nowhere and which reference data feeds which are not being given to the indicator.

    Let me suggest that before you keep on putting lines together, that you look at the source code of some indicators. Most are conceived as purely declarative (the entire logic is self-contained in __init__ with delayed operations), but there is one which could help:



  • @backtrader, You hit the nail on the head, I'm pretty clueless about how to develop my own indicator - all I seem to do is confuse myself further. Don't worry about me getting discouraged, I'm here to learn as much as I can and to hopefully be able to give back to the community. I really appreciate the work that y'all do.

    With that said, thank you for linking the ZeroLagIndicator, as my indicator is an Ehlers as well, and so it helped my understand the use of datafeeds in referencing past values for the calculation of a new one.

    I combed through the code between the backtrader version of ZeroLagIndicator and the Ehlers version on his site to better see how to implement my own. I have two questions:

    1. In the for in range loop it has:

      for value1 in range(*self.limits):
      

    where

     self.limits = [-self.p.gainlimit, self.p.gainlimit +1]  
    

    I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?

    1. Below is the code from the backtrader version:

      def next(self):
          leasterror = MAXINT  # 1000000 in original code
          bestec = ema = self.ema[0]  # seed value 1st time for ec
          price = self.data[0]
          ec1 = self.lines.ec[-1]
          alpha, alpha1 = self.ema.alpha, self.ema.alpha1
          for value1 in range(*self.limits):
              gain = value1 / 10
              ec = alpha * (ema + gain * (price - ec1)) + alpha1 * ec1
              error = abs(price - ec)
              if error < leasterror:
                  leasterror = error
                  bestec = ec
      
          self.lines.ec[0] = bestec
      

    It seems as though only the bestec are chosen to be used for the line opposed to the ec of the best gain as from ehlers code. Now I realize that in the backtrader version bestgain was replaced with bestec , however, in the Ehlers version the EC is calculated with Bestgain in place of Gain . In other words, instead of the final line in backtrader version being:

    self.lines.ec[0] = bestec
    

    it would instead be:

    self.lines.ec[0] = alpha*(ema + BestGain * (price-ec1)) + (1-alpha)*ec1 
    

    And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!


  • administrators

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!

    There is no BestGain in the code. The bestec is updated if needed in the loop with ec, which is derived from the current gain. Only the best will finally be taken into account.

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    1. In the for in range loop it has:
    for value1 in range(*self.limits):
    

    where

     self.limits = [-self.p.gainlimit, self.p.gainlimit +1]  
    

    I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?

    Let me be blunt. If you don't know what the * operator is doing the code above, what you need is not to read the source of any backtrader indicator. You need work on your Python skills. The * is used to unpack things from an iterable. See some sources:



  • Just wanted to update that I got the stochastic to work as so:

    class MyStochastic(bt.Indicator):
        lines = ('MyStoch','Stoc')
    
        def __init__(self):
        
            #period = 10
            f = math.degrees(1.44 * 3.14159 / 10)
            a1 = math.exp(-f)
            b1 = 2 * a1 * math.cos(f)
            c2 = b1
            c3 = -a1 * a1
            self.c1 = 1 - c2 - c3
            self.filts = self.datas[0] #Most recent close of Filt
            HighestC = self.filts
            LowestC = self.filts
            if(len(self.filts)>9):
            
                for count in range (9,0,-1):
                
                    if (self.filts[-count]>HighestC):
                        HighestC=self.filts[-count]
                    if (self.filts[-count]<LowestC):
                        LowestC=self.filts[-count]
                    
                self.lines.Stoc = (self.filts-LowestC)/(HighestC-LowestC)
                S1 = self.lines.Stoc[-1]
                self.lines.MyStoch = self.c1*(Stoc+S1)/2 + c2*self.lines.MyStoch(-1) +  c3*self.lines.MyStoch(-2)

  • administrators

    Once again this is not to discourage you, but that indicator is actually doing nothing.

    You are simply avoiding exceptions, but the indicator calculates no value at all.

    Let me quote the DummyIndicator from the documentation which was linked above (for reference: https://www.backtrader.com/docu/inddev.html)

    class DummyInd(bt.Indicator):
        lines = ('dummyline',)
    
        params = (('value', 5),)
    
        def next(self):
            self.lines.dummyline[0] = max(0.0, self.params.value)
    

    You have no next method and your calculations are based on current values and not declarative (I prefer the declarative approach, but you may do it as you wish)

    The dummy indicator does nothing special but calculates the value for each iteration of the data.

    In that documentation, there is also an example of how to calculate a SimpleMovingAverage in next and take into account the own period one wants to add to the mix (in your case it seems the default is 9)



  • @backtrader, yeah I realized I prematurely posted my success - I had a different indicator printing itself. I'm going to look more into using max(), adding safediv to custom ind, and utilizing the next() method. Thank you for bearing with me



  • So I think I'm making some good progress, the issue I seem to be having now is that under the next() method I'm receiving the valueerror: "max() arg is an empty sequence". I understand this means that there is essentially nothing getting computed within it, but I'm not quite sure why. Could it possibly have something to do with how I used delayed indexing with MyStoch?

    class MyStochastic(bt.Indicator):
        lines = ('MyStoch','Stoc','S1',)
    
        def __init__(self):
       
            self.f = math.degrees(1.44 * 3.14159 / 10)
            self.a1 = math.exp(-self.f)
            self.b1 = 2 * self.a1 * math.cos(self.f)
            self.c2 = self.b1
            self.c3 = -self.a1 * self.a1
            self.c1 = 1 - self.c2 - self.c3
            self.filts = self.datas[0] #Most recent close of Filt
            HighestC = self.filts
            LowestC = self.filts
            
            if(len(self.filts)>9):
                for count in range (9,0,-1):
                    if (self.filts[-count]>HighestC):
                        HighestC=self.filts[-count]
                    if (self.filts[-count]<LowestC):
                        LowestC=self.filts[-count]
            Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC))
            S1 = Stoc(-1)
           
        def next(self):
        
            self.lines.MyStoch[0] = max(self.c1*(self.Stoc+self.S1)/2 + self.c2*self.lines.MyStoch(-1) +  self.c3*self.lines.MyStoch(-2))


  • @guwop what do you try to achieve in the for loop? This moment is totally unclear for me, everything else is more or less clear.



  • @ab_trader, the for loop is to determine what the Highest and Lowest values were for a given lookback period, and it does so by starting from the oldest and working to the newest



  • @guwop check the slicing approaches in Concepts section. With .get and max() operator you might have better coding in next().

    Honestly I think that loop doesn't work as you want.

    Also backtrader has built-in indicators to define highest high and lowest low data values for certain period.



  • @ab_trader, I think you're right about the loop, I switched [-count] to (-count) and got a Boolean error so I'm messing with that now. I'll have to look for those indicators, they sound very useful. Thank you for the help



  • As an idea (didn't test it):

    class MyStochastic(bt.Indicator):
        lines = ('MyStoch', 'Stoc', 'S1',)
        params = (('period', 9),)
    
        def __init__(self):
       
            self.f = math.degrees(1.44 * 3.14159 / 10)
            self.a1 = math.exp(-self.f)
            self.b1 = 2 * self.a1 * math.cos(self.f)
            self.c2 = self.b1
            self.c3 = -self.a1 * self.a1
            self.c1 = 1 - self.c2 - self.c3
    
    	self.hhigh = bt.Highest(period = self.p.period)
    	self.llow = bt.Lowest(period = self.p.period)
            
    	self.l.Stoc = (self.datas[0].close - self.hhigh) / (self.hhigh - self.llow)
            self.l.S1 = self.Stoc(-1)
            self.l.MyStoch = self.c1 * (self.Stoc + self.S1) / 2
           
        def next(self):
    		
            self.l.MyStoch[0] = self.l.MyStoch[0] + self.c2 * self.l.MyStoch[-1] +  self.c3 * self.l.MyStoch[-2]
    

  • administrators

    @guwop said in Custom indicator/ understanding lines/list index out of range:

        def __init__(self):
            ...
            if(len(self.filts)>9):
                for count in range (9,0,-1):
                    if (self.filts[-count]>HighestC):
                        HighestC=self.filts[-count]
                    if (self.filts[-count]<LowestC):
                        LowestC=self.filts[-count]
    

    The inherent problem here is: the loop never executes. When things are in __init__ things are being initialized. Data feeds have a null length (they are also in the initialization phase) and the condition will never be True

    You miss the entire idea: only in next you get to the next iteration of data. __init__ is only called once during the entire lifecycle of the object.

            Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC))
            S1 = Stoc(-1)
    

    Max is meant to use more than 1 argument or else the maximum is the given value. In any case and without a working loop the maximum in that case will be 0/0 which is undetermined and will raise an exception.

    @guwop said in Custom indicator/ understanding lines/list index out of range:

    I'll have to look for those indicators, they sound very useful. Thank you for the help

    You could also look at the source of the Stochastic in backtrader.

    The usual approach for what you want to do:

    • Subclass the existing indicator
    • Add your extra lines
    • Perform the extra calculations for the lines you have added.


  • Okay so I've changed some things around, and now I'm able to get values for Stoc, S1, c1...3, the only part that's giving me issues now is getting a value for MyStoch as it keeps returning Nan - I imagine it has something to do with calculating MyStoch[-1] and [-2], but I haven't come to figure out how yet. @backtrader , thanks for the clarification of init's one time use. You'd think it'd be obvious that initializing would occur only once, but I think it was the concept of lines objects getting generated from operations in init that threw me off. Nonetheless things are looking much better now. @ab_trader , I haven't got a chance to try your code yet, but I really appreciate the effort; I'll give it a shot at some point.

    class MyStochastic(bt.Indicator):
        lines = ('MyStoch','Stoc','filts',)
    
        def __init__(self):
        
            self.f = math.degrees(1.44 * 3.14159 / 10)
            self.a1 = math.exp(-self.f)
            self.b1 = 2 * self.a1 * math.cos(self.f)
            self.c2 = self.b1
            self.c3 = -self.a1 * self.a1
            self.c1 = 1 - self.c2 - self.c3
            self.addminperiod(11)
    
      
            super(MyStochastic,self).__init__()
           
        def next(self):
            self.filts[0] = self.data[0]
    
            self.HighestC = bt.indicators.Highest(period=9)
            self.LowestC = bt.indicators.Lowest(period=9)
    
            self.l.Stoc = (self.filts(0)-self.LowestC/(self.HighestC-self.LowestC))
            S1 = self.Stoc[-1]
            self.lines.MyStoch[0] = (self.c1*(self.l.Stoc[0]+S1)/2 + self.c2*self.l.MyStoch[-1] +  self.c3*self.l.MyStoch[-2])
    

    @ab_trader , tried your code that you posted and worked fine out of the box, however, I still couldn't get past the nan values when calculating MyStoch.

    Any thoughts?


  • administrators