Custom indicator/ understanding lines/list index out of range

guwop

Hi there, I'm trying to make a custom stochastic oscillator, but I'm running into indexing errors from what I imagine to be my lack of understanding of how lines should be declared and called within an indicator. I know that inside an indicator being developed that the lines which it has must be declared as so:

class MyStochastic(bt.Indicator):
    lines = ('stoch',)

Currently, this is how I'm calling my indicator from within the strategy:

self.stoch = MyStochastic(self.datas[1])

What I'm seeming to fail to understand is how the line name knows which data feed to associate with... For example, here is the rest of myindicator code:

class MyStochastic(bt.Indicator):
    lines = ('filts','stoch',)

   def __init__(self):

    self.lines.filts = self.datas[1].close #Most recent close of Filt
    HighestC = self.lines.filts
    LowestC = self.lines.filts
    for count in range (len(self.filts)-1,0,-1):
        if (self.filts[-count]>HighestC):
            HighestC=self.filts[-count]
        if (self.filts[-count]<LowestC):
            LowestC=self.filts[-count]
    self.Stoc = (self.filts-LowestC)/(HighestC-LowestC)
    if len(self.Stoc)==1:
         self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2
    if len(self.Stoc)==2:
         self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1]
    if (len(self.Stoc)>2):
         self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2]

I'm trying to do in the for loop (which is where my indexing error is popping up) is to look through the past x datapoints and iterating through them from oldest to most recent.

So I guess my question is two-fold: How is it that after declaring a line name, that it will be able to associate itself with a datafeed; and why is it that I'm receiving an "index out of range" for the fist line where:

self.lines.filts = self.datas[1].close #Most recent close of Filt

I'm still not fully confident that I have used lines correctly. I'll continue to update this as I move along, however, if anybody can see anything glaringly wrong please let me know.

ab_trader

Here is several things need to be corrected:

Strategy's init() is for array-type calculations only. No looping thru the data feed can be there.
self.lines.filts and self.filts - are they supposed to be the same or different?
why datas[1]? in case of single data feed it will throw the indexing error, cause only datas[0] exists
lines of stoch are not defined
lines of filts are defined as simple close prices only if there are 2 data feeds minimum
c1, c2, c3 are not defined at all
self.Stoc and self.MyStoch will not be returned as indicator values and might not be accessible

backtrader

@ab_trader has pinpointed most of the problems, you may want to have a look at the docs

See:

Docs - Indicator Development
- Shows how individual calculations are done in next if needed (it seems pointless to do them during object construction)
Docs - Platform Concepts, here you may want to focus on
- Accessing lines
- Slicing
- Delayed Indexing
- Stage1 and Stage2 (for the declarative calculations with delayed indexing and other indicators or the [] calculations in next)

@guwop said in Custom indicator/ understanding lines/list index out of range:

How is it that after declaring a line name, that it will be able to associate itself with a datafeed

Fail to understand the question. There is no association between the lines (which are the output channels of your indicators) and the data feeds (which is the input)

@guwop said in Custom indicator/ understanding lines/list index out of range:

self.lines.filts = self.datas[1].close #Most recent close of Filt

This comment in this line of code says something which is clearly wrong. We don't even know if you are passing more than one data feed to the indicator, because there is no sample usage.

guwop

@ab_trader ,@backtrader , thank you both for the explanations and directions. I now understand better the characteristics of that which I'm trying to accomplish - very helpful.
I'd like to address a couple of questions that y'all had:

Yes, I am using 'datas[1]' because I have a second data feed which I want passed instead of the first; c1,c2,c3 were intentionally left out, they are simply variables; 'self.lines.filts' and 'self.filts' were originally supposed to be the same but thanks to y'all I realized that they are not.

What I still don't get is why I'm receiving "list index out of range" when trying to pass the datafeed to the indicator. It worked fine when doing so within my strategy. I'm pasting my full code so that there isn't any confusion.

class MyStochastic(bt.Indicator):
    
    def __init__(self):
        lines = ('filts',)
        #period = 10
        f = math.degrees(1.44 * 3.14159 / 10)
        a1 = math.exp(-f)
        b1 = 2 * a1 * math.cos(f)
        c2 = b1
        c3 = -a1 * a1
        c1 = 1 - c2 - c3

        self.FiltClose = self.datas[1].close #Most recent close of Filt
        HighestC = self.lines.filts
        LowestC = self.lines.filts

    def next(self):
        for count in range (len(self.filts)-1,0,-1):
            if (self.filts[-count]>HighestC):
                HighestC=self.filts[-count]
            if (self.filts[-count]<LowestC):
                LowestC=self.filts[-count]
         self.Stoc = (self.filts-LowestC)/(HighestC-LowestC)

        if len(self.Stoc)==1:
            self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2
        if len(self.Stoc)==2:
            self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1]
        if (len(self.Stoc)>2):
            self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2]


# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function fot this strategy'''
        dt = dt or self.datas[0].datetime.date(0)   
        print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self):
           # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close
        self.FiltClose = self.datas[1].close

    # To keep track of pending orders and buy price/commission

        self.order = None
        self.buyprice = None
        self.buycomm = None

        self.rsi = bt.indicators.RSI_SMA(self.FiltClose, period=5,upperband=80,lowerband=20,safediv=True)

        self.stoch = MyStochastic(self.FiltClose)


    def notify_order(self, order):

        if order.status in [order.Submitted, order.Accepted]:
            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return

        # Check if an order has been completed
        # Attention: broker could reject order if not enough cash
        if order.status in [order.Completed]:
            if order.isbuy():
                self.log(
                       'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                    (order.executed.price,    
                     order.executed.value,
                     order.executed.comm))

                self.buyprice = order.executed.price
                self.buycomm = order.executed.comm

            else:  # Sell
                self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                         (order.executed.price,
                          order.executed.value,
                          order.executed.comm))

            self.bar_executed = len(self)

        elif order.status in [order.Canceled, order.Margin, order.Rejected]:

            self.log('Order Canceled/Margin/Rejected')

        self.order = None

    def notify_trade(self, trade):
            if not trade.isclosed:
               return

        self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                     (trade.pnl, trade.pnlcomm))



    def next(self):

        print(self.stoch)

        # Simply log the closing price of the series from the reference
        self.log('Close, %.2f' % self.dataclose[0])

        # Check if an order is pending ... if yes, we cannot send a 2nd one 
        if self.order:
            return

        # Check if we are in the market
        if not self.position:
            # Not yet ... we MIGHT BUY if ...
            if self.rsi[0] <= 20:

                # BUY, BUY, BUY!!! (with all possible default parameters)

                self.log('BUY CREATE, %.2f' % self.dataclose[0])

                # Keep track of the created order to avoid a 2nd order

                self.order = self.buy()

        else:

            if self.rsi[0] >= 80:

                # SELL, SELL, SELL!!! (with all possible default parameters)

                self.log('SELL CREATE, %.2f' % self.dataclose[0])

                # Keep track of the created order to avoid a 2nd order

                self.order = self.sell()

if __name__ == '__main__':

    # Create a cerebro entity
    cerebro = bt.Cerebro()

    cerebro.addstrategy(TestStrategy)

    datapath1 = ('C:/Users/Guwop/Desktop/Binance/BTCUSDT-AdjMar_Binance.csv')

    # Create a Data Feed

    data = bt.feeds.GenericCSVData(
        dataname=datapath1,
        fromdate=datetime.datetime(2018, 3, 8),
        todate=datetime.datetime(2018, 3, 25),
        dtformat = '%m/%d/%Y %H:%M',timeframe=bt.TimeFrame.Minutes
        )


    datapath2 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-FiltMar_Binance2.csv')

    Filt = bt.feeds.GenericCSVData(
        dataname=datapath2,
        fromdate = datetime.datetime(2018, 3, 8), 
        todate = datetime.datetime(2018,3,25),
        dtformat = '%m/%d/%Y %H:%M',
        datetime=0,
        close=1,
        open=-1,
        high=-1,
        low=-1,
        volume=-1,
        openinterest=-1, timeframe=bt.TimeFrame.Minutes)

    Filt.plotinfo.plotmaster = data

    # Add the Data Feed to Cerebro
    cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes,compression=96)
    cerebro.replaydata(Filt, timeframe=bt.TimeFrame.Minutes,compression=96)

    # Set our desired cash start

    cerebro.broker.setcash(100000.0)

    cerebro.addsizer(bt.sizers.FixedSize, stake=1)

    cerebro.broker.setcommission(commission=0.001) # 0.1% ... divide by 100 to remove the %

    # Print out the starting conditions

    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything

    cerebro.run()

    # Print out the final result

    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

backtrader

@guwop said in Custom indicator/ understanding lines/list index out of range:

    def __init__(self):
        lines = ('filts',)

This is obviously out of place, so it is difficult to believe it is a working indicator.

@guwop said in Custom indicator/ understanding lines/list index out of range:

self.stoch = MyStochastic(self.FiltClose)

Only 1 data feed is being provided to the indicator. It is easy to understand why self.datas[1].close fails to work. The documentation link (Platform Concepts, see above), explains how to work and pass data feeds.

Futhermore, you have already taken close as the reference you pass to your indicator, so trying to access close again is poised to fail.

@guwop said in Custom indicator/ understanding lines/list index out of range:

      HighestC = self.lines.filts
      LowestC = self.lines.filts

This seems meaningless.

Please don't take this as discouraging but it seems

You fail to understand what the input/s to the indicators is/are
You fail to understand what the output/s of the indicator is/are
Why some things are done in a declarative manner in __init__ and step-by-step calculations are done in next

This is because things are completely out of place and there are calculations all over the place that end up nowhere and which reference data feeds which are not being given to the indicator.

Let me suggest that before you keep on putting lines together, that you look at the source code of some indicators. Most are conceived as purely declarative (the entire logic is self-contained in __init__ with delayed operations), but there is one which could help:

ZeroLagIndicator - See it here: https://github.com/backtrader/backtrader/blob/master/backtrader/indicators/zlind.py

guwop

@backtrader, You hit the nail on the head, I'm pretty clueless about how to develop my own indicator - all I seem to do is confuse myself further. Don't worry about me getting discouraged, I'm here to learn as much as I can and to hopefully be able to give back to the community. I really appreciate the work that y'all do.

With that said, thank you for linking the ZeroLagIndicator, as my indicator is an Ehlers as well, and so it helped my understand the use of datafeeds in referencing past values for the calculation of a new one.

I combed through the code between the backtrader version of ZeroLagIndicator and the Ehlers version on his site to better see how to implement my own. I have two questions:

In the for in range loop it has:
```
for value1 in range(*self.limits):
```

where

 self.limits = [-self.p.gainlimit, self.p.gainlimit +1]

I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?

Below is the code from the backtrader version:

def next(self):
    leasterror = MAXINT  # 1000000 in original code
    bestec = ema = self.ema[0]  # seed value 1st time for ec
    price = self.data[0]
    ec1 = self.lines.ec[-1]
    alpha, alpha1 = self.ema.alpha, self.ema.alpha1
    for value1 in range(*self.limits):
        gain = value1 / 10
        ec = alpha * (ema + gain * (price - ec1)) + alpha1 * ec1
        error = abs(price - ec)
        if error < leasterror:
            leasterror = error
            bestec = ec

    self.lines.ec[0] = bestec

It seems as though only the bestec are chosen to be used for the line opposed to the ec of the best gain as from ehlers code. Now I realize that in the backtrader version bestgain was replaced with bestec , however, in the Ehlers version the EC is calculated with Bestgain in place of Gain . In other words, instead of the final line in backtrader version being:

self.lines.ec[0] = bestec

it would instead be:

self.lines.ec[0] = alpha*(ema + BestGain * (price-ec1)) + (1-alpha)*ec1

And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!

backtrader

@guwop said in Custom indicator/ understanding lines/list index out of range:

And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!

There is no BestGain in the code. The bestec is updated if needed in the loop with ec, which is derived from the current gain. Only the best will finally be taken into account.

@guwop said in Custom indicator/ understanding lines/list index out of range:

In the for in range loop it has:
for value1 in range(*self.limits):
where
 self.limits = [-self.p.gainlimit, self.p.gainlimit +1]  
I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?

Let me be blunt. If you don't know what the * operator is doing the code above, what you need is not to read the source of any backtrader indicator. You need work on your Python skills. The * is used to unpack things from an iterable. See some sources:

guwop

Just wanted to update that I got the stochastic to work as so:

class MyStochastic(bt.Indicator):
    lines = ('MyStoch','Stoc')

    def __init__(self):
    
        #period = 10
        f = math.degrees(1.44 * 3.14159 / 10)
        a1 = math.exp(-f)
        b1 = 2 * a1 * math.cos(f)
        c2 = b1
        c3 = -a1 * a1
        self.c1 = 1 - c2 - c3
        self.filts = self.datas[0] #Most recent close of Filt
        HighestC = self.filts
        LowestC = self.filts
        if(len(self.filts)>9):
        
            for count in range (9,0,-1):
            
                if (self.filts[-count]>HighestC):
                    HighestC=self.filts[-count]
                if (self.filts[-count]<LowestC):
                    LowestC=self.filts[-count]
                
            self.lines.Stoc = (self.filts-LowestC)/(HighestC-LowestC)
            S1 = self.lines.Stoc[-1]
            self.lines.MyStoch = self.c1*(Stoc+S1)/2 + c2*self.lines.MyStoch(-1) +  c3*self.lines.MyStoch(-2)

backtrader

Once again this is not to discourage you, but that indicator is actually doing nothing.

You are simply avoiding exceptions, but the indicator calculates no value at all.

Let me quote the DummyIndicator from the documentation which was linked above (for reference: https://www.backtrader.com/docu/inddev.html)

class DummyInd(bt.Indicator):
    lines = ('dummyline',)

    params = (('value', 5),)

    def next(self):
        self.lines.dummyline[0] = max(0.0, self.params.value)

You have no next method and your calculations are based on current values and not declarative (I prefer the declarative approach, but you may do it as you wish)

The dummy indicator does nothing special but calculates the value for each iteration of the data.

In that documentation, there is also an example of how to calculate a SimpleMovingAverage in next and take into account the own period one wants to add to the mix (in your case it seems the default is 9)

guwop

@backtrader, yeah I realized I prematurely posted my success - I had a different indicator printing itself. I'm going to look more into using max(), adding safediv to custom ind, and utilizing the next() method. Thank you for bearing with me

guwop

So I think I'm making some good progress, the issue I seem to be having now is that under the next() method I'm receiving the valueerror: "max() arg is an empty sequence". I understand this means that there is essentially nothing getting computed within it, but I'm not quite sure why. Could it possibly have something to do with how I used delayed indexing with MyStoch?

class MyStochastic(bt.Indicator):
    lines = ('MyStoch','Stoc','S1',)

    def __init__(self):
   
        self.f = math.degrees(1.44 * 3.14159 / 10)
        self.a1 = math.exp(-self.f)
        self.b1 = 2 * self.a1 * math.cos(self.f)
        self.c2 = self.b1
        self.c3 = -self.a1 * self.a1
        self.c1 = 1 - self.c2 - self.c3
        self.filts = self.datas[0] #Most recent close of Filt
        HighestC = self.filts
        LowestC = self.filts
        
        if(len(self.filts)>9):
            for count in range (9,0,-1):
                if (self.filts[-count]>HighestC):
                    HighestC=self.filts[-count]
                if (self.filts[-count]<LowestC):
                    LowestC=self.filts[-count]
        Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC))
        S1 = Stoc(-1)
       
    def next(self):
    
        self.lines.MyStoch[0] = max(self.c1*(self.Stoc+self.S1)/2 + self.c2*self.lines.MyStoch(-1) +  self.c3*self.lines.MyStoch(-2))

ab_trader

@guwop what do you try to achieve in the for loop? This moment is totally unclear for me, everything else is more or less clear.

guwop

@ab_trader, the for loop is to determine what the Highest and Lowest values were for a given lookback period, and it does so by starting from the oldest and working to the newest

ab_trader

@guwop check the slicing approaches in Concepts section. With .get and max() operator you might have better coding in next().

Honestly I think that loop doesn't work as you want.

Also backtrader has built-in indicators to define highest high and lowest low data values for certain period.

guwop

@ab_trader, I think you're right about the loop, I switched [-count] to (-count) and got a Boolean error so I'm messing with that now. I'll have to look for those indicators, they sound very useful. Thank you for the help

ab_trader

As an idea (didn't test it):

class MyStochastic(bt.Indicator):
    lines = ('MyStoch', 'Stoc', 'S1',)
    params = (('period', 9),)

    def __init__(self):
   
        self.f = math.degrees(1.44 * 3.14159 / 10)
        self.a1 = math.exp(-self.f)
        self.b1 = 2 * self.a1 * math.cos(self.f)
        self.c2 = self.b1
        self.c3 = -self.a1 * self.a1
        self.c1 = 1 - self.c2 - self.c3

	self.hhigh = bt.Highest(period = self.p.period)
	self.llow = bt.Lowest(period = self.p.period)
        
	self.l.Stoc = (self.datas[0].close - self.hhigh) / (self.hhigh - self.llow)
        self.l.S1 = self.Stoc(-1)
        self.l.MyStoch = self.c1 * (self.Stoc + self.S1) / 2
       
    def next(self):
		
        self.l.MyStoch[0] = self.l.MyStoch[0] + self.c2 * self.l.MyStoch[-1] +  self.c3 * self.l.MyStoch[-2]

backtrader

@guwop said in Custom indicator/ understanding lines/list index out of range:

    def __init__(self):
        ...
        if(len(self.filts)>9):
            for count in range (9,0,-1):
                if (self.filts[-count]>HighestC):
                    HighestC=self.filts[-count]
                if (self.filts[-count]<LowestC):
                    LowestC=self.filts[-count]

The inherent problem here is: the loop never executes. When things are in __init__ things are being initialized. Data feeds have a null length (they are also in the initialization phase) and the condition will never be True

You miss the entire idea: only in next you get to the next iteration of data. __init__ is only called once during the entire lifecycle of the object.

        Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC))
        S1 = Stoc(-1)

Max is meant to use more than 1 argument or else the maximum is the given value. In any case and without a working loop the maximum in that case will be 0/0 which is undetermined and will raise an exception.

@guwop said in Custom indicator/ understanding lines/list index out of range:

I'll have to look for those indicators, they sound very useful. Thank you for the help

You could also look at the source of the Stochastic in backtrader.

The usual approach for what you want to do:

Subclass the existing indicator
Add your extra lines
Perform the extra calculations for the lines you have added.

guwop

Okay so I've changed some things around, and now I'm able to get values for Stoc, S1, c1...3, the only part that's giving me issues now is getting a value for MyStoch as it keeps returning Nan - I imagine it has something to do with calculating MyStoch[-1] and [-2], but I haven't come to figure out how yet. @backtrader , thanks for the clarification of init's one time use. You'd think it'd be obvious that initializing would occur only once, but I think it was the concept of lines objects getting generated from operations in init that threw me off. Nonetheless things are looking much better now. @ab_trader , I haven't got a chance to try your code yet, but I really appreciate the effort; I'll give it a shot at some point.

class MyStochastic(bt.Indicator):
    lines = ('MyStoch','Stoc','filts',)

    def __init__(self):
    
        self.f = math.degrees(1.44 * 3.14159 / 10)
        self.a1 = math.exp(-self.f)
        self.b1 = 2 * self.a1 * math.cos(self.f)
        self.c2 = self.b1
        self.c3 = -self.a1 * self.a1
        self.c1 = 1 - self.c2 - self.c3
        self.addminperiod(11)

  
        super(MyStochastic,self).__init__()
       
    def next(self):
        self.filts[0] = self.data[0]

        self.HighestC = bt.indicators.Highest(period=9)
        self.LowestC = bt.indicators.Lowest(period=9)

        self.l.Stoc = (self.filts(0)-self.LowestC/(self.HighestC-self.LowestC))
        S1 = self.Stoc[-1]
        self.lines.MyStoch[0] = (self.c1*(self.l.Stoc[0]+S1)/2 + self.c2*self.l.MyStoch[-1] +  self.c3*self.l.MyStoch[-2])

@ab_trader , tried your code that you posted and worked fine out of the box, however, I still couldn't get past the nan values when calculating MyStoch.

Any thoughts?

backtrader

See here for a longer post

https://medium.com/@danjrod/custom-indicator-development-in-python-with-backtrader-bc775552dc3e