Custom indicator/ understanding lines/list index out of range
-
Hi there, I'm trying to make a custom stochastic oscillator, but I'm running into indexing errors from what I imagine to be my lack of understanding of how lines should be declared and called within an indicator. I know that inside an indicator being developed that the lines which it has must be declared as so:
class MyStochastic(bt.Indicator): lines = ('stoch',)Currently, this is how I'm calling my indicator from within the strategy:
self.stoch = MyStochastic(self.datas[1])What I'm seeming to fail to understand is how the line name knows which data feed to associate with... For example, here is the rest of myindicator code:
class MyStochastic(bt.Indicator): lines = ('filts','stoch',) def __init__(self): self.lines.filts = self.datas[1].close #Most recent close of Filt HighestC = self.lines.filts LowestC = self.lines.filts for count in range (len(self.filts)-1,0,-1): if (self.filts[-count]>HighestC): HighestC=self.filts[-count] if (self.filts[-count]<LowestC): LowestC=self.filts[-count] self.Stoc = (self.filts-LowestC)/(HighestC-LowestC) if len(self.Stoc)==1: self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2 if len(self.Stoc)==2: self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] if (len(self.Stoc)>2): self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2]I'm trying to do in the for loop (which is where my indexing error is popping up) is to look through the past x datapoints and iterating through them from oldest to most recent.
So I guess my question is two-fold: How is it that after declaring a line name, that it will be able to associate itself with a datafeed; and why is it that I'm receiving an "index out of range" for the fist line where:
self.lines.filts = self.datas[1].close #Most recent close of FiltI'm still not fully confident that I have used lines correctly. I'll continue to update this as I move along, however, if anybody can see anything glaringly wrong please let me know.
-
Here is several things need to be corrected:
- Strategy's
init()is for array-type calculations only. No looping thru the data feed can be there. self.lines.filtsandself.filts- are they supposed to be the same or different?- why
datas[1]? in case of single data feed it will throw the indexing error, cause onlydatas[0]exists - lines of
stochare not defined - lines of
filtsare defined as simplecloseprices only if there are 2 data feeds minimum - c1, c2, c3 are not defined at all
self.Stocandself.MyStochwill not be returned as indicator values and might not be accessible
- Strategy's
-
@ab_trader has pinpointed most of the problems, you may want to have a look at the docs
See:
-
- Shows how individual calculations are done in
nextif needed (it seems pointless to do them during object construction)
- Shows how individual calculations are done in
-
Docs - Platform Concepts, here you may want to focus on
- Accessing lines
- Slicing
- Delayed Indexing
- Stage1 and Stage2 (for the declarative calculations with delayed indexing and other indicators or the
[]calculations innext)
@guwop said in Custom indicator/ understanding lines/list index out of range:
How is it that after declaring a line name, that it will be able to associate itself with a datafeed
Fail to understand the question. There is no association between the lines (which are the output channels of your indicators) and the data feeds (which is the input)
@guwop said in Custom indicator/ understanding lines/list index out of range:
self.lines.filts = self.datas[1].close #Most recent close of FiltThis comment in this line of code says something which is clearly wrong. We don't even know if you are passing more than one data feed to the indicator, because there is no sample usage.
-
-
@ab_trader ,@backtrader , thank you both for the explanations and directions. I now understand better the characteristics of that which I'm trying to accomplish - very helpful.
I'd like to address a couple of questions that y'all had:Yes, I am using 'datas[1]' because I have a second data feed which I want passed instead of the first; c1,c2,c3 were intentionally left out, they are simply variables; 'self.lines.filts' and 'self.filts' were originally supposed to be the same but thanks to y'all I realized that they are not.
What I still don't get is why I'm receiving "list index out of range" when trying to pass the datafeed to the indicator. It worked fine when doing so within my strategy. I'm pasting my full code so that there isn't any confusion.
class MyStochastic(bt.Indicator): def __init__(self): lines = ('filts',) #period = 10 f = math.degrees(1.44 * 3.14159 / 10) a1 = math.exp(-f) b1 = 2 * a1 * math.cos(f) c2 = b1 c3 = -a1 * a1 c1 = 1 - c2 - c3 self.FiltClose = self.datas[1].close #Most recent close of Filt HighestC = self.lines.filts LowestC = self.lines.filts def next(self): for count in range (len(self.filts)-1,0,-1): if (self.filts[-count]>HighestC): HighestC=self.filts[-count] if (self.filts[-count]<LowestC): LowestC=self.filts[-count] self.Stoc = (self.filts-LowestC)/(HighestC-LowestC) if len(self.Stoc)==1: self.MyStoch[0] = c1*(self.Stoc[0]+self.Stoc[0])/2 if len(self.Stoc)==2: self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] if (len(self.Stoc)>2): self.MyStoch = c1*(self.Stoc[0]+self.Stoc[-1])/2 + c2*self.MyStoch[-1] + c3*self.MyStoch[-2] # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close self.FiltClose = self.datas[1].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None self.rsi = bt.indicators.RSI_SMA(self.FiltClose, period=5,upperband=80,lowerband=20,safediv=True) self.stoch = MyStochastic(self.FiltClose) def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): print(self.stoch) # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.rsi[0] <= 20: # BUY, BUY, BUY!!! (with all possible default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: if self.rsi[0] >= 80: # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() cerebro.addstrategy(TestStrategy) datapath1 = ('C:/Users/Guwop/Desktop/Binance/BTCUSDT-AdjMar_Binance.csv') # Create a Data Feed data = bt.feeds.GenericCSVData( dataname=datapath1, fromdate=datetime.datetime(2018, 3, 8), todate=datetime.datetime(2018, 3, 25), dtformat = '%m/%d/%Y %H:%M',timeframe=bt.TimeFrame.Minutes ) datapath2 = ('C:/Users/Nathan/Desktop/Binance/BTCUSDT-FiltMar_Binance2.csv') Filt = bt.feeds.GenericCSVData( dataname=datapath2, fromdate = datetime.datetime(2018, 3, 8), todate = datetime.datetime(2018,3,25), dtformat = '%m/%d/%Y %H:%M', datetime=0, close=1, open=-1, high=-1, low=-1, volume=-1, openinterest=-1, timeframe=bt.TimeFrame.Minutes) Filt.plotinfo.plotmaster = data # Add the Data Feed to Cerebro cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes,compression=96) cerebro.replaydata(Filt, timeframe=bt.TimeFrame.Minutes,compression=96) # Set our desired cash start cerebro.broker.setcash(100000.0) cerebro.addsizer(bt.sizers.FixedSize, stake=1) cerebro.broker.setcommission(commission=0.001) # 0.1% ... divide by 100 to remove the % # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
-
@guwop said in Custom indicator/ understanding lines/list index out of range:
def __init__(self): lines = ('filts',)This is obviously out of place, so it is difficult to believe it is a working indicator.
@guwop said in Custom indicator/ understanding lines/list index out of range:
self.stoch = MyStochastic(self.FiltClose)Only 1 data feed is being provided to the indicator. It is easy to understand why
self.datas[1].closefails to work. The documentation link (Platform Concepts, see above), explains how to work and pass data feeds.Futhermore, you have already taken
closeas the reference you pass to your indicator, so trying to accesscloseagain is poised to fail.@guwop said in Custom indicator/ understanding lines/list index out of range:
HighestC = self.lines.filts LowestC = self.lines.filtsThis seems meaningless.
Please don't take this as discouraging but it seems
- You fail to understand what the input/s to the indicators is/are
- You fail to understand what the output/s of the indicator is/are
- Why some things are done in a declarative manner in
__init__and step-by-step calculations are done innext
This is because things are completely out of place and there are calculations all over the place that end up nowhere and which reference data feeds which are not being given to the indicator.
Let me suggest that before you keep on putting lines together, that you look at the source code of some indicators. Most are conceived as purely declarative (the entire logic is self-contained in
__init__with delayed operations), but there is one which could help:ZeroLagIndicator- See it here: https://github.com/backtrader/backtrader/blob/master/backtrader/indicators/zlind.py
-
@backtrader, You hit the nail on the head, I'm pretty clueless about how to develop my own indicator - all I seem to do is confuse myself further. Don't worry about me getting discouraged, I'm here to learn as much as I can and to hopefully be able to give back to the community. I really appreciate the work that y'all do.
With that said, thank you for linking the ZeroLagIndicator, as my indicator is an Ehlers as well, and so it helped my understand the use of datafeeds in referencing past values for the calculation of a new one.
I combed through the code between the backtrader version of ZeroLagIndicator and the Ehlers version on his site to better see how to implement my own. I have two questions:
-
In the for in range loop it has:
for value1 in range(*self.limits):
where
self.limits = [-self.p.gainlimit, self.p.gainlimit +1]I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?
-
Below is the code from the backtrader version:
def next(self): leasterror = MAXINT # 1000000 in original code bestec = ema = self.ema[0] # seed value 1st time for ec price = self.data[0] ec1 = self.lines.ec[-1] alpha, alpha1 = self.ema.alpha, self.ema.alpha1 for value1 in range(*self.limits): gain = value1 / 10 ec = alpha * (ema + gain * (price - ec1)) + alpha1 * ec1 error = abs(price - ec) if error < leasterror: leasterror = error bestec = ec self.lines.ec[0] = bestec
It seems as though only the bestec are chosen to be used for the line opposed to the ec of the best gain as from ehlers code. Now I realize that in the backtrader version bestgain was replaced with bestec , however, in the Ehlers version the EC is calculated with Bestgain in place of Gain . In other words, instead of the final line in backtrader version being:
self.lines.ec[0] = bestecit would instead be:
self.lines.ec[0] = alpha*(ema + BestGain * (price-ec1)) + (1-alpha)*ec1And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!
-
-
@guwop said in Custom indicator/ understanding lines/list index out of range:
And so I was wondering whether it does indeed do that and I'm just unaware, if it was done that way intentionally, or if it is a mistake. Thank you again!
There is no
BestGainin the code. Thebestecis updated if needed in the loop withec, which is derived from the currentgain. Only the best will finally be taken into account.@guwop said in Custom indicator/ understanding lines/list index out of range:
- In the for in range loop it has:
for value1 in range(*self.limits):where
self.limits = [-self.p.gainlimit, self.p.gainlimit +1]I was wondering what the use of the * operator in the arguments was used for? I couldn't find any documentation on it. Is it similar to SQL where it refers to ALL in a set? If so, why is simply self.limits not viable?
Let me be blunt. If you don't know what the
*operator is doing the code above, what you need is not to read the source of any backtrader indicator. You need work on your Python skills. The*is used to unpack things from an iterable. See some sources:
-
Just wanted to update that I got the stochastic to work as so:
class MyStochastic(bt.Indicator): lines = ('MyStoch','Stoc') def __init__(self): #period = 10 f = math.degrees(1.44 * 3.14159 / 10) a1 = math.exp(-f) b1 = 2 * a1 * math.cos(f) c2 = b1 c3 = -a1 * a1 self.c1 = 1 - c2 - c3 self.filts = self.datas[0] #Most recent close of Filt HighestC = self.filts LowestC = self.filts if(len(self.filts)>9): for count in range (9,0,-1): if (self.filts[-count]>HighestC): HighestC=self.filts[-count] if (self.filts[-count]<LowestC): LowestC=self.filts[-count] self.lines.Stoc = (self.filts-LowestC)/(HighestC-LowestC) S1 = self.lines.Stoc[-1] self.lines.MyStoch = self.c1*(Stoc+S1)/2 + c2*self.lines.MyStoch(-1) + c3*self.lines.MyStoch(-2)
-
Once again this is not to discourage you, but that indicator is actually doing nothing.
You are simply avoiding exceptions, but the indicator calculates no value at all.
Let me quote the
DummyIndicatorfrom the documentation which was linked above (for reference: https://www.backtrader.com/docu/inddev.html)class DummyInd(bt.Indicator): lines = ('dummyline',) params = (('value', 5),) def next(self): self.lines.dummyline[0] = max(0.0, self.params.value)You have no
nextmethod and your calculations are based on current values and not declarative (I prefer the declarative approach, but you may do it as you wish)The dummy indicator does nothing special but calculates the value for each iteration of the data.
In that documentation, there is also an example of how to calculate a
SimpleMovingAverageinnextand take into account the own period one wants to add to the mix (in your case it seems the default is9)
-
@backtrader, yeah I realized I prematurely posted my success - I had a different indicator printing itself. I'm going to look more into using max(), adding safediv to custom ind, and utilizing the next() method. Thank you for bearing with me
-
So I think I'm making some good progress, the issue I seem to be having now is that under the next() method I'm receiving the valueerror: "max() arg is an empty sequence". I understand this means that there is essentially nothing getting computed within it, but I'm not quite sure why. Could it possibly have something to do with how I used delayed indexing with MyStoch?
class MyStochastic(bt.Indicator): lines = ('MyStoch','Stoc','S1',) def __init__(self): self.f = math.degrees(1.44 * 3.14159 / 10) self.a1 = math.exp(-self.f) self.b1 = 2 * self.a1 * math.cos(self.f) self.c2 = self.b1 self.c3 = -self.a1 * self.a1 self.c1 = 1 - self.c2 - self.c3 self.filts = self.datas[0] #Most recent close of Filt HighestC = self.filts LowestC = self.filts if(len(self.filts)>9): for count in range (9,0,-1): if (self.filts[-count]>HighestC): HighestC=self.filts[-count] if (self.filts[-count]<LowestC): LowestC=self.filts[-count] Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC)) S1 = Stoc(-1) def next(self): self.lines.MyStoch[0] = max(self.c1*(self.Stoc+self.S1)/2 + self.c2*self.lines.MyStoch(-1) + self.c3*self.lines.MyStoch(-2))
-
@guwop what do you try to achieve in the for loop? This moment is totally unclear for me, everything else is more or less clear.
-
@ab_trader, the for loop is to determine what the Highest and Lowest values were for a given lookback period, and it does so by starting from the oldest and working to the newest
-
@guwop check the slicing approaches in Concepts section. With .get and max() operator you might have better coding in next().
Honestly I think that loop doesn't work as you want.
Also backtrader has built-in indicators to define highest high and lowest low data values for certain period.
-
@ab_trader, I think you're right about the loop, I switched [-count] to (-count) and got a Boolean error so I'm messing with that now. I'll have to look for those indicators, they sound very useful. Thank you for the help
-
As an idea (didn't test it):
class MyStochastic(bt.Indicator): lines = ('MyStoch', 'Stoc', 'S1',) params = (('period', 9),) def __init__(self): self.f = math.degrees(1.44 * 3.14159 / 10) self.a1 = math.exp(-self.f) self.b1 = 2 * self.a1 * math.cos(self.f) self.c2 = self.b1 self.c3 = -self.a1 * self.a1 self.c1 = 1 - self.c2 - self.c3 self.hhigh = bt.Highest(period = self.p.period) self.llow = bt.Lowest(period = self.p.period) self.l.Stoc = (self.datas[0].close - self.hhigh) / (self.hhigh - self.llow) self.l.S1 = self.Stoc(-1) self.l.MyStoch = self.c1 * (self.Stoc + self.S1) / 2 def next(self): self.l.MyStoch[0] = self.l.MyStoch[0] + self.c2 * self.l.MyStoch[-1] + self.c3 * self.l.MyStoch[-2]
-
@guwop said in Custom indicator/ understanding lines/list index out of range:
def __init__(self): ... if(len(self.filts)>9): for count in range (9,0,-1): if (self.filts[-count]>HighestC): HighestC=self.filts[-count] if (self.filts[-count]<LowestC): LowestC=self.filts[-count]The inherent problem here is: the loop never executes. When things are in
__init__things are being initialized. Data feeds have a null length (they are also in the initialization phase) and the condition will never beTrueYou miss the entire idea: only in
nextyou get to the next iteration of data.__init__is only called once during the entire lifecycle of the object.Stoc = bt.Max((self.filts-LowestC)/(HighestC-LowestC)) S1 = Stoc(-1)Maxis meant to use more than 1 argument or else the maximum is the given value. In any case and without a working loop the maximum in that case will be0/0which is undetermined and will raise an exception.@guwop said in Custom indicator/ understanding lines/list index out of range:
I'll have to look for those indicators, they sound very useful. Thank you for the help
You could also look at the source of the
Stochasticin backtrader.The usual approach for what you want to do:
- Subclass the existing indicator
- Add your extra lines
- Perform the extra calculations for the lines you have added.
-
Okay so I've changed some things around, and now I'm able to get values for Stoc, S1, c1...3, the only part that's giving me issues now is getting a value for MyStoch as it keeps returning Nan - I imagine it has something to do with calculating MyStoch[-1] and [-2], but I haven't come to figure out how yet. @backtrader , thanks for the clarification of init's one time use. You'd think it'd be obvious that initializing would occur only once, but I think it was the concept of lines objects getting generated from operations in init that threw me off. Nonetheless things are looking much better now. @ab_trader , I haven't got a chance to try your code yet, but I really appreciate the effort; I'll give it a shot at some point.
class MyStochastic(bt.Indicator): lines = ('MyStoch','Stoc','filts',) def __init__(self): self.f = math.degrees(1.44 * 3.14159 / 10) self.a1 = math.exp(-self.f) self.b1 = 2 * self.a1 * math.cos(self.f) self.c2 = self.b1 self.c3 = -self.a1 * self.a1 self.c1 = 1 - self.c2 - self.c3 self.addminperiod(11) super(MyStochastic,self).__init__() def next(self): self.filts[0] = self.data[0] self.HighestC = bt.indicators.Highest(period=9) self.LowestC = bt.indicators.Lowest(period=9) self.l.Stoc = (self.filts(0)-self.LowestC/(self.HighestC-self.LowestC)) S1 = self.Stoc[-1] self.lines.MyStoch[0] = (self.c1*(self.l.Stoc[0]+S1)/2 + self.c2*self.l.MyStoch[-1] + self.c3*self.l.MyStoch[-2])@ab_trader , tried your code that you posted and worked fine out of the box, however, I still couldn't get past the nan values when calculating MyStoch.
Any thoughts?
-
See here for a longer post