DivideZeroError on some indicators

sfkiwi

First off, there is a related post on DivideZeroError here but it doesn't seem to answer the issue that I am experiencing.

I have quite a large number of indicators in my strategy but some of them keep throwing the same DivideZeroError. As far as I can tell I'm using the indicators correctly, for example BollingerBands are fairly widely used.

BollingerBands as far as I can tell doesn't have any division in it but every time I enable it I get this same error.

~/miniconda3/envs/pythontrading/lib/python3.5/site-packages/backtrader/linebuffer.py in next(self)
    743     def next(self):
    744         if self.bline:
--> 745             self[0] = self.operation(self.a[0], self.b[0])
    746         elif not self.r:
    747             if not self.btime:

ZeroDivisionError: float division by zero

I have tried putting a try/Except block around both the init and the next functions and neither catch the error so it seems like its happening somewhere else but only gets triggered when I enable these specific indicators.

I am running a minute timescale datafeed which is then resampled to 5M, 30M, 60M 4H.

backtrader

We don't need to see where you trigger the error inside the library. We need to see the code with which you trigger the error (and associated data)

sfkiwi

@backtrader

Here is the associated code. I'm not sure exactly where the error is being triggered. It looks to me to be in the next() function of the strategy which triggers an operation on an indicator, however the indicators that seem to be causing the problem are not accessed in the Strategy.

# Inside Strategy

def __init__(self):

  # These indicators don't cause any problem
  self.trend = btind.SMA(self.data3, period=10); self.trend.csv =self.p.l
  self.trendslope = self.trend - self.trend(-1); self.trendslope.csv = self.p.l

  self.r1 = btind.RSI(self.data0, plotname='rsi_1M'); self.r1.csv = self.p.l
  self.r5 = btind.RSI(self.data1, plotname='rsi_5M'); self.r5.csv = self.p.l
  self.r30 = btind.RSI(self.data2, plotname='rsi_30M'); self.r30.csv = self.p.l
  self.r60 = btind.RSI(self.data3, plotname='rsi_1H'); self.r60.csv = self.p.l
  self.r240 = btind.RSI(self.data4, plotname='rsi_4H'); self.r240.csv = self.p.l
        
  self.s1 = StochasticRSI(self.r1, plotname='sto_1M'); self.s1.csv = self.p.l
  self.s5 = StochasticRSI(self.r5, plotname='sto_5M'); self.s5.csv = self.p.l
  self.s30 = StochasticRSI(self.r30, plotname='sto_30M'); self.s30.csv = self.p.l
  self.s60 = StochasticRSI(self.r60, plotname='sto_1H'); self.s60.csv = self.p.l
  self.s240 = StochasticRSI(self.r240, plotname='sto_4H'); self.s240.csv = self.p.l
        
  btind.ADX(plot=True).csv = self.p.l
  btind.ADXR(plot=False).csv = self.p.l
  btind.ATR(plot=True).csv = self.p.l
  btind.AbsPriceOsc(plot=False).csv = self.p.l
  btind.AccDeOsc(plot=True).csv = self.p.l

  #####
  # Enabling any of the following indicators causes the DivideZeroError, but 
  # there are other indicators not shown here that also cause the problem
  btind.AdaptiveMovingAverageOsc(plot=False).csv = self.p.l
  btind.AdaptiveMovingAverage(plot=False).csv = self.p.l
  btind.BBands(plot=False).csv = self.p.l
  ######

def next(self):
  if self.orefs:
       return

  if not self.position:
    
    # Long Entry
    if self.s240.D[0] < 20 and self.trendslope > 0:

      if ((self.s60.D[0] > self.s60.K[0]) and 
        (self.s30.D[0] > self.s30.K[0]) and 
        (self.s5.D[0] > self.s5.K[0])):

        if (self.s1.K[0] < 20) and (self.s1.D[0] > self.s1.K[0]):

          os = self.buy_bracket(
            exectype=bt.Order.Market, 
            stopprice=self.data0.close[0] * (1 - self.p.sl), 
            limitprice=self.data0.close[0] * (1 + self.p.tp))

          self.orefs = [o.ref for o in os]

def getwriterheaders(self):
  self.indobscsv = [self]
            
  indobs = itertools.chain(
    self.getindicators_lines(), self.getobservers())
    self.indobscsv.extend(filter(lambda x: x.csv, indobs))
            
  headers = list()
            
  # prepare the indicators/observers data headers
  for iocsv in self.indobscsv:
    name = iocsv.plotinfo.plotname or iocsv.__class__.__name__
    linealiases = iocsv.getlinealiases()
            
    def joinnames(alias):
      if alias:
        join = name + '_' + alias
        return join
      else:
        return alias
                                
    aliases = tuple(map(joinnames, linealiases))
    headers.extend(aliases)
            
    return headers
                
def getwritervalues(self):
  values = list()
            
  for iocsv in self.indobscsv:
    name = iocsv.plotinfo.plotname or iocsv.__class__.__name__
    lio = len(iocsv)
    if lio:
      values.extend(map(lambda l: l[0], iocsv.lines.itersize()))
    else:
      values.extend([''] * iocsv.lines.size())
            
    return values

# Running the Strategy
cerebro = bt.Cerebro()
cerebro.addstrategy(St, tp=0.005, sl=0.007, printlog=True, l=True)
cerebro.addwriter(bt.WriterFile, csv=True, csv_counter=False, out='log.csv')

# Data is a Pandas DataFrame of 1min OHLCV
data = bt.feeds.PandasData(dataname=candles_test, openinterest=None, timeframe=bt.TimeFrame.Minutes)
cerebro.adddata(data)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=5, boundoff=1)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=30, boundoff=1)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=60, boundoff=1)
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=240, boundoff=1)
cerebro.broker.setcash(20000.0)
cerebro.addsizer(bt.sizers.PercentSizer, percents=98)
cerebro.broker.setcommission(commission=0.00125)
cerebro.run(runonce=False)

sfkiwi

@backtrader

Here is a log of the close price of the data just before the error occurs. You can see that there is a period where the price doesn't change for 19 consecutive minutes.

Only the RSI and Stochastic Indicators implement SafeDiv and these indicators don't seem to cause the issue to occur. Its other indicators like BollingerBands that trigger this error when enabled.

2018-05-10 03:25:00, running 9286.70
2018-05-10 03:26:00, running 9291.65
2018-05-10 03:27:00, running 9291.65
2018-05-10 03:28:00, running 9291.70
2018-05-10 03:29:00, running 9291.70
2018-05-10 03:30:00, running 9291.70
2018-05-10 03:31:00, running 9291.70
2018-05-10 03:32:00, running 9291.70
2018-05-10 03:33:00, running 9291.70
2018-05-10 03:34:00, running 9291.70
2018-05-10 03:35:00, running 9291.70
2018-05-10 03:36:00, running 9291.70
2018-05-10 03:37:00, running 9291.70
2018-05-10 03:38:00, running 9291.70
2018-05-10 03:39:00, running 9291.70
2018-05-10 03:40:00, running 9291.70
2018-05-10 03:41:00, running 9291.70
2018-05-10 03:42:00, running 9291.70
2018-05-10 03:43:00, running 9291.70
2018-05-10 03:44:00, running 9291.70
2018-05-10 03:45:00, running 9291.70
2018-05-10 03:46:00, running 9291.70

Edit:
One of the indicators that is causing a problem is the BollingerBandsPct. Looking into the code for the indicator it seems that if the close price doesn't change for more than the period of the indicator (default period=20) then the line.top and line.bottom will both be equal to the close price as there is no stddev for a series of values that are all the same (stddev=0).

If both line.top and line.bottom are equal to each other then in the Percent calculation the denominator will be top - bottom = 0.

class BollingerBands(Indicator):
    '''
    Defined by John Bollinger in the 80s. It measures volatility by defining
    upper and lower bands at distance x standard deviations

    Formula:
      - midband = SimpleMovingAverage(close, period)
      - topband = midband + devfactor * StandardDeviation(data, period)
      - botband = midband - devfactor * StandardDeviation(data, period)

    See:
      - http://en.wikipedia.org/wiki/Bollinger_Bands
    '''
    alias = ('BBands',)

    lines = ('mid', 'top', 'bot',)
    params = (('period', 20), ('devfactor', 2.0), ('movav', MovAv.Simple),)

    plotinfo = dict(subplot=False)
    plotlines = dict(
        mid=dict(ls='--'),
        top=dict(_samecolor=True),
        bot=dict(_samecolor=True),
    )

    def _plotlabel(self):
        plabels = [self.p.period, self.p.devfactor]
        plabels += [self.p.movav] * self.p.notdefault('movav')
        return plabels

    def __init__(self):
        self.lines.mid = ma = self.p.movav(self.data, period=self.p.period)

        #########################################
        # if the last 20 closing prices are all the same then StdDev will evaluate 
        # to zero
        #########################################
        stddev = self.p.devfactor * StdDev(self.data, ma, period=self.p.period,
                                           movav=self.p.movav)

        #########################################
        # both self.lines.top and self.lines.bottom will be equal to the ma
        # which is now equal to the close price
        #########################################
        self.lines.top = ma + stddev
        self.lines.bot = ma - stddev

        super(BollingerBands, self).__init__()


class BollingerBandsPct(BollingerBands):
    '''
    Extends the Bollinger Bands with a Percentage line
    '''
    lines = ('pctb',)
    plotlines = dict(pctb=dict(_name='%B'))  # display the line as %B on chart

    def __init__(self):
        super(BollingerBandsPct, self).__init__()

        #########################################
        # The denominator here will evaluate to zero as self.l.top
        # and self.l.bot are the same.
        #########################################
        self.l.pctb = (self.data - self.l.bot) / (self.l.top - self.l.bot)

Edit2:
I duplicated the BollingerBandsPct in my own code and replaced the percent calculation with bt.DivByZero and it resolved the problem for this indicator. Unfortunately there are many others similar to this.

Is there a particular reason why only the Stochastic and the Rsi indicators use the safeDiv function and not the other indicators?

class MyBollingerBandsPct(btind.BollingerBands):
    '''
    Extends the Bollinger Bands with a Percentage line
    '''
    lines = ('pctb',)
    plotlines = dict(pctb=dict(_name='%B'))  # display the line as %B on chart

    def __init__(self):
        super(MyBollingerBandsPct, self).__init__()
        self.l.pctb = bt.DivByZero((self.data - self.l.bot), (self.l.top - self.l.bot), zero=0)

backtrader

@sfkiwi said in DivideZeroError on some indicators:

it seems that if the close price doesn't change for more than the period of the indicator (default period=20) then the line.top and line.bottom will both be equal to the close price as there is no stddev for a series of values that are all the same (stddev=0).

Indeed. Hence the reason that you show what you do and with what you do it.

sfkiwi

@backtrader

Ok but, unlikely as it is, the price stayed stationary for more than 20mins so shouldn't the indicators still be protected from a divide by zero error?

I am adding this protection to my fork of the latest code in the repo although having some difficulty getting the latest code in the repo to run without error. Once I get it to work I can submit a PR unless there is a reason you don't want this protection on the indicators?

backtrader

@sfkiwi said in DivideZeroError on some indicators:

Ok but, unlikely as it is, the price stayed stationary for more than 20mins so shouldn't the indicators still be protected from a divide by zero error?

Nobody has claimed nothing about the price or the protection. Only that you said something was broken without posting any code or data. And when the code and data were posted, the error was somewhere else. Hence the need to post something sensible.

class BollingerBandsPct(BollingerBands):
    '''
    Extends the Bollinger Bands with a Percentage line
    '''
    lines = ('pctb',)
    params = (('safediv', False), ('safezero', 0.0))
    plotlines = dict(pctb=dict(_name='%B'))  # display the line as %B on chart

    def __init__(self):
        super(BollingerBandsPct, self).__init__()
        if not self.p.safediv:
            self.l.pctb = (self.data - self.l.bot) / (self.l.top - self.l.bot)
        else:
            self.l.pctb = DivByZero(self.data - self.l.bot, self.l.top - self.l.bot, self.p.safezero)

class BollingerBandsPct_SafeDiv(BollingerBandsPct):
    params = (('safediv', True),)

That's something you can directly use in your code. There is no need for a fork to do that.

scottz1

I get ZeroDivisionError in strategy init with this line:

        self.pctb = bt.indicators.BollingerBandsPct(self.datas[0], period=5, devfactor=1.0, plot=False)

No code in the strategy uses the indicator, and commenting that line out, the strategy runs.
I tried adding the above class for BBPct with safediv, but I don't know what imports I need for "BollingerBands" and "DivByZero" in the framework.

backtrader

@scottz1 said in DivideZeroError on some indicators:

but I don't know what imports I need for "BollingerBands"

BollingerBands is a standard indicator available in backtrader.indicators

@scottz1 said in DivideZeroError on some indicators:
and "DivByZero" in the framework.

DivByZero is a helper function in backtrader

HONGYI ZHENG

@sfkiwi I also use lots of indicators in my framework, hence modifying all of them for "zero division protection" is inefficient. Then I found a approach called "monkey-patching" (https://stackoverflow.com/questions/10429547/how-to-change-a-function-in-existing-3rd-party-library-in-python), by which we can directly and dynamically modify the error-causing line in "linebuffer.py" module. By this way, we only need to do the modification once, instead of for each indicator.

Ilan Attar

@HONGYI-ZHENG Any chance you could share an example of how you did this?
Any unexpected downsides?