Hurst Exponent behavior in a Synthetic Trend Serie

[Paduel Gerion]

Hi, Backtrader is fabulous! Mementum, congratulations on your work.

I tried to replicate the analysis done by Duk2 in this article El Exponente de Hurst y la memoria de los precios en bolsa, referring to the exponent of Hurst, but using Backtrader.

In her article she calculates a single value of the Hurst exponent for all the time serie, but I tried to use the Backtrader rolling indicator with three different periods (which generate different lags).

For this I created a syntetic trend serie. But the result surprised me because in the three periodsHurst exponent is less than 0.5, when it should be close to 1.

Here the code:

from __future__ import (absolute_import, division, print_function,
                       unicode_literals)

import datetime 
import backtrader as bt
import pandas as pd
from numpy import *

class HurstST(bt.Strategy):
   params = (
   )

   def __init__(self):
       # Three Hurst Exponents with diferent periods
       hurst_1 = bt.ind.HurstExponent(period=40)
       hurst_2 = bt.ind.HurstExponent(period=200)
       hurst_3 = bt.ind.HurstExponent(period=1200)

   def next(self):
       pass


if __name__ == '__main__':

   cerebro = bt.Cerebro()
  
  # Create a synthetic trend time serie in a pandas dataframe
   tr = pd.DataFrame(pd.date_range(start='2005-1-1', end='2017-01-31', freq='D'), columns=['date'])
   tr['open'] = tr['high'] = tr['low'] = tr['close'] = pd.DataFrame(log(cumsum(random.randn(100000) + 1) + 1000))
   tr.set_index(['date'], inplace=True)

   # Pass the syntectic dataframe to the backtrader datafeed and add it to the cerebro
   data = bt.feeds.PandasDirectData(dataname=tr,
                                    volume=-1,
                                    openinterest=-1
                                    )

   cerebro.adddata(data)

   cerebro.addstrategy(HurstST)

   cerebro.run(stdstats=False)

   cerebro.plot()

Here the chart :

Is this a mistake of mine or some problem with the indicator?

Thank you.

[backtrader]

The same was stated when it was first developed. See Community - linear regression and std #211

It was, so to say, peer-reviewed. If you feel it's wrong (and that was the 1st opinion) and you can point to an implementation formula/description, it can be for sure changed.

[Paduel Gerion]

Hello,

I have checked that the code for the Hurst Exponent is correct and consistent with the method used. However the results are not correct due to the parameters used for the calculation.

Because the nature of the method the result is sensitive to the size of the sample.

Thus I have done simulations using synthetic series (Browian Geometrical Movement, Mean Reversal and Trend), and checking that the values of the Hurst exponent are not consistent with samples of less than about 2000 observations, being stable from about 3000 And 5000 observations according to the sample.

Similarly, the value used to define the lags must be less than about 10 and the upper value of the range should be above 500.

Backtrader uses a range for lags between 2 and half of sample size (period), and I have verified that it is not a good system because even with large samples and large periods is unstable.

The simulations can be reproduced and tested in this Jupyter Notebook:

I also propose to modify the calculation of the Exponent of Hurst in Backtrader with the following code, and to advise in the documentation that is not reliable with period values inferior to 2,000 bars.

Code :

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from . import PeriodN


__all__ = ['HurstExponent', 'Hurst']


class HurstExponent(PeriodN):

    frompackages = (
        ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')),
    )

    alias = ('Hurst',)
    lines = ('hurst',)
    params = (('period', 2000), ('lag_start', 10), ('lag_end', 500),)

    def __init__(self):
        super(HurstExponent, self).__init__()
        # Prepare the lags array
        self.lags = asarray(range(self.p.lag_start, self.p.lag_end))
        self.log10lags = log10(self.lags)

    def next(self):
        # Fetch the data
        ts = asarray(self.data.get(size=self.p.period))

        # Calculate the array of the variances of the lagged differences
        tau = [sqrt(std(subtract(ts[lag:], ts[:-lag]))) for lag in self.lags]

        # Use a linear fit to estimate the Hurst Exponent
        poly = polyfit(self.log10lags, log10(tau), 1)

        # Return the Hurst exponent from the polyfit output
        self.lines.hurst[0] = poly[0] * 2.0

In any case I find that this index is useful for an initial study of the series to determine what type of strategy is more propitious, but difficult to use in the strategy itself.

[backtrader]

Those are very reasonable proposals. Will be added.

[backtrader]

Looking into this and to avoid breaking the code anyone may have using this, the best alternative seems to:

• keep the lag_start and lag_end parameters you added but set them to None

  If any of both values is specified when adding the indicator to the mix, it will be used.

  Else: the original values will be used, namely: 2 and self.p.period // 2

And of course:

• Add the recommendations to carefully look into the values and propose longer periods like 2000 and the proposed 10 and 500 for the lag.

[backtrader]

Already in the development branch

[Paduel Gerion]

These is a great solution. Thanks for taking into account my proposals and include them in this great framework.