Hurst Exponent behavior in a Synthetic Trend Serie



  • Hi, Backtrader is fabulous! Mementum, congratulations on your work.

    I tried to replicate the analysis done by Duk2 in this article El Exponente de Hurst y la memoria de los precios en bolsa, referring to the exponent of Hurst, but using Backtrader.

    In her article she calculates a single value of the Hurst exponent for all the time serie, but I tried to use the Backtrader rolling indicator with three different periods (which generate different lags).

    For this I created a syntetic trend serie. But the result surprised me because in the three periodsHurst exponent is less than 0.5, when it should be close to 1.

    Here the code:

    from __future__ import (absolute_import, division, print_function,
                           unicode_literals)
    
    import datetime 
    import backtrader as bt
    import pandas as pd
    from numpy import *
    
    class HurstST(bt.Strategy):
       params = (
       )
    
       def __init__(self):
           # Three Hurst Exponents with diferent periods
           hurst_1 = bt.ind.HurstExponent(period=40)
           hurst_2 = bt.ind.HurstExponent(period=200)
           hurst_3 = bt.ind.HurstExponent(period=1200)
    
       def next(self):
           pass
    
    
    if __name__ == '__main__':
    
       cerebro = bt.Cerebro()
      
      # Create a synthetic trend time serie in a pandas dataframe
       tr = pd.DataFrame(pd.date_range(start='2005-1-1', end='2017-01-31', freq='D'), columns=['date'])
       tr['open'] = tr['high'] = tr['low'] = tr['close'] = pd.DataFrame(log(cumsum(random.randn(100000) + 1) + 1000))
       tr.set_index(['date'], inplace=True)
    
       # Pass the syntectic dataframe to the backtrader datafeed and add it to the cerebro
       data = bt.feeds.PandasDirectData(dataname=tr,
                                        volume=-1,
                                        openinterest=-1
                                        )
    
       cerebro.adddata(data)
    
       cerebro.addstrategy(HurstST)
    
       cerebro.run(stdstats=False)
    
       cerebro.plot()
    

    Here the chart :

    0_1495109824708_Hurst trend.png

    Is this a mistake of mine or some problem with the indicator?

    Thank you.


  • administrators

    The same was stated when it was first developed. See Community - linear regression and std #211

    It was, so to say, peer-reviewed. If you feel it's wrong (and that was the 1st opinion) and you can point to an implementation formula/description, it can be for sure changed.



  • Hello,

    I have checked that the code for the Hurst Exponent is correct and consistent with the method used. However the results are not correct due to the parameters used for the calculation.

    Because the nature of the method the result is sensitive to the size of the sample.

    Thus I have done simulations using synthetic series (Browian Geometrical Movement, Mean Reversal and Trend), and checking that the values of the Hurst exponent are not consistent with samples of less than about 2000 observations, being stable from about 3000 And 5000 observations according to the sample.

    Similarly, the value used to define the lags must be less than about 10 and the upper value of the range should be above 500.

    Backtrader uses a range for lags between 2 and half of sample size (period), and I have verified that it is not a good system because even with large samples and large periods is unstable.

    The simulations can be reproduced and tested in this Jupyter Notebook:

    I also propose to modify the calculation of the Exponent of Hurst in Backtrader with the following code, and to advise in the documentation that is not reliable with period values inferior to 2,000 bars.

    Code :

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    from . import PeriodN
    
    
    __all__ = ['HurstExponent', 'Hurst']
    
    
    class HurstExponent(PeriodN):
    
        frompackages = (
            ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')),
        )
    
        alias = ('Hurst',)
        lines = ('hurst',)
        params = (('period', 2000), ('lag_start', 10), ('lag_end', 500),)
    
        def __init__(self):
            super(HurstExponent, self).__init__()
            # Prepare the lags array
            self.lags = asarray(range(self.p.lag_start, self.p.lag_end))
            self.log10lags = log10(self.lags)
    
        def next(self):
            # Fetch the data
            ts = asarray(self.data.get(size=self.p.period))
    
            # Calculate the array of the variances of the lagged differences
            tau = [sqrt(std(subtract(ts[lag:], ts[:-lag]))) for lag in self.lags]
    
            # Use a linear fit to estimate the Hurst Exponent
            poly = polyfit(self.log10lags, log10(tau), 1)
    
            # Return the Hurst exponent from the polyfit output
            self.lines.hurst[0] = poly[0] * 2.0
    

    In any case I find that this index is useful for an initial study of the series to determine what type of strategy is more propitious, but difficult to use in the strategy itself.


  • administrators

    Those are very reasonable proposals. Will be added.


  • administrators

    Looking into this and to avoid breaking the code anyone may have using this, the best alternative seems to:

    • keep the lag_start and lag_end parameters you added but set them to None

      If any of both values is specified when adding the indicator to the mix, it will be used.

      Else: the original values will be used, namely: 2 and self.p.period // 2

    And of course:

    • Add the recommendations to carefully look into the values and propose longer periods like 2000 and the proposed 10 and 500 for the lag.

  • administrators

    Already in the development branch



  • These is a great solution. Thanks for taking into account my proposals and include them in this great framework.



Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.