Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    linear regression and std #211

    Indicators/Strategies/Analyzers
    4
    41
    17503
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      backtrader administrators last edited by

      From Issue #211


      Hi,

      Could you include in the next release both linear regression and standard deviation? I think these indicators help people to calculate ratios over the time series.

      S1= timeseries close
      S2= timeseries close
      rolling_beta = pd.ols(y=S1, x=S2, window_type='rolling', window=30)
      spread = S2 - rolling_beta.beta['x'] * S1
      std_30 = pd.Series.rolling(spread,window=30,center=False).std()
      

      Rgds,
      JJ

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        Standard Deviation is already included (since many versions ago)

        • https://www.backtrader.com/docu/indautoref.html
        1 Reply Last reply Reply Quote 1
        • J
          junajo10 last edited by

          ok, thanks, i will wait up Standard Deviation. Congrats on the community!!!!

          1 Reply Last reply Reply Quote 1
          • B
            backtrader administrators last edited by

            There seem to be several linear regressions, including channels, slopes ...

            1 Reply Last reply Reply Quote 1
            • J
              junajo10 last edited by

              This code could be adapted to use two timeseries backtrader:
              https://github.com/vikasrtr/pyLinearRegression/blob/master/models/LinearRegression.py
              https://github.com/vikasrtr/pyLinearRegression/blob/master/models/LinearRegressionGradientDescent.py

              1 Reply Last reply Reply Quote 1
              • J
                junajo10 last edited by backtrader

                def linreg(X, Y):
                    """
                
                        Linear regression  y = ax + b
                
                    """
                    if len(X) != len(Y):  raise ValueError, 'unequal length'
                    N = len(X)
                    Sx = Sy = Sxx = Syy = Sxy = 0.0
                    for x, y in map(None, X, Y):
                        Sx = Sx + x
                        Sy = Sy + y
                        Sxx = Sxx + x*x
                        Syy = Syy + y*y
                        Sxy = Sxy + x*y
                    det = Sxx * N - Sx * Sx
                    a, b = (Sxy * N - Sy * Sx)/det, (Sxx * Sy - Sx * Sxy)/det
                    meanerror = residual = 0.0
                    for x, y in map(None, X, Y):
                        meanerror = meanerror + (y - Sy/N)**2
                        residual = residual + (y - a * x - b)**2
                    RR = 1 - residual/meanerror
                    ss = residual / (N-2)
                    Var_a, Var_b = ss * N / det, ss * Sxx / det
                    return a, b, RR
                
                1 Reply Last reply Reply Quote 1
                • J
                  junajo10 last edited by junajo10

                  My idea is to use linear regression to test these posts:

                  https://www.quantopian.com/posts/how-to-build-a-pairs-trading-strategy-on-quantopian

                  https://www.quantstart.com/articles/Backtesting-An-Intraday-Mean-Reversion-Pairs-Strategy-Between-SPY-And-IWM

                  https://www.quantinsti.com/blog/pair-trading-strategy-backtesting-using-quantstrat/

                  1 Reply Last reply Reply Quote 1
                  • J
                    junajo10 last edited by junajo10

                    if X and Y are cointegrated:
                    
                    calculate Beta between X and Y 
                    
                    calculate spread as X - Beta * Y
                    
                    calculate z-score of spread
                    
                    # entering trade (spread is away from mean by two sigmas):
                    if z-score > 2:
                        sell spread (sell 1000 of X, buy 1000 * Beta of Y)
                    if z-score < -2:
                        buy spread (buy 1000 of X, sell 1000 * Beta of Y)
                    
                    # exiting trade (spread converged close to mean):
                    if we're short spread and z-score < 1:
                        close the trades
                    if we're long spread and z-score > -1:
                        close the trades
                    
                    loop: repeat above on each new bar, recalculating rolling Beta and spread etc.
                    R 1 Reply Last reply Reply Quote 1
                    • R
                      remroc @junajo10 last edited by

                      hi guys,

                      I am interested in that as well :D

                      Remroc

                      1 Reply Last reply Reply Quote 0
                      • B
                        backtrader administrators last edited by backtrader

                        It seems that the beta is the important part here

                        import pandas as pd
                        import backtrader as bt
                        
                        class OLS_Beta(bt.indicators.PeriodN):
                            _mindatas = 2  # ensure at least 2 data feeds are passed
                            lines = (('beta'),)
                            params = (('period', 30),)
                        
                            def next(self):
                                y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
                                r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
                                self.lines.beta[0] = r_beta.beta['x']
                        

                        or the spread directly

                        import pandas as pd
                        import backtrader as bt
                        
                        class Spread(bt.indicators.PeriodN):
                            _mindatas = 2  # ensure at least 2 data feeds are passed
                            lines = (('spread'),)
                            params = (('period', 30),)
                        
                            def next(self):
                                y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
                                r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
                                self.lines.spread[0] = self.data1[0] - r_beta.beta['x'] * self.data0[0]
                        

                        Is this in line?

                        1 Reply Last reply Reply Quote 1
                        • J
                          junajo10 last edited by

                          Yes, I think these classes are covering our requeriments

                          1 Reply Last reply Reply Quote 0
                          • R
                            remroc last edited by

                            Awesome. I ll give it a try tonight...
                            U rock as usual, thanks Dro :astonished:

                            1 Reply Last reply Reply Quote 0
                            • R
                              remroc last edited by remroc

                              Hello DRo,

                              I was not able to run your proposition. I am receiving :

                              TypeError: 'builtin_function_or_method' object is not iterable
                              

                              And, it seems pd.ols will be deprecated. I have tried with statsmodel.api :
                              http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/ols.html

                              import statsmodels.api as sm
                              
                              
                              class OLS_Transformation(btind.PeriodN):
                                  _mindatas = 2  # ensure at least 2 data feeds are passed
                                  lines = (('slope'),('intercept'),('spread'),('spread_mean'),('spread_std'),('zscore'),)
                                  params = (('period', 10),)
                              
                                  def next(self):
                                      #y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
                                      p0 =  self.data0.get(size=self.p.period)
                                      p1 =  sm.add_constant(self.data1.get(size=self.p.period),prepend=True)
                                      slope, intercept = sm.OLS(p0,p1).fit().params
                                      #r_beta = pd.ols(y=p1, x=x, window_type='rolling', window=self.params.period)
                                      self.lines.slope[0] = slope
                                      self.lines.intercept[0] = intercept
                                      self.lines.spread[0] = self.data0.close[0] - (slope * self.data1.close[0] + intercept)
                                      self.lines.spread_mean[0] = btind.MovAv.SMA(self.lines.spread, period=self.p.period)
                                      self.lines.spread_std[0] = btind.StandardDeviation(self.lines.spread, period=self.p.period)
                                      self.lines.zscore[0] = (self.lines.spread[0] - self.lines.spread_mean[0])/self.lines.spread_std[0]
                              

                              Do you have any recommandations for this path ?

                              Many thanks...

                              remroc

                              1 Reply Last reply Reply Quote 0
                              • B
                                backtrader administrators last edited by

                                Those were typed snippets no actual tested code. The actual line which produced the error would be helpful in understanding what part of the snippet is trying to iterate a builtin_function_or_method

                                1 Reply Last reply Reply Quote 0
                                • R
                                  remroc last edited by remroc

                                  Hello DRo,

                                  Below is the received error by using the snippet.

                                  For info, in code I am using only Class OLS_Beta and have initiated a self.beta signal in the Strategy's init through :

                                           self.beta = OLS_Beta(self.data0, self.data1)
                                  

                                  The window parameter of pd.ols is window and not windows...

                                  Many thanks for your insights :D

                                  C:\Dev\Anaconda2\python.exe C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py
                                  C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py:48: FutureWarning: The pandas.stats.ols module is deprecated and will be removed in a future version. We refer to external packages like statsmodels, see some examples here: http://statsmodels.sourceforge.net/stable/regression.html
                                    r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
                                  Traceback (most recent call last):
                                    File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 329, in <module>
                                      runstrategy()
                                    File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 272, in runstrategy
                                      oldsync=args.oldsync)
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 809, in run
                                      runstrat = self.runstrategies(iterstrat)
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 926, in runstrategies
                                      self._runonce(runstrats)
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 1245, in _runonce
                                      strat._once()
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 274, in _once
                                      indicator._once()
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 294, in _once
                                      self.oncestart(self._minperiod - 1, self._minperiod)
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\indicator.py", line 124, in oncestart_via_nextstart
                                      self.nextstart()
                                    File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 324, in nextstart
                                      self.next()
                                    File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 48, in next
                                      r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\interface.py", line 143, in ols
                                      return klass(**kwargs)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 642, in __init__
                                      OLS.__init__(self, y=y, x=x, weights=weights, **self._args)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 70, in __init__
                                      self._index, self._time_has_obs) = self._prepare_data()
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 102, in _prepare_data
                                      self._weights_orig)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 1298, in _filter_data
                                      lhs = Series(lhs, index=rhs.index)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\series.py", line 137, in __init__
                                      index = _ensure_index(index)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 3409, in _ensure_index
                                      return Index(index_like)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 287, in __new__
                                      subarr = com._asarray_tuplesafe(data, dtype=object)
                                    File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\common.py", line 1384, in _asarray_tuplesafe
                                      values = list(values)
                                  TypeError: 'builtin_function_or_method' object is not iterable
                                  
                                  Process finished with exit code 1
                                  
                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    backtrader administrators last edited by backtrader

                                    The problem with the snippet being that pd.ols chokes on regular Python array-like structures (array.array, list, etc) needing pandas specific structures.

                                    The adapted and working code

                                    class OLS_Beta(bt.indicators.PeriodN):
                                        _mindatas = 2  # ensure at least 2 data feeds are passed
                                        lines = (('beta'),)
                                        params = (('period', 30),)
                                    
                                        def next(self):
                                            y, x = (pd.Series(d.get(size=self.p.period)) for d in self.datas)
                                            r_beta = pd.ols(y=y, x=x, window_type='full_sample')
                                            self.lines.beta[0] = r_beta.beta['x']
                                    
                                    

                                    In which the values from the dataX feeds is put into a pd.Series instance. There is, imho, no need to use a rolling operation because pd.ols only receives the needed data each time (the latest available data).

                                    The same concept can be applied other pandas operations and I guess to the code ported over to statsmodel

                                    R 1 Reply Last reply Reply Quote 1
                                    • R
                                      remroc @backtrader last edited by

                                      @backtrader thanks DRo !

                                      I will try this tomorrow...

                                      1 Reply Last reply Reply Quote 0
                                      • R
                                        remroc last edited by remroc

                                        Hello DRo,

                                        Thanks, it is working. Pair Trading is operational...

                                        I have implemented the statsmodel as well and seems to retrieve the same beta and spread...

                                        Quick question, how to :

                                        • show 2 data.lines in the same subplot (ie PEP and KO in the same subplot) ?
                                        • increase the height of the indicator's subplot ?
                                        • plot only 1 line of a multi-lines indicator in a dedicated subplot ?

                                        Many thanks,

                                        RemRoc

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          backtrader administrators last edited by

                                          Plotting options are detailed here: https://www.backtrader.com/docu/plotting/plotting.html

                                          R 1 Reply Last reply Reply Quote 0
                                          • R
                                            remroc @backtrader last edited by remroc

                                            @backtrader thanks,

                                            Is it possible to contribute to backtrader by sharing the pair trading strategy as a sample of the distribution ?

                                            If yes, please let me know how to do it ?

                                            Thanks

                                            Remroc

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 1 / 3
                                            • First post
                                              Last post
                                            Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                                            $(document).ready(function () { app.coldLoad(); }); }