linear regression and std #211

backtrader

From Issue #211

Hi,

Could you include in the next release both linear regression and standard deviation? I think these indicators help people to calculate ratios over the time series.

S1= timeseries close
S2= timeseries close
rolling_beta = pd.ols(y=S1, x=S2, window_type='rolling', window=30)
spread = S2 - rolling_beta.beta['x'] * S1
std_30 = pd.Series.rolling(spread,window=30,center=False).std()

Rgds,
JJ

backtrader

Standard Deviation is already included (since many versions ago)

https://www.backtrader.com/docu/indautoref.html

junajo10

ok, thanks, i will wait up Standard Deviation. Congrats on the community!!!!

backtrader

There seem to be several linear regressions, including channels, slopes ...

junajo10

This code could be adapted to use two timeseries backtrader:
https://github.com/vikasrtr/pyLinearRegression/blob/master/models/LinearRegression.py
https://github.com/vikasrtr/pyLinearRegression/blob/master/models/LinearRegressionGradientDescent.py

junajo10

def linreg(X, Y):
    """

        Linear regression  y = ax + b

    """
    if len(X) != len(Y):  raise ValueError, 'unequal length'
    N = len(X)
    Sx = Sy = Sxx = Syy = Sxy = 0.0
    for x, y in map(None, X, Y):
        Sx = Sx + x
        Sy = Sy + y
        Sxx = Sxx + x*x
        Syy = Syy + y*y
        Sxy = Sxy + x*y
    det = Sxx * N - Sx * Sx
    a, b = (Sxy * N - Sy * Sx)/det, (Sxx * Sy - Sx * Sxy)/det
    meanerror = residual = 0.0
    for x, y in map(None, X, Y):
        meanerror = meanerror + (y - Sy/N)**2
        residual = residual + (y - a * x - b)**2
    RR = 1 - residual/meanerror
    ss = residual / (N-2)
    Var_a, Var_b = ss * N / det, ss * Sxx / det
    return a, b, RR

junajo10

My idea is to use linear regression to test these posts:

https://www.quantopian.com/posts/how-to-build-a-pairs-trading-strategy-on-quantopian

https://www.quantstart.com/articles/Backtesting-An-Intraday-Mean-Reversion-Pairs-Strategy-Between-SPY-And-IWM

https://www.quantinsti.com/blog/pair-trading-strategy-backtesting-using-quantstrat/

junajo10

if X and Y are cointegrated:

calculate Beta between X and Y 

calculate spread as X - Beta * Y

calculate z-score of spread

# entering trade (spread is away from mean by two sigmas):
if z-score > 2:
    sell spread (sell 1000 of X, buy 1000 * Beta of Y)
if z-score < -2:
    buy spread (buy 1000 of X, sell 1000 * Beta of Y)

# exiting trade (spread converged close to mean):
if we're short spread and z-score < 1:
    close the trades
if we're long spread and z-score > -1:
    close the trades

loop: repeat above on each new bar, recalculating rolling Beta and spread etc.

remroc

hi guys,

I am interested in that as well :D

Remroc

backtrader

It seems that the beta is the important part here

import pandas as pd
import backtrader as bt

class OLS_Beta(bt.indicators.PeriodN):
    _mindatas = 2  # ensure at least 2 data feeds are passed
    lines = (('beta'),)
    params = (('period', 30),)

    def next(self):
        y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
        r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
        self.lines.beta[0] = r_beta.beta['x']

or the spread directly

import pandas as pd
import backtrader as bt

class Spread(bt.indicators.PeriodN):
    _mindatas = 2  # ensure at least 2 data feeds are passed
    lines = (('spread'),)
    params = (('period', 30),)

    def next(self):
        y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
        r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
        self.lines.spread[0] = self.data1[0] - r_beta.beta['x'] * self.data0[0]

Is this in line?

junajo10

Yes, I think these classes are covering our requeriments

remroc

Awesome. I ll give it a try tonight...
U rock as usual, thanks Dro :astonished:

remroc

Hello DRo,

I was not able to run your proposition. I am receiving :

TypeError: 'builtin_function_or_method' object is not iterable

And, it seems pd.ols will be deprecated. I have tried with statsmodel.api :
http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/ols.html

import statsmodels.api as sm


class OLS_Transformation(btind.PeriodN):
    _mindatas = 2  # ensure at least 2 data feeds are passed
    lines = (('slope'),('intercept'),('spread'),('spread_mean'),('spread_std'),('zscore'),)
    params = (('period', 10),)

    def next(self):
        #y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
        p0 =  self.data0.get(size=self.p.period)
        p1 =  sm.add_constant(self.data1.get(size=self.p.period),prepend=True)
        slope, intercept = sm.OLS(p0,p1).fit().params
        #r_beta = pd.ols(y=p1, x=x, window_type='rolling', window=self.params.period)
        self.lines.slope[0] = slope
        self.lines.intercept[0] = intercept
        self.lines.spread[0] = self.data0.close[0] - (slope * self.data1.close[0] + intercept)
        self.lines.spread_mean[0] = btind.MovAv.SMA(self.lines.spread, period=self.p.period)
        self.lines.spread_std[0] = btind.StandardDeviation(self.lines.spread, period=self.p.period)
        self.lines.zscore[0] = (self.lines.spread[0] - self.lines.spread_mean[0])/self.lines.spread_std[0]

Do you have any recommandations for this path ?

Many thanks...

remroc

backtrader

Those were typed snippets no actual tested code. The actual line which produced the error would be helpful in understanding what part of the snippet is trying to iterate a builtin_function_or_method

remroc

Hello DRo,

Below is the received error by using the snippet.

For info, in code I am using only Class OLS_Beta and have initiated a self.beta signal in the Strategy's init through :

         self.beta = OLS_Beta(self.data0, self.data1)

The window parameter of pd.ols is window and not windows...

Many thanks for your insights :D

C:\Dev\Anaconda2\python.exe C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py
C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py:48: FutureWarning: The pandas.stats.ols module is deprecated and will be removed in a future version. We refer to external packages like statsmodels, see some examples here: http://statsmodels.sourceforge.net/stable/regression.html
  r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
Traceback (most recent call last):
  File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 329, in <module>
    runstrategy()
  File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 272, in runstrategy
    oldsync=args.oldsync)
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 809, in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 926, in runstrategies
    self._runonce(runstrats)
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 1245, in _runonce
    strat._once()
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 274, in _once
    indicator._once()
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 294, in _once
    self.oncestart(self._minperiod - 1, self._minperiod)
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\indicator.py", line 124, in oncestart_via_nextstart
    self.nextstart()
  File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 324, in nextstart
    self.next()
  File "C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py", line 48, in next
    r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\interface.py", line 143, in ols
    return klass(**kwargs)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 642, in __init__
    OLS.__init__(self, y=y, x=x, weights=weights, **self._args)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 70, in __init__
    self._index, self._time_has_obs) = self._prepare_data()
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 102, in _prepare_data
    self._weights_orig)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 1298, in _filter_data
    lhs = Series(lhs, index=rhs.index)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\series.py", line 137, in __init__
    index = _ensure_index(index)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 3409, in _ensure_index
    return Index(index_like)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 287, in __new__
    subarr = com._asarray_tuplesafe(data, dtype=object)
  File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\common.py", line 1384, in _asarray_tuplesafe
    values = list(values)
TypeError: 'builtin_function_or_method' object is not iterable

Process finished with exit code 1

backtrader

The problem with the snippet being that pd.ols chokes on regular Python array-like structures (array.array, list, etc) needing pandas specific structures.

The adapted and working code

class OLS_Beta(bt.indicators.PeriodN):
    _mindatas = 2  # ensure at least 2 data feeds are passed
    lines = (('beta'),)
    params = (('period', 30),)

    def next(self):
        y, x = (pd.Series(d.get(size=self.p.period)) for d in self.datas)
        r_beta = pd.ols(y=y, x=x, window_type='full_sample')
        self.lines.beta[0] = r_beta.beta['x']

In which the values from the dataX feeds is put into a pd.Series instance. There is, imho, no need to use a rolling operation because pd.ols only receives the needed data each time (the latest available data).

The same concept can be applied other pandas operations and I guess to the code ported over to statsmodel

remroc

@backtrader thanks DRo !

I will try this tomorrow...

remroc

Hello DRo,

Thanks, it is working. Pair Trading is operational...

I have implemented the statsmodel as well and seems to retrieve the same beta and spread...

Quick question, how to :

show 2 data.lines in the same subplot (ie PEP and KO in the same subplot) ?
increase the height of the indicator's subplot ?
plot only 1 line of a multi-lines indicator in a dedicated subplot ?

Many thanks,

RemRoc

backtrader

Plotting options are detailed here: https://www.backtrader.com/docu/plotting/plotting.html

remroc

@backtrader thanks,

Is it possible to contribute to backtrader by sharing the pair trading strategy as a sample of the distribution ?

If yes, please let me know how to do it ?

Thanks

Remroc