For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

# linear regression and std #211

• From Issue #211

Hi,

Could you include in the next release both linear regression and standard deviation? I think these indicators help people to calculate ratios over the time series.

``````S1= timeseries close
S2= timeseries close
rolling_beta = pd.ols(y=S1, x=S2, window_type='rolling', window=30)
spread = S2 - rolling_beta.beta['x'] * S1
``````

Rgds,
JJ

• Standard Deviation is already included (since many versions ago)

• ok, thanks, i will wait up Standard Deviation. Congrats on the community!!!!

• There seem to be several linear regressions, including channels, slopes ...

• ``````def linreg(X, Y):
"""

Linear regression  y = ax + b

"""
if len(X) != len(Y):  raise ValueError, 'unequal length'
N = len(X)
Sx = Sy = Sxx = Syy = Sxy = 0.0
for x, y in map(None, X, Y):
Sx = Sx + x
Sy = Sy + y
Sxx = Sxx + x*x
Syy = Syy + y*y
Sxy = Sxy + x*y
det = Sxx * N - Sx * Sx
a, b = (Sxy * N - Sy * Sx)/det, (Sxx * Sy - Sx * Sxy)/det
meanerror = residual = 0.0
for x, y in map(None, X, Y):
meanerror = meanerror + (y - Sy/N)**2
residual = residual + (y - a * x - b)**2
RR = 1 - residual/meanerror
ss = residual / (N-2)
Var_a, Var_b = ss * N / det, ss * Sxx / det
return a, b, RR
``````

• ``````if X and Y are cointegrated:

calculate Beta between X and Y

calculate spread as X - Beta * Y

if z-score > 2:
sell spread (sell 1000 of X, buy 1000 * Beta of Y)
if z-score < -2:

if we're short spread and z-score < 1:
if we're long spread and z-score > -1:

loop: repeat above on each new bar, recalculating rolling Beta and spread etc.``````

• hi guys,

I am interested in that as well :D

Remroc

• It seems that the beta is the important part here

``````import pandas as pd

class OLS_Beta(bt.indicators.PeriodN):
_mindatas = 2  # ensure at least 2 data feeds are passed
lines = (('beta'),)
params = (('period', 30),)

def next(self):
y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
self.lines.beta = r_beta.beta['x']
``````

``````import pandas as pd

_mindatas = 2  # ensure at least 2 data feeds are passed
params = (('period', 30),)

def next(self):
y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
self.lines.spread = self.data1 - r_beta.beta['x'] * self.data0
``````

Is this in line?

• Yes, I think these classes are covering our requeriments

• Awesome. I ll give it a try tonight...
U rock as usual, thanks Dro :astonished:

• Hello DRo,

I was not able to run your proposition. I am receiving :

``````TypeError: 'builtin_function_or_method' object is not iterable
``````

And, it seems pd.ols will be deprecated. I have tried with statsmodel.api :
http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/ols.html

``````import statsmodels.api as sm

class OLS_Transformation(btind.PeriodN):
_mindatas = 2  # ensure at least 2 data feeds are passed
params = (('period', 10),)

def next(self):
#y, x = (d.get(size=self.p.period) for d in (self.data0, self.data1))
p0 =  self.data0.get(size=self.p.period)
slope, intercept = sm.OLS(p0,p1).fit().params
#r_beta = pd.ols(y=p1, x=x, window_type='rolling', window=self.params.period)
self.lines.slope = slope
self.lines.intercept = intercept
self.lines.spread = self.data0.close - (slope * self.data1.close + intercept)
``````

Do you have any recommandations for this path ?

Many thanks...

remroc

• Those were typed snippets no actual tested code. The actual line which produced the error would be helpful in understanding what part of the snippet is trying to iterate a `builtin_function_or_method`

• Hello DRo,

Below is the received error by using the snippet.

For info, in code I am using only Class OLS_Beta and have initiated a self.beta signal in the Strategy's init through :

``````         self.beta = OLS_Beta(self.data0, self.data1)
``````

The window parameter of pd.ols is window and not windows...

Many thanks for your insights :D

``````C:\Dev\Anaconda2\python.exe C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py
C:/Trading/backtrader-master-1.9.8.99-David/samples/pair-trading/pair-trading.py:48: FutureWarning: The pandas.stats.ols module is deprecated and will be removed in a future version. We refer to external packages like statsmodels, see some examples here: http://statsmodels.sourceforge.net/stable/regression.html
r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
Traceback (most recent call last):
runstrategy()
oldsync=args.oldsync)
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 809, in run
runstrat = self.runstrategies(iterstrat)
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 926, in runstrategies
self._runonce(runstrats)
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 1245, in _runonce
strat._once()
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 274, in _once
indicator._once()
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 294, in _once
self.oncestart(self._minperiod - 1, self._minperiod)
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\indicator.py", line 124, in oncestart_via_nextstart
self.nextstart()
File "C:\Dev\Anaconda2\lib\site-packages\backtrader\lineiterator.py", line 324, in nextstart
self.next()
r_beta = pd.ols(y=y, x=x, window_type='rolling', window=self.p.period)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\interface.py", line 143, in ols
return klass(**kwargs)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 642, in __init__
OLS.__init__(self, y=y, x=x, weights=weights, **self._args)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 70, in __init__
self._index, self._time_has_obs) = self._prepare_data()
File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 102, in _prepare_data
self._weights_orig)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\stats\ols.py", line 1298, in _filter_data
lhs = Series(lhs, index=rhs.index)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\series.py", line 137, in __init__
index = _ensure_index(index)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 3409, in _ensure_index
return Index(index_like)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\indexes\base.py", line 287, in __new__
subarr = com._asarray_tuplesafe(data, dtype=object)
File "C:\Dev\Anaconda2\lib\site-packages\pandas\core\common.py", line 1384, in _asarray_tuplesafe
values = list(values)
TypeError: 'builtin_function_or_method' object is not iterable

Process finished with exit code 1
``````

• The problem with the snippet being that `pd.ols` chokes on regular Python array-like structures (`array.array`, `list`, etc) needing `pandas` specific structures.

``````class OLS_Beta(bt.indicators.PeriodN):
_mindatas = 2  # ensure at least 2 data feeds are passed
lines = (('beta'),)
params = (('period', 30),)

def next(self):
y, x = (pd.Series(d.get(size=self.p.period)) for d in self.datas)
r_beta = pd.ols(y=y, x=x, window_type='full_sample')
self.lines.beta = r_beta.beta['x']

``````

In which the values from the `dataX` feeds is put into a `pd.Series` instance. There is, imho, no need to use a rolling operation because `pd.ols` only receives the needed data each time (the latest available data).

The same concept can be applied other `pandas` operations and I guess to the code ported over to `statsmodel`

I will try this tomorrow...

• Hello DRo,

Thanks, it is working. Pair Trading is operational...

I have implemented the statsmodel as well and seems to retrieve the same beta and spread...

Quick question, how to :

• show 2 data.lines in the same subplot (ie PEP and KO in the same subplot) ?
• increase the height of the indicator's subplot ?
• plot only 1 line of a multi-lines indicator in a dedicated subplot ?

Many thanks,

RemRoc

• Plotting options are detailed here: https://www.backtrader.com/docu/plotting/plotting.html