Machine learning + backtrader

[junajo10]

Hi,

Do you have on your mind to add any machine learning library in backtrader or any ml sample?

Rgds,

Jj

[backtrader]

Not at the moment.

[junajo10]

Ok, thanks. In the future if you consider it please let me know. I would like to suggest some ideas

[RandyT]

Please suggest some ideas. This is an area I plan to go in near future after getting my head fully around current backtrader capability.

[revelator101]

What's stopping you from doing any machine learning? Why cant you just import the libraries, run the models & execute based on the parameters?

There's plenty of examples and ideas on Quantopian in python:

https://www.quantopian.com/posts/machine-learning-on-quantopian-part-3-building-an-algorithm

[RandyT]

Great thread. Thanks for sharing.

[junajo10]

Hi,

In the first phase the kind of things that i want to learn is the indicators that i should use depend on the asset class

https://www.linkedin.com/pulse/selecting-your-indicators-machine-learning-tad-slaff

Finally q-learning Could be the target

[ab_trader]

At this point machine learning opportunities are endless from both ML feature building and from ML algo selecting points of view. imho implementing and further support of special ML functions/procedures into bt will be just wasting of time.

[junajo10]

hi,

I was thinking more in defining a style guide that how to combine bt with the best ML libraries. For example in the regression linear example, the use is not direct .

[remroc]

@junajo10 I maybe we could give it a try if you have an article with an example...

[junajo10]

Here, i have found a case that could be a good starting point.

https://www.quantopian.com/posts/582de686358fe9440b0007f2#587e9dae9ea10879ef9ecb16

Kalman filter : Pairs trading

[remroc]

@junajo10 Thanks Junajo10 ! I need some documentation to understand this filter and it is related to ML ?

Do you have any tips ?

Thanks,

Remroc

[junajo10]

https://www.quantopian.com/posts/quantopian-lecture-series-kalman-filters

https://www.quantstart.com/articles/kalman-filter-based-pairs-trading-strategy-in-qstrader

http://numericalmethod.com/papers/course1/lecture5.pdf

[Maxim Korobov]

From 101 (csv, pandas) to some ML usage and only for trading + code, not just ML + code or trading theory.
https://www.udacity.com/course/machine-learning-for-trading--ud501

[remroc]

thanks guys. really good stuff :thumbsup:

[remroc]

Hi guys,

Here is an implementation of Kalman filter in a quantopian algo :
https://www.quantopian.com/posts/ernie-chans-ewa-slash-ewc-pair-trade-with-kalman-filter

Let's tackle this...

Remroc

[junajo10]

https://github.com/quantopian/research_public/blob/master/lectures/kalman_filter/kalman_filter_pair_trade.py

For me the problem is to migrate this code to backtrader .

[backtrader]

@junajo10 said in Machine learning + backtrader:

https://www.quantstart.com/articles/kalman-filter-based-pairs-trading-strategy-in-qstrader

Here is a version of the KalmanFilter described in that article (seems 1-to-1 identical to the one that implements Ernie Chan's strategy)

import numpy as np
import pandas as pd


class KalmanFilter(bt.Indicator):
    _mindatas = 2  # needs at least 2 data feeds

    lines = ('et', 'sqrt_qt',)

    params = dict(
        delta=1e-4,
        vt=1e-3,
    )

    def __init__(self):
        self.wt = self.p.delta / (1 - self.p.delta) * np.eye(2)
        self.theta = np.zeros(2)
        self.P = np.zeros((2, 2))
        self.R = None

        self.d1_prev = self.data1(-1)  # data1 yesterday's price

    def next(self):
        F = np.asarray([self.data0[0], 1.0]).reshape((1, 2))
        y = self.d1_prev[0]

        if self.R is not None:  # self.R starts as None, self.C set below
            self.R = self.C + self.wt
        else:
            self.R = np.zeros((2, 2))

        yhat = F.dot(self.theta)
        et = y - yhat

        # Q_t is the variance of the prediction of observations and hence
        # \sqrt{Q_t} is the standard deviation of the predictions
        Qt = F.dot(self.R).dot(F.T) + self.p.vt
        sqrt_Qt = np.sqrt(Qt)

        # The posterior value of the states \theta_t is distributed as a
        # multivariate Gaussian with mean m_t and variance-covariance C_t
        At = self.R.dot(F.T) / Qt
        self.theta = self.theta + At.flatten() * et
        self.C = self.R - At * F.dot(self.R)

        # Fill the lines
        self.lines.et[0] = et
        self.l.sqrt_qt[0] = sqrt_Qt

The filter per-se is not directly used in the decision making process. Knowing (from the article) which relations et and sqrt_qt need to have to trigger signals, the behavior can be packed in another indicator

class KalmanSignals(bt.Indicator):
    _mindatas = 2  # needs at least 2 data feeds

    lines = ('long', 'short',)

    def __init__(self):
        kf = KalmanFilter(self.data0, self.data1)
        et, sqrt_qt = kf.lines.et, kf.lines.sqrt_qt

        self.lines.long = et < -1.0 * sqrt_qt
        # longexit is et > -1.0 * sqrt_qt ... the opposite of long
        self.lines.short = et > sqrt_qt
        # shortexit is et < sqrt_qt ... the opposite of short

And the signals can be easily applied in the strategy next method for an easy to follow logic

class St(bt.Strategy):
    params = (
    )

    def __init__(self):
        self.ksig = KalmanSignals(self.data0, self.data1)

    def next(self):
        size = self.position.size
        if not size:
            if self.ksig.long:
                self.buy()
            elif self.ksig.short:
                self.sell()

        elif size > 0:
            if not self.ksig.long:
                self.close()
        elif not self.ksig.short:  # implicit size < 0
            self.close()

The difficult thing here is to select assets for which it makes sense to apply the filter.

[junajo10]

Thanks @backtrader .

@backtrader said "The difficult thing here is to select assets for which it makes sense to apply the filter".

For solving thsi problem use these tecniques:
Augmented Dickey Fuller
Hurst exponent
Half Life

[Maxim Korobov]

@backtrader, add this filter to main package.