Kalman et al.

backtrader

Use the link below to go the original post

Click here to see the full blog post

junajo10

self.P = np.zeros((2, 2)) should be self.C?

backtrader

May well be. Will be checked.

junajo10

Also, kf = KalmanFilterInd(self.data0, self.data1)

backtrader

That was a just a test to see how the prediction to itself would look like.

If you have a look at KalmanSignals you will see:

KalmanFilterInd()

with even no mention of the data feeds. This passes whatever data feeds are in the parent down to KalmanFilterInd and this is equal to KalmanFilterInd(self.data0, self.data1)

But thank you for spotting it, it is for sure worth some rewriting to make all options more comprehensible.

remroc

WOW I was just absent 2 weeks and that BOOM. Thanks DRo and Junajo10. U rock guys !!

backtrader

@junajo10 said in Kalman et al.:

self.P = np.zeros((2, 2)) should be self.C?

The original code has indeed self.P = np.zeros((2, 2)). Even if actually meant self.C = np.zeros((2, 2)) it doesn't play any role.

The 1st value of self.R is None, which implies that self.R will be set to np.zeros((2, 2)) and self.C will be calculated later for the 1st time (in the 2nd iteration, self.C will already have a value when self.R is no longer None)

junajo10

I don´t know the reason but the values of et and sqrt_qt are very different. I am testing the strategy using intraday data for one day.

first file

datetime,Open,High,Low,Last,volume
2017-02-14 09:14:00,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:14:10,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:14:20,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:14:30,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:14:40,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:14:50,10280.0,10280.0,10280.0,10280.0,814.0
2017-02-14 09:15:00,10280.0,10280.0,10270.0,10275.0,154.0
2017-02-14 09:15:10,10271.0,10274.0,10270.0,10272.0,151.0
2017-02-14 09:15:20,10271.0,10275.0,10245.0,10271.0,233.0
2017-02-14 09:15:30,10273.0,10274.0,10250.0,10272.0,175.0
2017-02-14 09:15:40,10272.0,10272.0,10268.0,10269.0,89.0
2017-02-14 09:15:50,10269.0,10272.0,10268.0,10271.0,30.0
2017-02-14 09:16:00,10271.0,10274.0,10270.0,10273.0,58.0
2017-02-14 09:16:10,10273.0,10277.0,10273.0,10277.0,87.0
2017-02-14 09:16:20,10277.0,10277.0,10275.0,10275.0,53.0
2017-02-14 09:16:30,10275.0,10278.0,10275.0,10278.0,104.0
2017-02-14 09:16:40,10278.0,10278.0,10276.0,10276.0,39.0
2017-02-14 09:16:50,10276.0,10276.0,10274.0,10275.0,55.0
2017-02-14 09:17:00,10276.0,10276.0,10269.0,10270.0,68.0
2017-02-14 09:17:10,10270.0,10271.0,10270.0,10270.0,31.0
2017-02-14 09:17:20,10270.0,10271.0,10269.0,10271.0,26.0
2017-02-14 09:17:30,10272.0,10273.0,10271.0,10271.0,14.0
2017-02-14 09:17:40,10271.0,10271.0,10270.0,10270.0,5.0
2017-02-14 09:17:50,10271.0,10271.0,10271.0,10271.0,13.0
2017-02-14 09:18:00,10270.0,10270.0,10268.0,10269.0,29.0
2017-02-14 09:18:10,10269.0,10270.0,10268.0,10270.0,15.0
2017-02-14 09:18:20,10271.0,10271.0,10269.0,10269.0,22.0
2017-02-14 09:18:30,10268.0,10268.0,10263.0,10264.0,55.0
2017-02-14 09:18:40,10265.0,10265.0,10261.0,10265.0,77.0
2017-02-14 09:18:50,10265.0,10266.0,10263.0,10264.0,21.0
2017-02-14 09:19:00,10263.0,10264.0,10262.0,10263.0,75.0
2017-02-14 09:19:10,10265.0,10265.0,10263.0,10265.0,79.0
2017-02-14 09:19:20,10265.0,10265.0,10263.0,10264.0,27.0
2017-02-14 09:19:30,10263.0,10264.0,10262.0,10263.0,37.0
2017-02-14 09:19:40,10264.0,10264.0,10262.0,10264.0,22.0
2017-02-14 09:19:50,10263.0,10264.0,10262.0,10262.0,38.0
2017-02-14 09:20:00,10263.0,10264.0,10260.0,10260.0,60.0
2017-02-14 09:20:10,10260.0,10260.0,10258.0,10259.0,51.0
2017-02-14 09:20:20,10260.0,10260.0,10257.0,10260.0,85.0

second file

datetime,Open,High,Low,Last,volume
2017-02-14 09:14:00,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:14:10,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:14:20,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:14:30,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:14:40,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:14:50,23650.0,23650.0,23650.0,23650.0,612.0
2017-02-14 09:15:00,23651.0,23660.0,23615.0,23639.0,304.0
2017-02-14 09:15:10,23640.0,23641.0,23631.0,23634.0,163.0
2017-02-14 09:15:20,23636.0,23642.0,23630.0,23639.0,131.0
2017-02-14 09:15:30,23641.0,23642.0,23637.0,23642.0,56.0
2017-02-14 09:15:40,23640.0,23642.0,23635.0,23635.0,53.0
2017-02-14 09:15:50,23638.0,23638.0,23633.0,23636.0,50.0
2017-02-14 09:16:00,23637.0,23647.0,23636.0,23645.0,95.0
2017-02-14 09:16:10,23645.0,23655.0,23642.0,23648.0,162.0
2017-02-14 09:16:20,23648.0,23654.0,23645.0,23651.0,59.0
2017-02-14 09:16:30,23652.0,23654.0,23650.0,23650.0,35.0
2017-02-14 09:16:40,23650.0,23654.0,23650.0,23650.0,45.0
2017-02-14 09:16:50,23652.0,23652.0,23646.0,23648.0,50.0
2017-02-14 09:17:00,23648.0,23651.0,23640.0,23640.0,87.0
2017-02-14 09:17:10,23642.0,23645.0,23642.0,23643.0,21.0
2017-02-14 09:17:20,23643.0,23646.0,23641.0,23646.0,33.0
2017-02-14 09:17:30,23644.0,23646.0,23642.0,23642.0,34.0
2017-02-14 09:17:40,23643.0,23645.0,23643.0,23645.0,18.0
2017-02-14 09:17:50,23646.0,23650.0,23646.0,23650.0,27.0
2017-02-14 09:18:00,23649.0,23649.0,23642.0,23644.0,52.0
2017-02-14 09:18:10,23645.0,23648.0,23645.0,23648.0,19.0
2017-02-14 09:18:20,23647.0,23648.0,23646.0,23648.0,24.0
2017-02-14 09:18:30,23646.0,23646.0,23638.0,23640.0,49.0
2017-02-14 09:18:40,23640.0,23642.0,23637.0,23640.0,49.0
2017-02-14 09:18:50,23640.0,23641.0,23639.0,23639.0,26.0
2017-02-14 09:19:00,23639.0,23642.0,23638.0,23642.0,22.0
2017-02-14 09:19:10,23642.0,23647.0,23641.0,23644.0,37.0
2017-02-14 09:19:20,23643.0,23644.0,23641.0,23644.0,21.0
2017-02-14 09:19:30,23644.0,23644.0,23640.0,23641.0,14.0
2017-02-14 09:19:40,23641.0,23641.0,23638.0,23640.0,34.0
2017-02-14 09:19:50,23641.0,23641.0,23639.0,23640.0,18.0
2017-02-14 09:20:00,23638.0,23641.0,23635.0,23635.0,68.0
2017-02-14 09:20:10,23636.0,23636.0,23632.0,23633.0,66.0
2017-02-14 09:20:20,23633.0,23636.0,23627.0,23636.0,158.0

My values: As you can see my errors are between 0 and 1 and my sqrt_qt is near to 74, i think these values can not be good

date;price1;price2;et;sqrt_qt;beta
2017-02-14T09:14:10;23650.0000;10280.0000;10280.0000;23651.0000;0.0000
2017-02-14T09:14:20;23650.0000;10280.0000;0.0000;74.7883;0.4347
2017-02-14T09:14:30;23650.0000;10280.0000;0.0000;74.7883;0.4347
2017-02-14T09:14:40;23650.0000;10280.0000;0.0000;74.7883;0.4347
2017-02-14T09:14:50;23650.0000;10280.0000;0.0000;74.7883;0.4347
2017-02-14T09:15:00;23639.0000;10275.0000;-0.2188;74.7535;0.4347
2017-02-14T09:15:10;23634.0000;10272.0000;-0.8268;74.7377;0.4346
2017-02-14T09:15:20;23639.0000;10271.0000;-3.1730;74.7535;0.4346
2017-02-14T09:15:30;23642.0000;10272.0000;-0.3034;74.7630;0.4345
2017-02-14T09:15:40;23635.0000;10269.0000;0.0412;74.7409;0.4345
2017-02-14T09:15:50;23636.0000;10271.0000;1.5655;74.7440;0.4345
2017-02-14T09:16:00;23645.0000;10273.0000;-1.9108;74.7725;0.4345
2017-02-14T09:16:10;23648.0000;10277.0000;2.6966;74.7820;0.4344
2017-02-14T09:16:20;23651.0000;10275.0000;-3.3037;74.7914;0.4346
2017-02-14T09:16:30;23650.0000;10278.0000;3.4344;74.7883;0.4344
2017-02-14T09:16:40;23650.0000;10276.0000;-2.0000;74.7883;0.4346
2017-02-14T09:16:50;23648.0000;10275.0000;-0.1310;74.7820;0.4345
2017-02-14T09:17:00;23640.0000;10270.0000;-1.5242;74.7567;0.4345
2017-02-14T09:17:10;23643.0000;10270.0000;-1.3032;74.7662;0.4344
2017-02-14T09:17:20;23646.0000;10271.0000;-0.3031;74.7756;0.4344
2017-02-14T09:17:30;23642.0000;10271.0000;1.7374;74.7630;0.4343
2017-02-14T09:17:40;23645.0000;10270.0000;-2.3033;74.7725;0.4344
2017-02-14T09:17:50;23650.0000;10271.0000;-1.1716;74.7883;0.4343
2017-02-14T09:18:00;23644.0000;10269.0000;0.6056;74.7693;0.4343
2017-02-14T09:18:10;23648.0000;10270.0000;-0.7372;74.7820;0.4343
2017-02-14T09:18:20;23648.0000;10269.0000;-1.0000;74.7820;0.4343
2017-02-14T09:18:30;23640.0000;10264.0000;-1.5262;74.7567;0.4342
2017-02-14T09:18:40;23640.0000;10265.0000;1.0000;74.7567;0.4342
2017-02-14T09:18:50;23639.0000;10264.0000;-0.5658;74.7535;0.4342
2017-02-14T09:19:00;23642.0000;10263.0000;-2.3025;74.7630;0.4342
2017-02-14T09:19:10;23644.0000;10265.0000;1.1318;74.7693;0.4341
2017-02-14T09:19:20;23644.0000;10264.0000;-1.0000;74.7693;0.4341
2017-02-14T09:19:30;23641.0000;10263.0000;0.3023;74.7598;0.4341
2017-02-14T09:19:40;23640.0000;10264.0000;1.4341;74.7567;0.4341
2017-02-14T09:19:50;23640.0000;10262.0000;-2.0000;74.7567;0.4342
2017-02-14T09:20:00;23635.0000;10260.0000;0.1704;74.7409;0.4341
2017-02-14T09:20:10;23633.0000;10259.0000;-0.1318;74.7345;0.4341
2017-02-14T09:20:20;23636.0000;10260.0000;-0.3022;74.7440;0.4341

junajo10

I think the problem is here:

    # Q_t is the variance of the prediction of observations and hence
    # \sqrt{Q_t} is the standard deviation of the predictions
    Qt = F.dot(self.R).dot(F.T) + self.p.vt

The post said:"variance of the last observations", i think we have to take n qt , calculate variance and then the sqrt . What do you think?

backtrader

Will recheck the post and the ported code on that area. Thanks

junajo10

I think the post is wrong

backtrader

The original post in Quantopian by Aidan O'Mahony shows the same calculation.

Reference: Quantopian - Ernie Chan's EWA/EWC pair trade with Kalman filter

...
        Q = x.dot(context.R).dot(x.T) + context.Ve
        sqrt_Q = np.sqrt(Q)
...

junajo10

https://github.com/lukstei/trading-backtest/blob/master/src/main/java/org/lst/trading/main/strategy/kalman/CointegrationTradingStrategy.java

junajo10

I think i have got good values, I have changed the init values:
('delta',1e-7 ),
('vt',1e-3),

RandyT

@junajo10 thought I would join the conversation here as well.

Noticed that the Java system you reference below, he is using 1e-10 for delta and 1e-7 for the vt parameter.

Any other thoughts there?

RandyT

What is needed to change the scale of the KalmanFilterInd? I cannot seem to find the plotinfo magic to scale this up a bit.

backtrader

@RandyT There is no magic. The problem is that the initial values of the filter are orders of magnitude greater than the ones when the filter has settled.

RandyT

I see your use of get_ylim() and set_ylim() but could not find a parameter that would take advantage of this. Seems you use it for volume for this exact purpose.