Kalman et al.


  • administrators

    Use the link below to go the original post

    Click here to see the full blog post



  • self.P = np.zeros((2, 2)) should be self.C?


  • administrators

    May well be. Will be checked.



  • Also, kf = KalmanFilterInd(self.data0, self.data1)


  • administrators

    That was a just a test to see how the prediction to itself would look like.

    If you have a look at KalmanSignals you will see:

    KalmanFilterInd()
    

    with even no mention of the data feeds. This passes whatever data feeds are in the parent down to KalmanFilterInd and this is equal to KalmanFilterInd(self.data0, self.data1)

    But thank you for spotting it, it is for sure worth some rewriting to make all options more comprehensible.



  • WOW I was just absent 2 weeks and that BOOM. Thanks DRo and Junajo10. U rock guys !!


  • administrators

    @junajo10 said in Kalman et al.:

    self.P = np.zeros((2, 2)) should be self.C?

    The original code has indeed self.P = np.zeros((2, 2)). Even if actually meant self.C = np.zeros((2, 2)) it doesn't play any role.

    The 1st value of self.R is None, which implies that self.R will be set to np.zeros((2, 2)) and self.C will be calculated later for the 1st time (in the 2nd iteration, self.C will already have a value when self.R is no longer None)



  • I don´t know the reason but the values of et and sqrt_qt are very different. I am testing the strategy using intraday data for one day.

    first file

    datetime,Open,High,Low,Last,volume
    2017-02-14 09:14:00,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:14:10,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:14:20,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:14:30,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:14:40,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:14:50,10280.0,10280.0,10280.0,10280.0,814.0
    2017-02-14 09:15:00,10280.0,10280.0,10270.0,10275.0,154.0
    2017-02-14 09:15:10,10271.0,10274.0,10270.0,10272.0,151.0
    2017-02-14 09:15:20,10271.0,10275.0,10245.0,10271.0,233.0
    2017-02-14 09:15:30,10273.0,10274.0,10250.0,10272.0,175.0
    2017-02-14 09:15:40,10272.0,10272.0,10268.0,10269.0,89.0
    2017-02-14 09:15:50,10269.0,10272.0,10268.0,10271.0,30.0
    2017-02-14 09:16:00,10271.0,10274.0,10270.0,10273.0,58.0
    2017-02-14 09:16:10,10273.0,10277.0,10273.0,10277.0,87.0
    2017-02-14 09:16:20,10277.0,10277.0,10275.0,10275.0,53.0
    2017-02-14 09:16:30,10275.0,10278.0,10275.0,10278.0,104.0
    2017-02-14 09:16:40,10278.0,10278.0,10276.0,10276.0,39.0
    2017-02-14 09:16:50,10276.0,10276.0,10274.0,10275.0,55.0
    2017-02-14 09:17:00,10276.0,10276.0,10269.0,10270.0,68.0
    2017-02-14 09:17:10,10270.0,10271.0,10270.0,10270.0,31.0
    2017-02-14 09:17:20,10270.0,10271.0,10269.0,10271.0,26.0
    2017-02-14 09:17:30,10272.0,10273.0,10271.0,10271.0,14.0
    2017-02-14 09:17:40,10271.0,10271.0,10270.0,10270.0,5.0
    2017-02-14 09:17:50,10271.0,10271.0,10271.0,10271.0,13.0
    2017-02-14 09:18:00,10270.0,10270.0,10268.0,10269.0,29.0
    2017-02-14 09:18:10,10269.0,10270.0,10268.0,10270.0,15.0
    2017-02-14 09:18:20,10271.0,10271.0,10269.0,10269.0,22.0
    2017-02-14 09:18:30,10268.0,10268.0,10263.0,10264.0,55.0
    2017-02-14 09:18:40,10265.0,10265.0,10261.0,10265.0,77.0
    2017-02-14 09:18:50,10265.0,10266.0,10263.0,10264.0,21.0
    2017-02-14 09:19:00,10263.0,10264.0,10262.0,10263.0,75.0
    2017-02-14 09:19:10,10265.0,10265.0,10263.0,10265.0,79.0
    2017-02-14 09:19:20,10265.0,10265.0,10263.0,10264.0,27.0
    2017-02-14 09:19:30,10263.0,10264.0,10262.0,10263.0,37.0
    2017-02-14 09:19:40,10264.0,10264.0,10262.0,10264.0,22.0
    2017-02-14 09:19:50,10263.0,10264.0,10262.0,10262.0,38.0
    2017-02-14 09:20:00,10263.0,10264.0,10260.0,10260.0,60.0
    2017-02-14 09:20:10,10260.0,10260.0,10258.0,10259.0,51.0
    2017-02-14 09:20:20,10260.0,10260.0,10257.0,10260.0,85.0
    

    second file

    datetime,Open,High,Low,Last,volume
    2017-02-14 09:14:00,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:14:10,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:14:20,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:14:30,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:14:40,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:14:50,23650.0,23650.0,23650.0,23650.0,612.0
    2017-02-14 09:15:00,23651.0,23660.0,23615.0,23639.0,304.0
    2017-02-14 09:15:10,23640.0,23641.0,23631.0,23634.0,163.0
    2017-02-14 09:15:20,23636.0,23642.0,23630.0,23639.0,131.0
    2017-02-14 09:15:30,23641.0,23642.0,23637.0,23642.0,56.0
    2017-02-14 09:15:40,23640.0,23642.0,23635.0,23635.0,53.0
    2017-02-14 09:15:50,23638.0,23638.0,23633.0,23636.0,50.0
    2017-02-14 09:16:00,23637.0,23647.0,23636.0,23645.0,95.0
    2017-02-14 09:16:10,23645.0,23655.0,23642.0,23648.0,162.0
    2017-02-14 09:16:20,23648.0,23654.0,23645.0,23651.0,59.0
    2017-02-14 09:16:30,23652.0,23654.0,23650.0,23650.0,35.0
    2017-02-14 09:16:40,23650.0,23654.0,23650.0,23650.0,45.0
    2017-02-14 09:16:50,23652.0,23652.0,23646.0,23648.0,50.0
    2017-02-14 09:17:00,23648.0,23651.0,23640.0,23640.0,87.0
    2017-02-14 09:17:10,23642.0,23645.0,23642.0,23643.0,21.0
    2017-02-14 09:17:20,23643.0,23646.0,23641.0,23646.0,33.0
    2017-02-14 09:17:30,23644.0,23646.0,23642.0,23642.0,34.0
    2017-02-14 09:17:40,23643.0,23645.0,23643.0,23645.0,18.0
    2017-02-14 09:17:50,23646.0,23650.0,23646.0,23650.0,27.0
    2017-02-14 09:18:00,23649.0,23649.0,23642.0,23644.0,52.0
    2017-02-14 09:18:10,23645.0,23648.0,23645.0,23648.0,19.0
    2017-02-14 09:18:20,23647.0,23648.0,23646.0,23648.0,24.0
    2017-02-14 09:18:30,23646.0,23646.0,23638.0,23640.0,49.0
    2017-02-14 09:18:40,23640.0,23642.0,23637.0,23640.0,49.0
    2017-02-14 09:18:50,23640.0,23641.0,23639.0,23639.0,26.0
    2017-02-14 09:19:00,23639.0,23642.0,23638.0,23642.0,22.0
    2017-02-14 09:19:10,23642.0,23647.0,23641.0,23644.0,37.0
    2017-02-14 09:19:20,23643.0,23644.0,23641.0,23644.0,21.0
    2017-02-14 09:19:30,23644.0,23644.0,23640.0,23641.0,14.0
    2017-02-14 09:19:40,23641.0,23641.0,23638.0,23640.0,34.0
    2017-02-14 09:19:50,23641.0,23641.0,23639.0,23640.0,18.0
    2017-02-14 09:20:00,23638.0,23641.0,23635.0,23635.0,68.0
    2017-02-14 09:20:10,23636.0,23636.0,23632.0,23633.0,66.0
    2017-02-14 09:20:20,23633.0,23636.0,23627.0,23636.0,158.0
    

    My values: As you can see my errors are between 0 and 1 and my sqrt_qt is near to 74, i think these values can not be good

    date;price1;price2;et;sqrt_qt;beta
    2017-02-14T09:14:10;23650.0000;10280.0000;10280.0000;23651.0000;0.0000
    2017-02-14T09:14:20;23650.0000;10280.0000;0.0000;74.7883;0.4347
    2017-02-14T09:14:30;23650.0000;10280.0000;0.0000;74.7883;0.4347
    2017-02-14T09:14:40;23650.0000;10280.0000;0.0000;74.7883;0.4347
    2017-02-14T09:14:50;23650.0000;10280.0000;0.0000;74.7883;0.4347
    2017-02-14T09:15:00;23639.0000;10275.0000;-0.2188;74.7535;0.4347
    2017-02-14T09:15:10;23634.0000;10272.0000;-0.8268;74.7377;0.4346
    2017-02-14T09:15:20;23639.0000;10271.0000;-3.1730;74.7535;0.4346
    2017-02-14T09:15:30;23642.0000;10272.0000;-0.3034;74.7630;0.4345
    2017-02-14T09:15:40;23635.0000;10269.0000;0.0412;74.7409;0.4345
    2017-02-14T09:15:50;23636.0000;10271.0000;1.5655;74.7440;0.4345
    2017-02-14T09:16:00;23645.0000;10273.0000;-1.9108;74.7725;0.4345
    2017-02-14T09:16:10;23648.0000;10277.0000;2.6966;74.7820;0.4344
    2017-02-14T09:16:20;23651.0000;10275.0000;-3.3037;74.7914;0.4346
    2017-02-14T09:16:30;23650.0000;10278.0000;3.4344;74.7883;0.4344
    2017-02-14T09:16:40;23650.0000;10276.0000;-2.0000;74.7883;0.4346
    2017-02-14T09:16:50;23648.0000;10275.0000;-0.1310;74.7820;0.4345
    2017-02-14T09:17:00;23640.0000;10270.0000;-1.5242;74.7567;0.4345
    2017-02-14T09:17:10;23643.0000;10270.0000;-1.3032;74.7662;0.4344
    2017-02-14T09:17:20;23646.0000;10271.0000;-0.3031;74.7756;0.4344
    2017-02-14T09:17:30;23642.0000;10271.0000;1.7374;74.7630;0.4343
    2017-02-14T09:17:40;23645.0000;10270.0000;-2.3033;74.7725;0.4344
    2017-02-14T09:17:50;23650.0000;10271.0000;-1.1716;74.7883;0.4343
    2017-02-14T09:18:00;23644.0000;10269.0000;0.6056;74.7693;0.4343
    2017-02-14T09:18:10;23648.0000;10270.0000;-0.7372;74.7820;0.4343
    2017-02-14T09:18:20;23648.0000;10269.0000;-1.0000;74.7820;0.4343
    2017-02-14T09:18:30;23640.0000;10264.0000;-1.5262;74.7567;0.4342
    2017-02-14T09:18:40;23640.0000;10265.0000;1.0000;74.7567;0.4342
    2017-02-14T09:18:50;23639.0000;10264.0000;-0.5658;74.7535;0.4342
    2017-02-14T09:19:00;23642.0000;10263.0000;-2.3025;74.7630;0.4342
    2017-02-14T09:19:10;23644.0000;10265.0000;1.1318;74.7693;0.4341
    2017-02-14T09:19:20;23644.0000;10264.0000;-1.0000;74.7693;0.4341
    2017-02-14T09:19:30;23641.0000;10263.0000;0.3023;74.7598;0.4341
    2017-02-14T09:19:40;23640.0000;10264.0000;1.4341;74.7567;0.4341
    2017-02-14T09:19:50;23640.0000;10262.0000;-2.0000;74.7567;0.4342
    2017-02-14T09:20:00;23635.0000;10260.0000;0.1704;74.7409;0.4341
    2017-02-14T09:20:10;23633.0000;10259.0000;-0.1318;74.7345;0.4341
    2017-02-14T09:20:20;23636.0000;10260.0000;-0.3022;74.7440;0.4341
    


  • I think the problem is here:

        # Q_t is the variance of the prediction of observations and hence
        # \sqrt{Q_t} is the standard deviation of the predictions
        Qt = F.dot(self.R).dot(F.T) + self.p.vt
    

    The post said:"variance of the last observations", i think we have to take n qt , calculate variance and then the sqrt . What do you think?


  • administrators

    Will recheck the post and the ported code on that area. Thanks



  • I think the post is wrong


  • administrators

    The original post in Quantopian by Aidan O'Mahony shows the same calculation.

    Reference: Quantopian - Ernie Chan's EWA/EWC pair trade with Kalman filter

    ...
            Q = x.dot(context.R).dot(x.T) + context.Ve
            sqrt_Q = np.sqrt(Q)
    ...
    




  • I think i have got good values, I have changed the init values:
    ('delta',1e-7 ),
    ('vt',1e-3),



  • @junajo10 thought I would join the conversation here as well.

    Noticed that the Java system you reference below, he is using 1e-10 for delta and 1e-7 for the vt parameter.

    Any other thoughts there?



  • What is needed to change the scale of the KalmanFilterInd? I cannot seem to find the plotinfo magic to scale this up a bit.


  • administrators

    @RandyT There is no magic. The problem is that the initial values of the filter are orders of magnitude greater than the ones when the filter has settled.



  • I see your use of get_ylim() and set_ylim() but could not find a parameter that would take advantage of this. Seems you use it for volume for this exact purpose.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.