Different results with resample data and add data methods!

aqa20

I get very different results between the two approaches shouldn't both of them deliver the same result!?

Approach one:

import backtrader as bt
from datetime import datetime


class CustomDataset(bt.feeds.GenericCSVData):
    ''' Instinitate CustomDataset '''
    params = (
        ('time', -1),
        ('datetime', 0),
        ('open', 1),
        ('high', 2),
        ('low', 3),
        ('close', 4),
        ('volume', 5),
        ('openinterest', 6),
    )


cerebro = bt.Cerebro()

data = CustomDataset(
    name='data',
    dataname='./5m-timeframe.csv',
    timeframe=bt.TimeFrame.Minutes,
    fromdate=datetime(2021, 3, 1),
    todate=datetime(2021, 3, 10)
)


data1 = CustomDataset(
    name='data1',
    dataname='./1h-timeframe.csv',
    timeframe=bt.TimeFrame.Minutes,
    fromdate=datetime(2021, 3, 1),
    todate=datetime(2021, 3, 10)
)

cerebro.adddata(data)
cerebro.adddata(data1)

Approach 2

import backtrader as bt
from datetime import datetime


class CustomDataset(bt.feeds.GenericCSVData):
    ''' Instinitate CustomDataset '''
    params = (
        ('time', -1),
        ('datetime', 0),
        ('open', 1),
        ('high', 2),
        ('low', 3),
        ('close', 4),
        ('volume', 5),
        ('openinterest', 6),
    )


cerebro = bt.Cerebro()

data = CustomDataset(
    name='data',
    dataname='./1m-timeframe.csv',
    timeframe=bt.TimeFrame.Minutes,
    fromdate=datetime(2021, 3, 1),
    todate=datetime(2021, 3, 10)
)

cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=5, name='5m')
cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=60, name='60m')

dasch

@aqa20 in your approach one you did not define the data compression, so all your csv are defined as 1 minute data, which is wrong. Try approach one with adding the param compression.

aqa20

@dasch said in Different results with resample data and add data methods!:

our approach one you did not define the data compression, so all your csv are defined as 1 minute data, which is wrong. Try approach one with adding the param compression.

I'm loading data from 2 different files, the first one is 5m data, and the second is 1h data, and I haven't added the compression parameter in the first approach because the default value for compression is one

dasch

@aqa20 so for 5 minutes you would use these params:

compression=5, timeframe=bt.TimeFrame.Minutes

for 1 hour data you would use these params:

compression=60, timeframe=bt.TimeFrame.Minutes

aqa20

@dasch Well that's what I've done already, check the code

dasch

but you need to define, what kind of data you are using. so in approach one you just don't define the data granularity which is required. the result should be the same after defining the compression in approach one.

dasch

@aqa20 said in Different results with resample data and add data methods!:

@dasch Well that's what I've done already, check the code

no, you did not define the data correctly in approach one. you left out the compression, which is needed there.

dasch

@aqa20 approach two works because there you defined to what compreession you want to resample your data to. backtrader will check if the timestamp of current candle is over the period and forward the data when its over (which is basically every row, since you already use the matching timeframe/compression in your source data).
If you just define a data source and add it to bbacktrader, then backtrader does not know, what granularity your data has.
this is why you need to define in approach one also the compression of your data.

EMR

@aqa20

What are the differences that you find between these 2 approaches ?
In approach 1 you load datas which are already in 5 minutes & 60 minutes compression. Do you know if this compression is the same as when done by backtrader (ie bar2edge, boundoff, rightedge etc.)
Do you have a different result in approach 1 if you add, along with the timeframe parameter, the parameters compression = 5 & compression = 60 to data & data1 ?
Have you checked for missing minutes in the 1 minute data file ?

aqa20

@dasch I've tried to add compression to the first approach, but I still get different result from 2 approaches

aqa20

@emr
The differences is in the chart and the time execution orders,
In approach one if the buy order was executed on [05-03-21 12:00], the buy order will be executed on [05-03-21 13:00] on the the second approach, and same for sell orders.

Do you know if this compression is the same as when done by backtrader (ie bar2edge, boundoff, rightedge etc?

I believe so here is a sample of the 1m data

2021-03-24 00:00:00,54342.8,54359.0,54230.0,54260.35,1616544059999,2153
2021-03-24 00:01:00,54260.36,54422.43,54248.53,54408.13,1616544119999,1364
2021-03-24 00:02:00,54402.67,54515.73,54394.46,54515.73,1616544179999,1194
2021-03-24 00:03:00,54511.02,54606.15,54511.02,54592.13,1616544239999,1324
2021-03-24 00:04:00,54592.12,54669.35,54563.21,54597.62,1616544299999,1381
2021-03-24 00:05:00,54595.26,54626.39,54518.44,54555.23,1616544359999,909

Sample from the 1h data:

2021-03-24 00:00:00,255.5191,256.9375,250.8104,251.6466,1616547599999,51155
2021-03-24 01:00:00,251.5354,256.0,251.1514,255.6516,1616551199999,20791
2021-03-24 02:00:00,255.6426,256.9996,254.8947,255.6458,1616554799999,19972
2021-03-24 03:00:00,255.6458,256.6457,254.2957,256.1307,1616558399999,21009
2021-03-24 04:00:00,256.1242,256.98,253.9556,255.0727,1616561999999,22185
2021-03-24 05:00:00,255.0662,255.7192,252.8269,255.5432,1616565599999,18944

Sample from 5m data:

2021-03-24 00:00:00,255.5191,256.9375,254.6,256.813,1616544299999,3381
2021-03-24 00:05:00,256.8,256.9372,254.11,254.45,1616544599999,4276
2021-03-24 00:10:00,254.3532,255.0489,253.05,253.6577,1616544899999,4797
2021-03-24 00:15:00,253.8025,253.809,252.102,252.1554,1616545199999,4867
2021-03-24 00:20:00,252.305,253.4392,251.7721,252.3381,1616545499999,6368
2021-03-24 00:25:00,252.3289,252.63,250.8104,252.0226,1616545799999,6691
2021-03-24 00:30:00,252.0087,253.0,251.5001,252.4069,1616546099999,4125

Do you have a different result in approach 1 if you add, along with the timeframe parameter, the parameters compression = 5 & compression = 60 to data & data1 ?

Yes!

Have you checked for missing minutes in the 1 minute data file ?
Yes i checked for that using pandas

aqa20

@aqa20 I'm sorry actually It don't get different result when adding, 5 and 60 compression permeates.

so the answer for this question:
" [Do you have a different result in approach 1 if you add, along with the timeframe parameter, the parameters compression = 5 & compression = 60 to data & data1 ?](link url)"
Is no

run-out

@aqa20
The only way you will resolve this with certainty is to turn off/comment out/don't print the trading algo part, and just print the data and carefully compare the two versions.

You will immediately see where the discrepencies are and can start to debug from there.

I believe the previous posters have all given you good clues. Do some investigation on the data itself and try to find where the problem lies.

It might be possible your data/datasource methodology is not lined up with backtrader, since there are variables that can change when resampling.

aqa20

@run-out I have done that and in the first approach the data was correct,
in next method I print the datetime along with other values OHLVC, and the values were matching perfectly with the data
source in the first approach, however in the second approach it was completely different values! but I still don't know where is the problem in approach 2!

dasch

@aqa20 take a look here: https://community.backtrader.com/topic/1629/problem-with-resampling-1m-data-to-5m-15m/

aqa20

@dasch Thanks man, but for seek of simplicity I will stick with the first approach for now since it's working and I already have the data of time frames that I need, but now I'm trying to use the first approach in the live trading, so let me know if you can help with that but without using the resample data, I want to achieve that with first approach

EMR

@aqa20 said in Different results with resample data and add data methods!:

@emr

Do you know if this compression is the same as when done by backtrader (ie bar2edge, boundoff, rightedge etc?

I believe so

I noticed that you can have very different presentation of 5mn, 1h etc data depending of the data provider. For me, it was differences between data from MT5 & data resampled by BT. For example, 1h data from MT5 had a different close value. You may want to check the M5 & H1 open & close values to find if it is the one of neighboring minutes and also check the doc to find if some additional parameters (boundoff, rightedge etc.) can bring you the results you want

Also, I noticed during my tests that if some minutes are missing, the resampling to for ex. H8 may lead to errors (closure of the bar occur later than expected - I still have to find the time to log an issue in BT2 for that)