Problem with resampling 1m data to 5m/15m

ppawel

I have 1m OHLC data and would like to compress that into 5m and 15m data. I know Backtrader has resampling and I tried different options but I still can't get it to align correctly with my data and charts at TradingView.

What I want to achieve:

1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:00

What Backtrader does by default:

1m bars 17:01-17:05 (inclusive) -> 5m bar dated 17:05

Obviously this gives me OHLC data that is not matching what I want so I used the boundoff=1 option. This results in correct OHLC for the 5m bars but they are dated incorrectly for my needs, like this:

1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:04

I tried other resampling options but perhaps I don't fully understand them and I could not get the result I want.

Could you give me some hints if it can be achieved before I dive into the Backtrader code?

backtrader

You want rightedge=False. Docs - Data Resampling

But what you probably really want is to resample your data outise of backtrader and thenn lod it, to avoid a problem with the convention.

ppawel

@backtrader I tried that already and it does not seem to have any effect - I get the same results with either rightedge=false or true (default I guess).

Might that be a conflict of this option with boundoff=1?

In any case, I would prefer to avoid processing the data outside of Backtrader because I already am storing 1m bar data so no point storing 5m as well. So I'm thinking about writing a data source for Backtrader which converts this automatically or adding an option to the resampling method that outputs correct bars for my use case. If I would make it work with resampling, does it make sense to put it in a PR?

backtrader

There were several patches to resampling, which probably had an impact on rightedge.

Use this code as a template to achieve your desired effect.

    rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=5)

    def _new_load(self):
        ret = self._old_load()
        if ret:
            dt = self.datetime.datetime(0)
            self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=5))

        return ret

    rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)

lexar

Has there been any update to this code? I tried the suggested code and it still doesn't work.

Basically I am trying to have resampling of 1m to 1h.
So as an example the close of 1m at 10:59 should be the 1h close of 10:00.

However I see that the 1m close at 10:59 becomes the 1h close of 11:00.

How can I fix this?

bigdavediode

Lexar, having a minute bar from 10:59 go back in time to generate a result at 10:00 is a good way to make a system that won't work.

balibou

Thanks for this solution @backtrader it works like a charm !
I was wondering how you can pass an argument to the new_load function in order to deal with a variable minute (for dealing with multiple timeframes in a strategy).

I looked on stack overflow and I tried this (but got this error AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute '_old_load'):

def _new_load(self, minutes):
    ret = self._old_load()
    if ret:
        datetime = self.datetime.datetime(0)
        self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1))

    return ret

minutes = 240
rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes)
rd._old_lo
ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)

Thanks :)

Nam Nguyen

Hello,

Did anyone figure out a stable way to resample it?

I'm not sure this code will be work well with live data feed.

def _new_load(self, minutes):
    ret = self._old_load()
    if ret:
        datetime = self.datetime.datetime(0)
        self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1))

    return ret

minutes = 240
rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes)
rd._old_lo
ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)

I try to understand Resampler class but didn't make it works.

        rightedge=False,
        bar2edge=True,
        adjbartime=True,
        boundoff=1,

with this setting only last bar get right result, earlier data have good prices and volume but time is wrong.

run-out

Could you include sample outputs expected and what you are actually getting?

Nam Nguyen

Hello @run-out , thank you for your question, I reproduce buy test like this:

I want to resample M1 to M5 data export from Metatrader 5. So I export M1 data from Metatrader 5, and my strategy has using M1 and M5 datafeeds, then I resample data from M1 to M5 instead of get another M5 data (and I thing in live datafeed, Tick data will resample to M1 data the same way resample M1 to M5, right?). So, then I want to verify that my resample setting work well.

M1 datafeeds is data from 2020.09.01 01:00:00 to 2020.09.01 01:10:00 in 10 minutes to resample

<DATE>	<TIME>	<OPEN>	<HIGH>	<LOW>	<CLOSE>	<TICKVOL>	<VOL>	<SPREAD>
2020.09.01	01:00:00	1.33655	1.33691	1.33655	1.33667	76	0	28
2020.09.01	01:01:00	1.33667	1.33676	1.33665	1.33676	21	0	26
2020.09.01	01:02:00	1.33676	1.33680	1.33672	1.33673	12	0	18
2020.09.01	01:03:00	1.33673	1.33674	1.33670	1.33671	30	0	16
2020.09.01	01:04:00	1.33672	1.33681	1.33670	1.33679	67	0	16
2020.09.01	01:05:00	1.33679	1.33681	1.33675	1.33681	87	0	20
2020.09.01	01:06:00	1.33679	1.33686	1.33676	1.33678	78	0	19
2020.09.01	01:07:00	1.33679	1.33686	1.33677	1.33685	87	0	18
2020.09.01	01:08:00	1.33684	1.33687	1.33676	1.33677	54	0	13
2020.09.01	01:09:00	1.33677	1.33678	1.33670	1.33678	22	0	10
2020.09.01	01:10:00	1.33678	1.33678	1.33676	1.33677	12	0	12

M5 datafeeds is data from 2020.09.01 01:00:00 to 2020.09.01 01:10:00 in 10 minutes to verify.

<DATE>	<TIME>	<OPEN>	<HIGH>	<LOW>	<CLOSE>	<TICKVOL>	<VOL>	<SPREAD>
2020.09.01	01:00:00	1.33655	1.33691	1.33655	1.33679	206	0	16
2020.09.01	01:05:00	1.33679	1.33687	1.33670	1.33678	328	0	10
2020.09.01	01:10:00	1.33678	1.33685	1.33664	1.33667	84	0	6

My testing source code:

import backtrader as bt


class MT5CSVData(bt.feeds.GenericCSVData):
    params = dict(
        dtformat='%Y.%m.%d',
        tmformat='%H:%M:%S',
        datetime=0,
        time=1,
        open=2,
        high=3,
        low=4,
        close=5,
        volume=6,
        openinterest=-1,

        separator="\t",
        timeframe=bt.TimeFrame.Minutes,
        compression=1,
    )


class DebugStrategy(bt.Strategy):
    def next(self):
        print(dict(
            datetime=self.data.datetime.datetime().strftime("%Y-%m-%d %H:%M:%S"),
            open=self.data.open[0],
            high=self.data.high[0],
            low=self.data.low[0],
            close=self.data.close[0],
            volume=self.data.volume[0])
        )


if __name__ == "__main__":
    data = MT5CSVData(
        dataname="./GBPUSD_M1_202009010100_202009010110.csv",
        timeframe=bt.TimeFrame.Minutes,
        compression=1,
    )

    cerebro = bt.Cerebro(stdstats=False)

    kwargs = dict(
        timeframe=bt.TimeFrame.Minutes,
        compression=5,
        rightedge=True,
        bar2edge=True,
        adjbartime=True,
        # boundoff=1,
    )
    cerebro.resampledata(data, **kwargs)

    cerebro.addstrategy(DebugStrategy)

    cerebro.run(
        exactbars=0
    )

Setting 1: Default setting:

        rightedge=True,
        bar2edge=True,
        adjbartime=True,
        boundoff=0,

I get response:

{'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33667, 'volume': 76.0}
{'datetime': '2020-09-01 01:05:00', 'open': 1.33667, 'high': 1.33681, 'low': 1.33665, 'close': 1.33681, 'volume': 217.0}
{'datetime': '2020-09-01 01:10:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33677, 'volume': 253.0}

Set rightedge=False get the same response.

Setting 2: boundoff=1

        rightedge=False,
        bar2edge=True,
        adjbartime=True,
        boundoff=1,

I get good response with M5 verify data at open, high, low, close and volume data but wrong at datetime

{'datetime': '2020-09-01 01:04:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0}
{'datetime': '2020-09-01 01:09:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0}
{'datetime': '2020-09-01 01:10:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}

With update code:

    rd = cerebro.resampledata(data, **kwargs)

    def _new_load(self):
        ret = self._old_load()
        if ret:
            dt = self.datetime.datetime(0)
            self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=4))

        return ret

    rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)

I get perfect response

{'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0}
{'datetime': '2020-09-01 01:05:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0}
{'datetime': '2020-09-01 01:05:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}

But I curious do we have any stable way to resolve this problem? Do Tick data with resample same with this way?
I didn't test on live date to verify it.

run-out

@Nam-Nguyen said in Problem with resampling 1m data to 5m/15m:

Metatrader 5

Anyone have experience with MetaTrader datafeeds?

dasch

i fixed issues with my datafeeds by adjusting the time to the end of the period they are on. Backtrader expects somehow the endtime of a period, while most data will have the starttime of a period as the datetime.

A custom csv feed with time adjustment which can be used to fix that:

import backtrader as bt
from backtrader.utils import date2num
from datetime import datetime, time, timedelta



def getstarttime(timeframe, compression, dt, sessionstart=None, offset=0):
    '''
    This method will return the start of the period based on current
    time (or provided time).
    '''
    if sessionstart is None:
        # use UTC 22:00 (5:00 pm New York) as default
        sessionstart = time(hour=22, minute=0, second=0)
    if dt is None:
        dt = datetime.utcnow()
    if timeframe == bt.TimeFrame.Seconds:
        dt = dt.replace(
            second=(dt.second // compression) * compression,
            microsecond=0)
        if offset:
            dt = dt - timedelta(seconds=compression*offset)
    elif timeframe == bt.TimeFrame.Minutes:
        if compression >= 60:
            hours = 0
            minutes = 0
            # get start of day
            dtstart = getstarttime(bt.TimeFrame.Days, 1, dt, sessionstart)
            # diff start of day with current time to get seconds
            # since start of day
            dtdiff = dt - dtstart
            hours = dtdiff.seconds//((60*60)*(compression//60))
            minutes = compression % 60
            dt = dtstart + timedelta(hours=hours, minutes=minutes)
        else:
            dt = dt.replace(
                minute=(dt.minute // compression) * compression,
                second=0,
                microsecond=0)
        if offset:
            dt = dt - timedelta(minutes=compression*offset)
    elif timeframe == bt.TimeFrame.Days:
        if dt.hour < sessionstart.hour:
            dt = dt - timedelta(days=1)
        if offset:
            dt = dt - timedelta(days=offset)
        dt = dt.replace(
            hour=sessionstart.hour,
            minute=sessionstart.minute,
            second=sessionstart.second,
            microsecond=sessionstart.microsecond)
    elif timeframe == bt.TimeFrame.Weeks:
        if dt.weekday() != 6:
            # sunday is start of week at 5pm new york
            dt = dt - timedelta(days=dt.weekday() + 1)
        if offset:
            dt = dt - timedelta(days=offset * 7)
        dt = dt.replace(
            hour=sessionstart.hour,
            minute=sessionstart.minute,
            second=sessionstart.second,
            microsecond=sessionstart.microsecond)
    elif timeframe == bt.TimeFrame.Months:
        if offset:
            dt = dt - timedelta(days=(min(28 + dt.day, 31)))
        # last day of month
        last_day_of_month = dt.replace(day=28) + timedelta(days=4)
        last_day_of_month = last_day_of_month - timedelta(
            days=last_day_of_month.day)
        last_day_of_month = last_day_of_month.day
        # start of month
        if dt.day < last_day_of_month:
            dt = dt - timedelta(days=dt.day)
        dt = dt.replace(
            hour=sessionstart.hour,
            minute=sessionstart.minute,
            second=sessionstart.second,
            microsecond=sessionstart.microsecond)
    return dt


class CSVAdjustTime(bt.feeds.GenericCSVData):

    params = dict(
        adjstarttime=False,
    )

    def _loadline(self, linetokens):
        res = super(CSVAdjustTime, self)._loadline(linetokens)
        if self.p.adjstarttime:
            # move time to start time of next candle
            # and subtract 0.1 miliseconds (ensures no
            # rounding issues, 10 microseconds is minimum)
            new_date = getstarttime(
                self._timeframe,
                self._compression,
                self.datetime.datetime(0),
                self.sessionstart,
                -1) - timedelta(microseconds=100)
            self.datetime[0] = date2num(new_date)
        return res

dasch

by using the adjustment above, no need to alter any settings like

        rightedge=True,
        bar2edge=True,
        adjbartime=True,
        boundoff=0,

@Nam-Nguyen for the tick question, you will not experience any issues with backtrader when using tick data, since they are just events at a given time. they will resample correctly

dasch

@Nam-Nguyen you can use my code with the monkey patching of _load to have a common solution for this.

@run-out if the metatrader datafeeds have a datetime which is the start of the period then my solution will work with metatrader datafeeds

dasch

the datetime of the resampled data is the end of the period, there you may use the settings

        rightedge=True,
        bar2edge=True,
        adjbartime=True,
        boundoff=0,

to adjust the time to be the start of a period