Problem with resampling 1m data to 5m/15m
-
I have 1m OHLC data and would like to compress that into 5m and 15m data. I know Backtrader has resampling and I tried different options but I still can't get it to align correctly with my data and charts at TradingView.
What I want to achieve:
1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:00
What Backtrader does by default:
1m bars 17:01-17:05 (inclusive) -> 5m bar dated 17:05
Obviously this gives me OHLC data that is not matching what I want so I used the
boundoff=1
option. This results in correct OHLC for the 5m bars but they are dated incorrectly for my needs, like this:1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:04
I tried other resampling options but perhaps I don't fully understand them and I could not get the result I want.
Could you give me some hints if it can be achieved before I dive into the Backtrader code?
-
You want
rightedge=False
. Docs - Data ResamplingBut what you probably really want is to resample your data outise of backtrader and thenn lod it, to avoid a problem with the convention.
-
@backtrader I tried that already and it does not seem to have any effect - I get the same results with either
rightedge=false
ortrue
(default I guess).Might that be a conflict of this option with
boundoff=1
?In any case, I would prefer to avoid processing the data outside of Backtrader because I already am storing 1m bar data so no point storing 5m as well. So I'm thinking about writing a data source for Backtrader which converts this automatically or adding an option to the resampling method that outputs correct bars for my use case. If I would make it work with resampling, does it make sense to put it in a PR?
-
There were several patches to resampling, which probably had an impact on
rightedge
.Use this code as a template to achieve your desired effect.
rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=5) def _new_load(self): ret = self._old_load() if ret: dt = self.datetime.datetime(0) self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=5)) return ret rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)
-
Has there been any update to this code? I tried the suggested code and it still doesn't work.
Basically I am trying to have resampling of 1m to 1h.
So as an example the close of 1m at 10:59 should be the 1h close of 10:00.However I see that the 1m close at 10:59 becomes the 1h close of 11:00.
How can I fix this?
-
Lexar, having a minute bar from 10:59 go back in time to generate a result at 10:00 is a good way to make a system that won't work.
-
Thanks for this solution @backtrader it works like a charm !
I was wondering how you can pass an argument to the new_load function in order to deal with a variable minute (for dealing with multiple timeframes in a strategy).I looked on stack overflow and I tried this (but got this error
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute '_old_load'
):def _new_load(self, minutes): ret = self._old_load() if ret: datetime = self.datetime.datetime(0) self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1)) return ret minutes = 240 rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes) rd._old_lo ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)
Thanks :)
-
Hello,
Did anyone figure out a stable way to resample it?
I'm not sure this code will be work well with live data feed.
def _new_load(self, minutes): ret = self._old_load() if ret: datetime = self.datetime.datetime(0) self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1)) return ret minutes = 240 rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes) rd._old_lo ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)
I try to understand
Resampler
class but didn't make it works.rightedge=False, bar2edge=True, adjbartime=True, boundoff=1,
with this setting only last bar get right result, earlier data have good prices and volume but time is wrong.
-
Could you include sample outputs expected and what you are actually getting?
-
Hello @run-out , thank you for your question, I reproduce buy test like this:
I want to resample M1 to M5 data export from Metatrader 5. So I export M1 data from Metatrader 5, and my strategy has using M1 and M5 datafeeds, then I resample data from M1 to M5 instead of get another M5 data (and I thing in live datafeed, Tick data will resample to M1 data the same way resample M1 to M5, right?). So, then I want to verify that my resample setting work well.
- M1 datafeeds is data from
2020.09.01 01:00:00
to2020.09.01 01:10:00
in 10 minutes to resample
<DATE> <TIME> <OPEN> <HIGH> <LOW> <CLOSE> <TICKVOL> <VOL> <SPREAD> 2020.09.01 01:00:00 1.33655 1.33691 1.33655 1.33667 76 0 28 2020.09.01 01:01:00 1.33667 1.33676 1.33665 1.33676 21 0 26 2020.09.01 01:02:00 1.33676 1.33680 1.33672 1.33673 12 0 18 2020.09.01 01:03:00 1.33673 1.33674 1.33670 1.33671 30 0 16 2020.09.01 01:04:00 1.33672 1.33681 1.33670 1.33679 67 0 16 2020.09.01 01:05:00 1.33679 1.33681 1.33675 1.33681 87 0 20 2020.09.01 01:06:00 1.33679 1.33686 1.33676 1.33678 78 0 19 2020.09.01 01:07:00 1.33679 1.33686 1.33677 1.33685 87 0 18 2020.09.01 01:08:00 1.33684 1.33687 1.33676 1.33677 54 0 13 2020.09.01 01:09:00 1.33677 1.33678 1.33670 1.33678 22 0 10 2020.09.01 01:10:00 1.33678 1.33678 1.33676 1.33677 12 0 12
- M5 datafeeds is data from
2020.09.01 01:00:00
to2020.09.01 01:10:00
in 10 minutes to verify.
<DATE> <TIME> <OPEN> <HIGH> <LOW> <CLOSE> <TICKVOL> <VOL> <SPREAD> 2020.09.01 01:00:00 1.33655 1.33691 1.33655 1.33679 206 0 16 2020.09.01 01:05:00 1.33679 1.33687 1.33670 1.33678 328 0 10 2020.09.01 01:10:00 1.33678 1.33685 1.33664 1.33667 84 0 6
- My testing source code:
import backtrader as bt class MT5CSVData(bt.feeds.GenericCSVData): params = dict( dtformat='%Y.%m.%d', tmformat='%H:%M:%S', datetime=0, time=1, open=2, high=3, low=4, close=5, volume=6, openinterest=-1, separator="\t", timeframe=bt.TimeFrame.Minutes, compression=1, ) class DebugStrategy(bt.Strategy): def next(self): print(dict( datetime=self.data.datetime.datetime().strftime("%Y-%m-%d %H:%M:%S"), open=self.data.open[0], high=self.data.high[0], low=self.data.low[0], close=self.data.close[0], volume=self.data.volume[0]) ) if __name__ == "__main__": data = MT5CSVData( dataname="./GBPUSD_M1_202009010100_202009010110.csv", timeframe=bt.TimeFrame.Minutes, compression=1, ) cerebro = bt.Cerebro(stdstats=False) kwargs = dict( timeframe=bt.TimeFrame.Minutes, compression=5, rightedge=True, bar2edge=True, adjbartime=True, # boundoff=1, ) cerebro.resampledata(data, **kwargs) cerebro.addstrategy(DebugStrategy) cerebro.run( exactbars=0 )
- Setting 1: Default setting:
rightedge=True, bar2edge=True, adjbartime=True, boundoff=0,
I get response:
{'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33667, 'volume': 76.0} {'datetime': '2020-09-01 01:05:00', 'open': 1.33667, 'high': 1.33681, 'low': 1.33665, 'close': 1.33681, 'volume': 217.0} {'datetime': '2020-09-01 01:10:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33677, 'volume': 253.0}
Set
rightedge=False
get the same response.- Setting 2:
boundoff=1
rightedge=False, bar2edge=True, adjbartime=True, boundoff=1,
I get good response with M5 verify data at open, high, low, close and volume data but wrong at datetime
{'datetime': '2020-09-01 01:04:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0} {'datetime': '2020-09-01 01:09:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0} {'datetime': '2020-09-01 01:10:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}
- With update code:
rd = cerebro.resampledata(data, **kwargs) def _new_load(self): ret = self._old_load() if ret: dt = self.datetime.datetime(0) self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=4)) return ret rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)
I get perfect response
{'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0} {'datetime': '2020-09-01 01:05:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0} {'datetime': '2020-09-01 01:05:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}
But I curious do we have any stable way to resolve this problem? Do Tick data with resample same with this way?
I didn't test on live date to verify it. - M1 datafeeds is data from
-
@Nam-Nguyen said in Problem with resampling 1m data to 5m/15m:
Metatrader 5
Anyone have experience with MetaTrader datafeeds?
-
i fixed issues with my datafeeds by adjusting the time to the end of the period they are on. Backtrader expects somehow the endtime of a period, while most data will have the starttime of a period as the datetime.
A custom csv feed with time adjustment which can be used to fix that:
import backtrader as bt from backtrader.utils import date2num from datetime import datetime, time, timedelta def getstarttime(timeframe, compression, dt, sessionstart=None, offset=0): ''' This method will return the start of the period based on current time (or provided time). ''' if sessionstart is None: # use UTC 22:00 (5:00 pm New York) as default sessionstart = time(hour=22, minute=0, second=0) if dt is None: dt = datetime.utcnow() if timeframe == bt.TimeFrame.Seconds: dt = dt.replace( second=(dt.second // compression) * compression, microsecond=0) if offset: dt = dt - timedelta(seconds=compression*offset) elif timeframe == bt.TimeFrame.Minutes: if compression >= 60: hours = 0 minutes = 0 # get start of day dtstart = getstarttime(bt.TimeFrame.Days, 1, dt, sessionstart) # diff start of day with current time to get seconds # since start of day dtdiff = dt - dtstart hours = dtdiff.seconds//((60*60)*(compression//60)) minutes = compression % 60 dt = dtstart + timedelta(hours=hours, minutes=minutes) else: dt = dt.replace( minute=(dt.minute // compression) * compression, second=0, microsecond=0) if offset: dt = dt - timedelta(minutes=compression*offset) elif timeframe == bt.TimeFrame.Days: if dt.hour < sessionstart.hour: dt = dt - timedelta(days=1) if offset: dt = dt - timedelta(days=offset) dt = dt.replace( hour=sessionstart.hour, minute=sessionstart.minute, second=sessionstart.second, microsecond=sessionstart.microsecond) elif timeframe == bt.TimeFrame.Weeks: if dt.weekday() != 6: # sunday is start of week at 5pm new york dt = dt - timedelta(days=dt.weekday() + 1) if offset: dt = dt - timedelta(days=offset * 7) dt = dt.replace( hour=sessionstart.hour, minute=sessionstart.minute, second=sessionstart.second, microsecond=sessionstart.microsecond) elif timeframe == bt.TimeFrame.Months: if offset: dt = dt - timedelta(days=(min(28 + dt.day, 31))) # last day of month last_day_of_month = dt.replace(day=28) + timedelta(days=4) last_day_of_month = last_day_of_month - timedelta( days=last_day_of_month.day) last_day_of_month = last_day_of_month.day # start of month if dt.day < last_day_of_month: dt = dt - timedelta(days=dt.day) dt = dt.replace( hour=sessionstart.hour, minute=sessionstart.minute, second=sessionstart.second, microsecond=sessionstart.microsecond) return dt class CSVAdjustTime(bt.feeds.GenericCSVData): params = dict( adjstarttime=False, ) def _loadline(self, linetokens): res = super(CSVAdjustTime, self)._loadline(linetokens) if self.p.adjstarttime: # move time to start time of next candle # and subtract 0.1 miliseconds (ensures no # rounding issues, 10 microseconds is minimum) new_date = getstarttime( self._timeframe, self._compression, self.datetime.datetime(0), self.sessionstart, -1) - timedelta(microseconds=100) self.datetime[0] = date2num(new_date) return res
-
by using the adjustment above, no need to alter any settings like
rightedge=True, bar2edge=True, adjbartime=True, boundoff=0,
@Nam-Nguyen for the tick question, you will not experience any issues with backtrader when using tick data, since they are just events at a given time. they will resample correctly
-
@Nam-Nguyen you can use my code with the monkey patching of _load to have a common solution for this.
@run-out if the metatrader datafeeds have a datetime which is the start of the period then my solution will work with metatrader datafeeds
-
the datetime of the resampled data is the end of the period, there you may use the settings
rightedge=True, bar2edge=True, adjbartime=True, boundoff=0,
to adjust the time to be the start of a period