Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Problem with resampling 1m data to 5m/15m

    General Discussion
    8
    15
    2232
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ppawel
      ppawel last edited by

      I have 1m OHLC data and would like to compress that into 5m and 15m data. I know Backtrader has resampling and I tried different options but I still can't get it to align correctly with my data and charts at TradingView.

      What I want to achieve:

      1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:00

      What Backtrader does by default:

      1m bars 17:01-17:05 (inclusive) -> 5m bar dated 17:05

      Obviously this gives me OHLC data that is not matching what I want so I used the boundoff=1 option. This results in correct OHLC for the 5m bars but they are dated incorrectly for my needs, like this:

      1m bars 17:00-17:04 (inclusive) -> 5m bar dated 17:04

      I tried other resampling options but perhaps I don't fully understand them and I could not get the result I want.

      Could you give me some hints if it can be achieved before I dive into the Backtrader code?

      1 Reply Last reply Reply Quote 2
      • B
        backtrader administrators last edited by

        You want rightedge=False. Docs - Data Resampling

        But what you probably really want is to resample your data outise of backtrader and thenn lod it, to avoid a problem with the convention.

        ppawel 1 Reply Last reply Reply Quote 1
        • ppawel
          ppawel @backtrader last edited by

          @backtrader I tried that already and it does not seem to have any effect - I get the same results with either rightedge=false or true (default I guess).

          Might that be a conflict of this option with boundoff=1?

          In any case, I would prefer to avoid processing the data outside of Backtrader because I already am storing 1m bar data so no point storing 5m as well. So I'm thinking about writing a data source for Backtrader which converts this automatically or adding an option to the resampling method that outputs correct bars for my use case. If I would make it work with resampling, does it make sense to put it in a PR?

          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            There were several patches to resampling, which probably had an impact on rightedge.

            Use this code as a template to achieve your desired effect.

                rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=5)
            
                def _new_load(self):
                    ret = self._old_load()
                    if ret:
                        dt = self.datetime.datetime(0)
                        self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=5))
            
                    return ret
            
                rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)
            
            1 Reply Last reply Reply Quote 0
            • L
              lexar last edited by

              Has there been any update to this code? I tried the suggested code and it still doesn't work.

              Basically I am trying to have resampling of 1m to 1h.
              So as an example the close of 1m at 10:59 should be the 1h close of 10:00.

              However I see that the 1m close at 10:59 becomes the 1h close of 11:00.

              How can I fix this?

              1 Reply Last reply Reply Quote 0
              • B
                bigdavediode last edited by

                Lexar, having a minute bar from 10:59 go back in time to generate a result at 10:00 is a good way to make a system that won't work.

                1 Reply Last reply Reply Quote 0
                • B
                  balibou last edited by

                  Thanks for this solution @backtrader it works like a charm !
                  I was wondering how you can pass an argument to the new_load function in order to deal with a variable minute (for dealing with multiple timeframes in a strategy).

                  I looked on stack overflow and I tried this (but got this error AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute '_old_load'):

                  def _new_load(self, minutes):
                      ret = self._old_load()
                      if ret:
                          datetime = self.datetime.datetime(0)
                          self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1))
                  
                      return ret
                  
                  minutes = 240
                  rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes)
                  rd._old_lo
                  ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)
                  

                  Thanks :)

                  Nam Nguyen 1 Reply Last reply Reply Quote 0
                  • Nam Nguyen
                    Nam Nguyen @balibou last edited by

                    Hello,

                    Did anyone figure out a stable way to resample it?

                    I'm not sure this code will be work well with live data feed.

                    def _new_load(self, minutes):
                        ret = self._old_load()
                        if ret:
                            datetime = self.datetime.datetime(0)
                            self.datetime[0] = bt.date2num(datetime - timedelta(minutes=minutes - 1))
                    
                        return ret
                    
                    minutes = 240
                    rd = cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=minutes)
                    rd._old_lo
                    ad, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)(minutes)
                    

                    I try to understand Resampler class but didn't make it works.

                            rightedge=False,
                            bar2edge=True,
                            adjbartime=True,
                            boundoff=1,
                    

                    with this setting only last bar get right result, earlier data have good prices and volume but time is wrong.

                    1 Reply Last reply Reply Quote 0
                    • run-out
                      run-out last edited by

                      Could you include sample outputs expected and what you are actually getting?

                      RunBacktest.com

                      Nam Nguyen 1 Reply Last reply Reply Quote 0
                      • Nam Nguyen
                        Nam Nguyen @run-out last edited by

                        Hello @run-out , thank you for your question, I reproduce buy test like this:

                        I want to resample M1 to M5 data export from Metatrader 5. So I export M1 data from Metatrader 5, and my strategy has using M1 and M5 datafeeds, then I resample data from M1 to M5 instead of get another M5 data (and I thing in live datafeed, Tick data will resample to M1 data the same way resample M1 to M5, right?). So, then I want to verify that my resample setting work well.

                        1. M1 datafeeds is data from 2020.09.01 01:00:00 to 2020.09.01 01:10:00 in 10 minutes to resample
                        <DATE>	<TIME>	<OPEN>	<HIGH>	<LOW>	<CLOSE>	<TICKVOL>	<VOL>	<SPREAD>
                        2020.09.01	01:00:00	1.33655	1.33691	1.33655	1.33667	76	0	28
                        2020.09.01	01:01:00	1.33667	1.33676	1.33665	1.33676	21	0	26
                        2020.09.01	01:02:00	1.33676	1.33680	1.33672	1.33673	12	0	18
                        2020.09.01	01:03:00	1.33673	1.33674	1.33670	1.33671	30	0	16
                        2020.09.01	01:04:00	1.33672	1.33681	1.33670	1.33679	67	0	16
                        2020.09.01	01:05:00	1.33679	1.33681	1.33675	1.33681	87	0	20
                        2020.09.01	01:06:00	1.33679	1.33686	1.33676	1.33678	78	0	19
                        2020.09.01	01:07:00	1.33679	1.33686	1.33677	1.33685	87	0	18
                        2020.09.01	01:08:00	1.33684	1.33687	1.33676	1.33677	54	0	13
                        2020.09.01	01:09:00	1.33677	1.33678	1.33670	1.33678	22	0	10
                        2020.09.01	01:10:00	1.33678	1.33678	1.33676	1.33677	12	0	12
                        
                        1. M5 datafeeds is data from 2020.09.01 01:00:00 to 2020.09.01 01:10:00 in 10 minutes to verify.
                        <DATE>	<TIME>	<OPEN>	<HIGH>	<LOW>	<CLOSE>	<TICKVOL>	<VOL>	<SPREAD>
                        2020.09.01	01:00:00	1.33655	1.33691	1.33655	1.33679	206	0	16
                        2020.09.01	01:05:00	1.33679	1.33687	1.33670	1.33678	328	0	10
                        2020.09.01	01:10:00	1.33678	1.33685	1.33664	1.33667	84	0	6
                        
                        1. My testing source code:
                        import backtrader as bt
                        
                        
                        class MT5CSVData(bt.feeds.GenericCSVData):
                            params = dict(
                                dtformat='%Y.%m.%d',
                                tmformat='%H:%M:%S',
                                datetime=0,
                                time=1,
                                open=2,
                                high=3,
                                low=4,
                                close=5,
                                volume=6,
                                openinterest=-1,
                        
                                separator="\t",
                                timeframe=bt.TimeFrame.Minutes,
                                compression=1,
                            )
                        
                        
                        class DebugStrategy(bt.Strategy):
                            def next(self):
                                print(dict(
                                    datetime=self.data.datetime.datetime().strftime("%Y-%m-%d %H:%M:%S"),
                                    open=self.data.open[0],
                                    high=self.data.high[0],
                                    low=self.data.low[0],
                                    close=self.data.close[0],
                                    volume=self.data.volume[0])
                                )
                        
                        
                        if __name__ == "__main__":
                            data = MT5CSVData(
                                dataname="./GBPUSD_M1_202009010100_202009010110.csv",
                                timeframe=bt.TimeFrame.Minutes,
                                compression=1,
                            )
                        
                            cerebro = bt.Cerebro(stdstats=False)
                        
                            kwargs = dict(
                                timeframe=bt.TimeFrame.Minutes,
                                compression=5,
                                rightedge=True,
                                bar2edge=True,
                                adjbartime=True,
                                # boundoff=1,
                            )
                            cerebro.resampledata(data, **kwargs)
                        
                            cerebro.addstrategy(DebugStrategy)
                        
                            cerebro.run(
                                exactbars=0
                            )
                        

                        • Setting 1: Default setting:
                                rightedge=True,
                                bar2edge=True,
                                adjbartime=True,
                                boundoff=0,
                        

                        I get response:

                        {'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33667, 'volume': 76.0}
                        {'datetime': '2020-09-01 01:05:00', 'open': 1.33667, 'high': 1.33681, 'low': 1.33665, 'close': 1.33681, 'volume': 217.0}
                        {'datetime': '2020-09-01 01:10:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33677, 'volume': 253.0}
                        

                        Set rightedge=False get the same response.

                        • Setting 2: boundoff=1
                                rightedge=False,
                                bar2edge=True,
                                adjbartime=True,
                                boundoff=1,
                        

                        I get good response with M5 verify data at open, high, low, close and volume data but wrong at datetime

                        {'datetime': '2020-09-01 01:04:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0}
                        {'datetime': '2020-09-01 01:09:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0}
                        {'datetime': '2020-09-01 01:10:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}
                        
                        • With update code:
                            rd = cerebro.resampledata(data, **kwargs)
                        
                            def _new_load(self):
                                ret = self._old_load()
                                if ret:
                                    dt = self.datetime.datetime(0)
                                    self.datetime[0] = bt.date2num(dt - datetime.timedelta(minutes=4))
                        
                                return ret
                        
                            rd._old_load, rd._load = rd._load, _new_load.__get__(rd, rd.__class__)
                        

                        I get perfect response

                        {'datetime': '2020-09-01 01:00:00', 'open': 1.33655, 'high': 1.33691, 'low': 1.33655, 'close': 1.33679, 'volume': 206.0}
                        {'datetime': '2020-09-01 01:05:00', 'open': 1.33679, 'high': 1.33687, 'low': 1.3367, 'close': 1.33678, 'volume': 328.0}
                        {'datetime': '2020-09-01 01:05:00', 'open': 1.33678, 'high': 1.33678, 'low': 1.33676, 'close': 1.33677, 'volume': 12.0}
                        

                        But I curious do we have any stable way to resolve this problem? Do Tick data with resample same with this way?
                        I didn't test on live date to verify it.

                        run-out 1 Reply Last reply Reply Quote 0
                        • run-out
                          run-out @Nam Nguyen last edited by

                          @Nam-Nguyen said in Problem with resampling 1m data to 5m/15m:

                          Metatrader 5

                          Anyone have experience with MetaTrader datafeeds?

                          RunBacktest.com

                          1 Reply Last reply Reply Quote 0
                          • D
                            dasch last edited by

                            i fixed issues with my datafeeds by adjusting the time to the end of the period they are on. Backtrader expects somehow the endtime of a period, while most data will have the starttime of a period as the datetime.

                            A custom csv feed with time adjustment which can be used to fix that:

                            import backtrader as bt
                            from backtrader.utils import date2num
                            from datetime import datetime, time, timedelta
                            
                            
                            
                            def getstarttime(timeframe, compression, dt, sessionstart=None, offset=0):
                                '''
                                This method will return the start of the period based on current
                                time (or provided time).
                                '''
                                if sessionstart is None:
                                    # use UTC 22:00 (5:00 pm New York) as default
                                    sessionstart = time(hour=22, minute=0, second=0)
                                if dt is None:
                                    dt = datetime.utcnow()
                                if timeframe == bt.TimeFrame.Seconds:
                                    dt = dt.replace(
                                        second=(dt.second // compression) * compression,
                                        microsecond=0)
                                    if offset:
                                        dt = dt - timedelta(seconds=compression*offset)
                                elif timeframe == bt.TimeFrame.Minutes:
                                    if compression >= 60:
                                        hours = 0
                                        minutes = 0
                                        # get start of day
                                        dtstart = getstarttime(bt.TimeFrame.Days, 1, dt, sessionstart)
                                        # diff start of day with current time to get seconds
                                        # since start of day
                                        dtdiff = dt - dtstart
                                        hours = dtdiff.seconds//((60*60)*(compression//60))
                                        minutes = compression % 60
                                        dt = dtstart + timedelta(hours=hours, minutes=minutes)
                                    else:
                                        dt = dt.replace(
                                            minute=(dt.minute // compression) * compression,
                                            second=0,
                                            microsecond=0)
                                    if offset:
                                        dt = dt - timedelta(minutes=compression*offset)
                                elif timeframe == bt.TimeFrame.Days:
                                    if dt.hour < sessionstart.hour:
                                        dt = dt - timedelta(days=1)
                                    if offset:
                                        dt = dt - timedelta(days=offset)
                                    dt = dt.replace(
                                        hour=sessionstart.hour,
                                        minute=sessionstart.minute,
                                        second=sessionstart.second,
                                        microsecond=sessionstart.microsecond)
                                elif timeframe == bt.TimeFrame.Weeks:
                                    if dt.weekday() != 6:
                                        # sunday is start of week at 5pm new york
                                        dt = dt - timedelta(days=dt.weekday() + 1)
                                    if offset:
                                        dt = dt - timedelta(days=offset * 7)
                                    dt = dt.replace(
                                        hour=sessionstart.hour,
                                        minute=sessionstart.minute,
                                        second=sessionstart.second,
                                        microsecond=sessionstart.microsecond)
                                elif timeframe == bt.TimeFrame.Months:
                                    if offset:
                                        dt = dt - timedelta(days=(min(28 + dt.day, 31)))
                                    # last day of month
                                    last_day_of_month = dt.replace(day=28) + timedelta(days=4)
                                    last_day_of_month = last_day_of_month - timedelta(
                                        days=last_day_of_month.day)
                                    last_day_of_month = last_day_of_month.day
                                    # start of month
                                    if dt.day < last_day_of_month:
                                        dt = dt - timedelta(days=dt.day)
                                    dt = dt.replace(
                                        hour=sessionstart.hour,
                                        minute=sessionstart.minute,
                                        second=sessionstart.second,
                                        microsecond=sessionstart.microsecond)
                                return dt
                            
                            
                            class CSVAdjustTime(bt.feeds.GenericCSVData):
                            
                                params = dict(
                                    adjstarttime=False,
                                )
                            
                                def _loadline(self, linetokens):
                                    res = super(CSVAdjustTime, self)._loadline(linetokens)
                                    if self.p.adjstarttime:
                                        # move time to start time of next candle
                                        # and subtract 0.1 miliseconds (ensures no
                                        # rounding issues, 10 microseconds is minimum)
                                        new_date = getstarttime(
                                            self._timeframe,
                                            self._compression,
                                            self.datetime.datetime(0),
                                            self.sessionstart,
                                            -1) - timedelta(microseconds=100)
                                        self.datetime[0] = date2num(new_date)
                                    return res
                            
                            
                            1 Reply Last reply Reply Quote 3
                            • D
                              dasch last edited by

                              by using the adjustment above, no need to alter any settings like

                                      rightedge=True,
                                      bar2edge=True,
                                      adjbartime=True,
                                      boundoff=0,
                              

                              @Nam-Nguyen for the tick question, you will not experience any issues with backtrader when using tick data, since they are just events at a given time. they will resample correctly

                              1 Reply Last reply Reply Quote 2
                              • D
                                dasch last edited by

                                @Nam-Nguyen you can use my code with the monkey patching of _load to have a common solution for this.

                                @run-out if the metatrader datafeeds have a datetime which is the start of the period then my solution will work with metatrader datafeeds

                                1 Reply Last reply Reply Quote 2
                                • D
                                  dasch last edited by

                                  the datetime of the resampled data is the end of the period, there you may use the settings

                                          rightedge=True,
                                          bar2edge=True,
                                          adjbartime=True,
                                          boundoff=0,
                                  

                                  to adjust the time to be the start of a period

                                  1 Reply Last reply Reply Quote 2
                                  • 1 / 1
                                  • First post
                                    Last post
                                  Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors