Possible bug when mixing timeframes

Mariano Bengeldorf

Hi there... it's me again...
I am mixing datafeeds of different timeframes, following this documentation: https://www.backtrader.com/docu/data-multitimeframe/data-multitimeframe/

It works well, but I found some inconsistencies.
Basically what I am doing is entering variables by args using the parser.

parser.add_argument('-t', '--timeframe',
                        type=str,
                        nargs='+',
                        default='1w',
                        choices=['1m', '5m', '15m', '30m', '1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1w', '1y'],)

Then I am using a dict to interprete the choices:

tframes = {
    '1m': [bt.TimeFrame.Minutes, 1],
    '5m': [bt.TimeFrame.Minutes, 5],
    '15m': [bt.TimeFrame.Minutes, 15],
    '30m': [bt.TimeFrame.Minutes, 30],
    '1h': [bt.TimeFrame.Minutes, 60],
    '2h': [bt.TimeFrame.Minutes, 120],
    '3h': [bt.TimeFrame.Minutes, 180],
    '4h': [bt.TimeFrame.Minutes, 240],
    '6h': [bt.TimeFrame.Minutes, 360],
    '12h': [bt.TimeFrame.Minutes, 720], 
    '1d': [bt.TimeFrame.Days, 1],
    '1w': [bt.TimeFrame.Weeks, 1],
    '1M': [bt.TimeFrame.Months, 1],
    '1y': [bt.TimeFrame.Years, 1]

}

then I add the data from a CVS file (in minutes) to cerebro like this:

timeframes = args.timeframe
timeframeslen = len(timeframes)
cerebro.resampledata(data, timeframe=tframes[timeframes[i]][0], compression=tframes[timeframes[i]][1])

Finally I call a custom indicator like this (in this example I am mixing three different timeframes):

self.mresistance0 = SandR(self.data0)
self.mresistance0 = SandR(self.data1, plot = False)
self.mresistance0 = SandR(self.data2, plot = False)

Then in the terminal I can call the scrip and add arguments like "1d 1w 1M" and it will plot something like this(which is ok):

Or I can use "4h 1w 1M" and I get the following (which is also ok):

It is also ok what I get when I use "1h 1w 1M":

The issue shows up when I mix hourly datafeeds with daily datafeeds. What happens is that the second datafeed just disappears.
For example, when I use 1h 1d 1W, I get this:

1H-1D-1W x.png

In the chart above you can check that the 1d data is just not there, neither in the calculations

Another example, when I use 4H 1D 1M, the same issue:

4H-1D-1M x.png

Please, help! I am suffering so much!!

Thanks in advance

Mariano Bengeldorf

One correction:
In the third code that I posted above I made a mistake, this is what I should have posted:

args = parse_args()
    timeframes = args.timeframe
    timeframeslen = len(timeframes)
 data = readCVS()
        for i in range(timeframeslen):
            cerebro.resampledata(data, timeframe=tframes[timeframes[i]][0], compression=tframes[timeframes[i]][1])

Sorry for that...

backtrader

And the data?

In any case, I fail to understand what is your problem and your suffering. There seems to be a matplotlib thing with a specific combination of things, probably due to incompatible lengths along the axis.

The data is for sure being fed to next and as stated several times, "plotting is only a visual aid and not the target of the framework"

Mariano Bengeldorf

Hi, thanks for answering.
I am suffering because I've been trying to figure out why is this happing for hours with no luck at all...

Answering your question about the data: The same problem happens with the data, it is not just a plotting issue.

I entered print(self.data.close[0], self.data1.close[0], self.data2.close[0]) at the end of the Next method within the strategy to se what comes out.And I can confirm that issue is the same at a data level.
When I use something like "4h 1w 1M" as an argument I get the right output. But, when I enter an argument mixing hours with days (like "4H 1D 1M"), I just don't get anything printed at all. It looks like is just skipping the Next method of the strategy.
Any ideas?

Many thanks for your kind help...

ab_trader

Don't have much experience with the intraday data. But for me it seems something is wrong with sessionstart and sessionend.

backtrader

@Mariano-Bengeldorf said in Possible bug when mixing timeframes:

Answering your question about the data: The same problem happens with the data, it is not just a plotting issue.

Your next statements contradict this.

@Mariano-Bengeldorf said in Possible bug when mixing timeframes:

When I use something like "4h 1w 1M" as an argument I get the right output. But, when I enter an argument mixing hours with days (like "4H 1D 1M"), I just don't get anything printed at all. It looks like is just skipping the Next method of the strategy.

But the 4H data gets plotted, so the data is there ... contradicting your previous statement.

Any ideas?

I know it seems obvious to you, but you only show how you resample data (a redacted version, hence the errors) and not the rest of things you do, like adding indicators.

Mariano Bengeldorf

@backtrader I see your point now.
I am sorry for not being as clear as I am supposed to be.
What I am trying to explain is that when I use as an argument "4h 1d 1M"
the 4H and 1M do get plotted but the 1D does not (check the previous charts). That is the issue.
At the same time, when I use "4h 1d 1M" and create a print() in the next method within the strategy (the way I explained in my previous post) the data values do not get printed in the terminal because it looks like it is skipping the next method. The data is there (at least for the 4h and 1M, not for 1D) but the next method is skipped.
None of all this happens when I use as an argument like 4h 1w 1M, which is not mixing hours with days.
I hope I was clear enough this time. Sorry for the misunderstanding.
Thanks

backtrader

It's all clear ... you still fail to understand that posting some charts with some "redacted" code is good for nothing.

As you may see in the chart below ... it works, hence: "bug in your code".

Mariano Bengeldorf

Hi @backtrader
Thanks for your answer. I was just trying to make things simpler for you, I am sorry if it turned out to be the other way. Not my intention.
BTW, I love what your are doing and thanks for sharing with all of us.
I took a bit to answer because I was trying to figure it out by my self, but no luck at all.
Here goes the code, thanks for any advise you can provide:

from __future__ import (absolute_import, division, print_function, unicode_literals)
import backtrader as bt
from collections import OrderedDict
import pandas as pd
import argparse


def parse_args():
    parser = argparse.ArgumentParser(
        description='RSI strategy')

    parser.add_argument('-nh', '--noheaders',
                        action='store_true',
                        default=False,
                        required=False,
                        help='Do not use header rows')

    parser.add_argument('-t', '--timeframe',
                        type=str,
                        nargs='+',
                        default='1w',
                        choices=['1m', '5m', '15m', '30m', '1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1w', '1y'],
                        help='The timeframe for second indicator')

    parser.add_argument('-fp', '--filepath',
                        type=str,
                        default='binance-BTCUSDT-1m-2019-01-01.csv',
                        required=False,
                        help='File path (../../filename.csv')

    parser.add_argument('-hi', '--history',
                        type=int,
                        default=30,
                        required=False,
                        help='Number of days to retrieve')

    parser.add_argument('-c', '--cash',
                        type=int,
                        default=10000,
                        required=False,
                        help='Starting Cash')

    parser.add_argument('-st', '--stats',
                        action='store_true',
                        default=False,
                        required=False,
                        help='Print the stats')

    return parser.parse_args()


def readCVS():
    # Get a pandas dataframe
    datapath = args.filepath

    # Simulate the header row isn't there if noheaders requested
    skiprows = 1 if args.noheaders else 0
    header = None if args.noheaders else 0

    dataframe = pd.read_csv(datapath,
                            skiprows=skiprows,
                            header=header,
                            parse_dates=True,
                            index_col=0)

    dataframe.index = pd.to_datetime(dataframe.index, unit='ms')

    # Pass it to the backtrader datafeed and add it to the cerebro
    return bt.feeds.PandasData(dataname=dataframe, timeframe=bt.TimeFrame.Minutes, openinterest=None)


def resismet(maxima,resistances, resNum=15, tolPercRes=2):
    # creates a list of resistances with length resNum and elminates already broken resistances
    # within a tolerance of tolerancePerc
    toleranceResist = maxima * tolPercRes / 100 #todo mejorar este calculo
    if maxima > 0:
        if all(abs(maxima - i) > toleranceResist for i in resistances):
            resistances.append(maxima)
        else:
            removeResist = [i for i in resistances if maxima - i > 0 and maxima - i <= toleranceResist]
            if len(removeResist) != 0:
                for i in removeResist:
                    resistances.remove(i)
                resistances.append(maxima)

    lenR = len(resistances)

    if resNum != None and lenR > resNum:
        resistances[0:lenR - resNum] = []
    return (resistances)


class SandR(bt.Indicator):

    params = OrderedDict([
        ('nresist', 20),
        ('tolerance', 2),
        ('tolerance2', 2),
        ('period', 1),
        ('barplot', False),
        ('timeframeslen', 4)
    ])

    nresist = params['nresist']

    mlinelist = list()

    for i in range( nresist * 2):
        mlinelist.append('mresistance' + str(i))

    lines = tuple(mlinelist)  # creates the lines based on the nresist parameter

    plotinfo = dict(subplot=False,
                     )
    resdic = OrderedDict()

    for i in range(params['timeframeslen']-1,-1,-1):
        j = 'resistances' + str(i)
        resdic[j] = list()

    def __init__(self):
        #look for local maxima
        higherthanlast = bt.And(self.data.high(-1) > self.data.high(-2), self.data.high(-1) > self.data.high(0))
        self.maxima = higherthanlast * self.data.high(-1)


    def next(self, resdic = resdic):
        dataid = self.data._id-1
        resistancesid = 'resistances'+str(dataid)
        resdic[resistancesid] = resismet(self.maxima[0], resdic[resistancesid], self.params.nresist,
                                 self.params.tolerance)
        tolRestoSupPer = self.params.tolerance2/100

        if dataid == 0:
            inputhigh = self.data.high[-1]
            for j in resdic.values():
                ResToSupport = [i for i in j if inputhigh - i > i*tolRestoSupPer]
                if len(ResToSupport) != 0:
                    for k in ResToSupport:
                        j.remove(k)

        mresistances = [i for j in resdic.values() for i in j]

        for i in (range(len(mresistances)-1,-1,-1)):
            if mresistances.count(mresistances[i]) > 1:
                del mresistances[i]

        if dataid == 0:
            if len(mresistances) > 0:
                n = 0
                for i in self.mlinelist:
                    if len(mresistances) > n:
                        exec('self.lines.' + i + '[0]' '= mresistances[n]')
                        n = n + 1


class secondStrategy(bt.Strategy):

    def __init__(self):

        self.mresistance0 = SandR(self.data0, plotname='Resistance', plotvaluetags=False, timeframeslen = 2)

    def next(self):
        print('=========')


tframes = {
    '1m': [bt.TimeFrame.Minutes, 1],
    '5m': [bt.TimeFrame.Minutes, 5],
    '15m': [bt.TimeFrame.Minutes, 15],
    '30m': [bt.TimeFrame.Minutes, 30],
    '1h': [bt.TimeFrame.Minutes, 60],
    '2h': [bt.TimeFrame.Minutes, 120],
    '3h': [bt.TimeFrame.Minutes, 180],
    '4h': [bt.TimeFrame.Minutes, 240],
    '6h': [bt.TimeFrame.Minutes, 360],
    '12h': [bt.TimeFrame.Minutes, 720],
    '1d': [bt.TimeFrame.Days, 1],
    '1w': [bt.TimeFrame.Weeks, 1],
    '1M': [bt.TimeFrame.Months, 1],
    '1y': [bt.TimeFrame.Years, 1]

}

# Variable for our starting cash
startcash = 100000

if __name__ == '__main__':
    cerebro = bt.Cerebro()
    args = parse_args()
    timeframes = args.timeframe

    if not args.filepath:
        data = loadExchange(timeframes[0])
        cerebro.adddata(data)
        for i in range(1,len(timeframes)):
            cerebro.resampledata(data, timeframe=tframes[timeframes[i]][0], compression=tframes[timeframes[i]][1])
    else:
        data = readCVS()
        for i in range(len(timeframes)):
            cerebro.resampledata(data, timeframe=tframes[timeframes[i]][0], compression=tframes[timeframes[i]][1])

    cerebro.broker.setcash(args.cash)
    cerebro.addstrategy(secondStrategy)
    cerebro.run()

    # Get final portfolio Value
    portvalue = cerebro.broker.getvalue()
    pnl = portvalue - startcash

    # Print out the final result
    print('Final Portfolio Value: ${}'.format(portvalue))
    print('P/L: ${}'.format(pnl))

    # Finally plot the end results
    cerebro.plot(style='candlestick')

Again, thanks

Mariano Bengeldorf

@backtrader BTW, do you know how can I replace the following line?:

exec('self.lines.' + i + '[0] = DfMresistances.value.take([n])')

I tried to use thes two alternatives, but they don't work:

#option 1: 
setattr(self, 'lines.' + i + '[0]', DfMresistances.value.take([n]))

#option 2:
setattr(self.lines, + i + '[0]', DfMresistances.value.take([n]))