Porting a pandas dataframe dependent indicator

RandyT

Greetings all,

Working to port an indicator that is currently implemented using pandas dataframes. Not clear to me if there is function in the standard Indicator tools that would replace the pandas capability that I currently depend on.

Here is a bit of code to show what I am trying to do in backtrader. Any suggestions here would be appreciated.

class IND(bt.Indicator):
    lines = ('ind', 'dval1', 'dval2', 'dval3', 'dval4', 'dval5', 'dval6', 'dval7', 'dval8',)

    def __init__(self):
        super(IND, self).__init__()

    def next(self):
        self.dval1[0] = (self.data.close[0] / (self.data.high[0] + self.data.low[0]))
        self.dval2[0] = (self.dval1[0] * 0.90) + (self.dval1[-1] * 0.5)
        self.dval3[0] = (self.dval2[0] * 0.90) + (self.dval2[-1] * 0.5)
        self.dval4[0] = (self.dval3[0] * 0.90) + (self.dval3[-1] * 0.5)
        self.dval5[0] = (self.dval4[0] * 0.90) + (self.dval4[-1] * 0.5)
        self.dval6[0] = (self.dval5[0] * 0.90) + (self.dval5[-1] * 0.5)
        self.dval7[0] = (self.dval6[0] + self.dval6[-1]) / 2
        self.dval8[0] = (self.dval7[0] * 0.60) + (self.dval7[-1] * 0.25)

        self.lines.ind[0] = self.dval8[0]

And here is the relevant pandas dataframe code. Is there a way to do this without pandas or should I just implement as it was using pandas? Comments in the pandas datafeed section of the docs suggests to me that pandas is not required when doing these types of things in backtrader.

def update_indicators(self):
        df = self.hist_ohlc.append(self.last_prices.resample('D').ohlc()[0])

        df['value1'] = df['close'] / (df['high'] + df['low'])
        df['value2'] = (df['value1'] * 0.90) + (df['value1'].shift(1) * 0.5)
        df['value3'] = (df['value2'] * 0.90) + (df['value2'].shift(1) * 0.5)
        df['value4'] = (df['value3'] * 0.90) + (df['value3'].shift(1) * 0.5)
        df['value5'] = (df['value4'] * 0.90) + (df['value4'].shift(1) * 0.5)
        df['value6'] = (df['value5'] * 0.90) + (df['value5'].shift(1) * 0.5)
        df['value7'] = (df['value6'] + df['value6'].shift(1)) / 2
        df['value8'] = (df['value7'] * 0.60) + (df['value7'].shift(1) * 0.25)

        ind = df['value8']

backtrader

backtrader tries to give you what Pandas does (one-shot operations), whilst at the same time being built from the ground up to support step-by-step operations, which is of course needed to connect to a live trading and to do the best possible backtesting.

Let's try to use the arsenal provided by backtrader to port the code. It is actually not that different.

class IND(bt.Indicator):
    lines = ('ind',)

    def __init__(self):  # Indicator has no __init__, no need to use super here
        dval1 = self.data.close / (self.data.high + self.data.low)
        ...
        dval8 = dval7 * 0.6 + dval7(-1) * 0.25
        self.lines.ind = dval8

Et voilá!. Pass your pandas DataFrame using the backtrader.feeds.PandasData and if your column names are datetime, open, high, low, close, volume (and optionally openinterest) and it will work for this and any other indicator.

As you see, everything has been defined in __init__. Let's look at the 1st line:

        dval1 = self.data.close / (self.data.high + self.data.low)

It seems a regular arithmetic operation but there are no indices to self.data.close and the others. This is because the operation generates an object which is assigned to dval1. This object will later deliver values during the backtesting/trading phase and the values will be addressable (if needed to) with the [] operator.

Looking at the last calculation:

dval8 = dval7 * 0.6 + dval7(-1) * 0.25

dval7 has been calculated finally calculated from the operations that start with dval1. dval7 is also an object and this operation says:

When dval8 is calculated, please use the current value of dval7 multiplied by 0.6 and the previous value of dval7 ,using the (-1) notation, multiplied by 0.25

The final action is:

self.lines.ind = dval8

Which translates to:

During backtesting/trading, put the currently calculated value of dval8 in the current index of the output line ind

As you see, there is no need to declare the different dvalX as lines, because these are intermediate calculations and not output lines. These intermediate operations don't even need to be kept in instance attributes, because the final assingment to self.lines.ind ensures python keeps all objects alive.

RandyT

Thanks for the thorough explanation. Seems the tools are well thought out.

And a follow-on question, how would I get a slice of values held as dval8? I would typically do something as follows with the DataFrame. My attempts with dval8.get() is not yet giving me the expected result.

df[-50:].dval8.values

backtrader

The design of the platform decided against slicing even if this may sound non-pythonic. The rationale behind this:

The choice to use [0] for the current moment (values which are being produced/calculated in this iteration) and [-1], [-2] ... to address the previous values
As such ... a slicing to get the current moment and the previous 4 would be something like: [-4:0], which would be mostly confusing.

Now and straight to the question. It depends where you want to apply get.

If you are trying to do that in __init__: it won't work.

dval8 is an object still to be filled during the iterations of the backtesting/trading process. During __init__, it is empty, awaiting to be triggered during each iteration for calculations and adding values to its own internal buffer
If you are trying to do that in next:
```
my_values = dval8.get(size=x, ago=y)
```

In this context:

size should be straightforward: how many values must be returned
ago offset to look (backwards, because the future is not yet known)
0 starts with the current moment (this is the default in the call)
-1 starts with the previous value and the more negative the greater the offset backwards

If the current values in dval8 are for example [1, 2, 3, 4, 5, 6 , 7, 8, 9]:

dval.get(size=3, ago=0) returns [7, 8, 9]
dval.get(size=2, ago=-1) returns [7, 8]
dval.get(size=4, ago=-2) returns [4, 5, 6, 7]

Hopefully this helps

RandyT

Helpful. Thank you. Off and running here....