Rolling correlation matrix indicator

adarakchian

Dear all,

I am using an exponentially weighted correlation matrix and trying to figure out what is the best way to implement it as an indicator.

I am passing all datas to the indicator and on every bar I can compute the matrix. Now, what is the best way to store it? "Hacky" way is to just have it as a prop and update on every bar. However, this will not work for preload because it must be stored on the lines. Otherwise only the matrix will be available during the backtest. And lines by design only store floats. Not sure what is the best course of action here.

adarakchian

Solving the problem above made me wonder what is the "backtrader's way" of thinking about indicators/strategies (or any other object that holds lines) where the datapoint cannot be expressed as a float but rather a matrix (or any tensor)?

vladisld

@adarakchian said in Rolling correlation matrix indicator:

lines by design only store floats

Not all lines store floats, some store other types like datetime for example:

class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

Briefly looking at the source code of LineBuffer, I do not see where the assumption of the float value type is used. So just theoretically you may store values of any type inside the private lines of your indicator for example.

Please correct me if I'm wrong.

adarakchian

@vladisld

adarakchian

@vladisld thank you very much! This is exactly the pointer I was looking for!

adarakchian

@vladisld Hey Vlad, thanks again for your help! I've spent some time reading the above (and associated) and I think that it is impossible to achieve what I want without forking. Just thought I would share my findings, let me know if you agree or not.

The reason why lines are only accepting floats is the fact that the array in the LineBuffer is an instance of array.array set to hold double. So if we create a flag on LineBuffer, something like is_generic = True, then inside reset we can set array to list.

      def reset(self):

        if self.mode == self.QBuffer:
            self.array = collections.deque(maxlen=self.maxlen + self.extrasize)
            self.useislice = True
        elif self.is_generic:
            self.array = list()
            self.useislice = False
        else:
            self.array = array.array(str('d'))
            self.useislice = False

        self.lencount = 0
        self.idx = -1
        self.extension = 0

The above works. However, we need to inform LineBuffer which lines are supposed to be generic. And here starts the painful part. The buffer is instantiated inside Lines and the Lines are created inside MetaLineSeries. The best place to add the information about lines being "generic" is (in my opinion) in the init method of the Lines:

        for line, linealias in enumerate(self._getlines()):
            kwargs = dict()
            self.lines.append(LineBuffer(**kwargs))

can be changed to

        for line, linealias in enumerate(self._getlines()):
            kwargs = dict()
            if linealias in genric_lines:
                    kwargs['is_generic'] = True
            self.lines.append(LineBuffer(**kwargs))

However, there is no way (or at least I have not found one) to inject that custom Lines object into MetaLineSeries (it is set in the __new__ method via

lines = getattr(cls, 'lines', Lines)

So it means that one will need to re-build the whole structure, from bt.DataBase (which the data feed inherits from) to LineSeries to change the pointer inside MetaLinesSeries from backtrader's original Lines to the modified ones.

This is doable but it creates a new issue: datafeeds that inherit from the custom LineSeries do not pass the equality inside the LineSeriesMaker:

def LineSeriesMaker(arg, slave=False):
    if isinstance(arg, LineSeries):
        return arg

    return LineSeriesStub(arg, slave=slave)

because the arg is obviously not bt.LineSeries but myown.LineSeries.

And the LineSeriesMaker is actually called as a part of the LineIterator across all conceptual parts of the framework. And changing LineSeriesMaker would require updating all those parts (basestrategy, baseindicator, baseobserver, etc.). In this case forking would be much easier and neater but I am somewhat unwilling to go that way.

This is where I am at the moment. TLDR: cannot inject custom LineBuffer() as a dependency and fork seems to be the only (highly undesirable) way. Any advice?

vladisld

@adarakchian you're right regarding the array.array configured for holding double type only. However this is true for the default case.

Did you try to set the buffer to QBuffer mode instead. It seems that in QBuffer state the array.array is not used.

I've not tried it myself yet - but it seems it worth to check this path.

adarakchian

@vladisld Just had a eureka moment - there is a much simpler solution that seems to work. In the start method of the custom data feed that inherits from DataBase, we can simply do:

        for line_alias in private_lines:
            self._getline(line_alias).array = list()

Might be crude but does the job with minimal effort and no fork is needed :)

vladisld

@adarakchian so in both your way and the way I've proposed - the main issue is to replace the array.array with other container.

Using the QBuffer mode seems to be less intrusive way, while in your case the container type is changed forcefully.

In both cases the target seems to be the same: the container type should not make an assumption about the value type used for line data.

One more note: there are some assumptions about the underlying container type inside the LineBuffer - and this is reflected in the LineBuffer.mode ( QBuffer or Unbounded).

Depending on this mode, different methods are used for slicing and other operations inside the LineBuffer methods

Given that, it is important to keep the container type compatible with what LineBuffer logic is expecting IMHO