@vladisld Just had a eureka moment - there is a much simpler solution that seems to work. In the start method of the custom data feed that inherits from DataBase, we can simply do:

        for line_alias in private_lines:
            self._getline(line_alias).array = list()

Might be crude but does the job with minimal effort and no fork is needed :)

@vladisld Just had a eureka moment - there is a much simpler solution that seems to work. In the start method of the custom data feed that inherits from DataBase, we can simply do:

        for line_alias in private_lines:
            self._getline(line_alias).array = list()

Might be crude but does the job with minimal effort and no fork is needed :)

@vladisld Hey Vlad, thanks again for your help! I've spent some time reading the above (and associated) and I think that it is impossible to achieve what I want without forking. Just thought I would share my findings, let me know if you agree or not.

The reason why lines are only accepting floats is the fact that the array in the LineBuffer is an instance of array.array set to hold double. So if we create a flag on LineBuffer, something like is_generic = True, then inside reset we can set array to list.

      def reset(self):

        if self.mode == self.QBuffer:
            self.array = collections.deque(maxlen=self.maxlen + self.extrasize)
            self.useislice = True
        elif self.is_generic:
            self.array = list()
            self.useislice = False
        else:
            self.array = array.array(str('d'))
            self.useislice = False

        self.lencount = 0
        self.idx = -1
        self.extension = 0

The above works. However, we need to inform LineBuffer which lines are supposed to be generic. And here starts the painful part. The buffer is instantiated inside Lines and the Lines are created inside MetaLineSeries. The best place to add the information about lines being "generic" is (in my opinion) in the init method of the Lines:

        for line, linealias in enumerate(self._getlines()):
            kwargs = dict()
            self.lines.append(LineBuffer(**kwargs))

can be changed to

        for line, linealias in enumerate(self._getlines()):
            kwargs = dict()
            if linealias in genric_lines:
                    kwargs['is_generic'] = True
            self.lines.append(LineBuffer(**kwargs))

However, there is no way (or at least I have not found one) to inject that custom Lines object into MetaLineSeries (it is set in the __new__ method via

lines = getattr(cls, 'lines', Lines)

So it means that one will need to re-build the whole structure, from bt.DataBase (which the data feed inherits from) to LineSeries to change the pointer inside MetaLinesSeries from backtrader's original Lines to the modified ones.

This is doable but it creates a new issue: datafeeds that inherit from the custom LineSeries do not pass the equality inside the LineSeriesMaker:

def LineSeriesMaker(arg, slave=False):
    if isinstance(arg, LineSeries):
        return arg

    return LineSeriesStub(arg, slave=slave)

because the arg is obviously not bt.LineSeries but myown.LineSeries.

And the LineSeriesMaker is actually called as a part of the LineIterator across all conceptual parts of the framework. And changing LineSeriesMaker would require updating all those parts (basestrategy, baseindicator, baseobserver, etc.). In this case forking would be much easier and neater but I am somewhat unwilling to go that way.

This is where I am at the moment. TLDR: cannot inject custom LineBuffer() as a dependency and fork seems to be the only (highly undesirable) way. Any advice?

@vladisld thank you very much! This is exactly the pointer I was looking for!

@vladisld

Solving the problem above made me wonder what is the "backtrader's way" of thinking about indicators/strategies (or any other object that holds lines) where the datapoint cannot be expressed as a float but rather a matrix (or any tensor)?

Dear all,

I am using an exponentially weighted correlation matrix and trying to figure out what is the best way to implement it as an indicator.

I am passing all datas to the indicator and on every bar I can compute the matrix. Now, what is the best way to store it? "Hacky" way is to just have it as a prop and update on every bar. However, this will not work for preload because it must be stored on the lines. Otherwise only the matrix will be available during the backtest. And lines by design only store floats. Not sure what is the best course of action here.