Rolling correlation matrix indicator
I am using an exponentially weighted correlation matrix and trying to figure out what is the best way to implement it as an indicator.
I am passing all
datasto the indicator and on every bar I can compute the matrix. Now, what is the best way to store it? "Hacky" way is to just have it as a prop and update on every bar. However, this will not work for
preloadbecause it must be stored on the lines. Otherwise only the matrix will be available during the backtest. And lines by design only store floats. Not sure what is the best course of action here.
Solving the problem above made me wonder what is the "backtrader's way" of thinking about indicators/strategies (or any other object that holds lines) where the datapoint cannot be expressed as a float but rather a matrix (or any tensor)?
vladisld last edited by vladisld
lines by design only store floats
Not all lines store floats, some store other types like datetime for example:
class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.datas.datetime.date(0) print('%s, %s' % (dt.isoformat(), txt))
Briefly looking at the source code of LineBuffer, I do not see where the assumption of the float value type is used. So just theoretically you may store values of any type inside the private lines of your indicator for example.
Please correct me if I'm wrong.
Using custom type for line values will probably require such custom type to support some builtin operators ( like
__lt__and so on ) if some would like to support line operations ( see line operators docs here )
@vladisld thank you very much! This is exactly the pointer I was looking for!
@vladisld Hey Vlad, thanks again for your help! I've spent some time reading the above (and associated) and I think that it is impossible to achieve what I want without forking. Just thought I would share my findings, let me know if you agree or not.
The reason why lines are only accepting floats is the fact that the
LineBufferis an instance of
array.arrayset to hold
double. So if we create a flag on
LineBuffer, something like
is_generic = True, then inside
resetwe can set
def reset(self): if self.mode == self.QBuffer: self.array = collections.deque(maxlen=self.maxlen + self.extrasize) self.useislice = True elif self.is_generic: self.array = list() self.useislice = False else: self.array = array.array(str('d')) self.useislice = False self.lencount = 0 self.idx = -1 self.extension = 0
The above works. However, we need to inform
LineBufferwhich lines are supposed to be
generic. And here starts the painful part. The buffer is instantiated inside
Linesare created inside
MetaLineSeries. The best place to add the information about lines being "generic" is (in my opinion) in the
initmethod of the
for line, linealias in enumerate(self._getlines()): kwargs = dict() self.lines.append(LineBuffer(**kwargs))
can be changed to
for line, linealias in enumerate(self._getlines()): kwargs = dict() if linealias in genric_lines: kwargs['is_generic'] = True self.lines.append(LineBuffer(**kwargs))
However, there is no way (or at least I have not found one) to inject that custom
MetaLineSeries(it is set in the
lines = getattr(cls, 'lines', Lines)
So it means that one will need to re-build the whole structure, from
bt.DataBase(which the data feed inherits from) to
LineSeriesto change the pointer inside
MetaLinesSeriesfrom backtrader's original
Linesto the modified ones.
This is doable but it creates a new issue: datafeeds that inherit from the custom
LineSeriesdo not pass the equality inside the
def LineSeriesMaker(arg, slave=False): if isinstance(arg, LineSeries): return arg return LineSeriesStub(arg, slave=slave)
argis obviously not
LineSeriesMakeris actually called as a part of the
LineIteratoracross all conceptual parts of the framework. And changing
LineSeriesMakerwould require updating all those parts (basestrategy, baseindicator, baseobserver, etc.). In this case forking would be much easier and neater but I am somewhat unwilling to go that way.
This is where I am at the moment. TLDR: cannot inject custom
LineBuffer()as a dependency and fork seems to be the only (highly undesirable) way. Any advice?
@adarakchian you're right regarding the array.array configured for holding double type only. However this is true for the default case.
Did you try to set the buffer to QBuffer mode instead. It seems that in QBuffer state the array.array is not used.
I've not tried it myself yet - but it seems it worth to check this path.
@vladisld Just had a eureka moment - there is a much simpler solution that seems to work. In the
startmethod of the custom data feed that inherits from
DataBase, we can simply do:
for line_alias in private_lines: self._getline(line_alias).array = list()
Might be crude but does the job with minimal effort and no fork is needed :)
@adarakchian so in both your way and the way I've proposed - the main issue is to replace the array.array with other container.
Using the QBuffer mode seems to be less intrusive way, while in your case the container type is changed forcefully.
In both cases the target seems to be the same: the container type should not make an assumption about the value type used for line data.
One more note: there are some assumptions about the underlying container type inside the LineBuffer - and this is reflected in the LineBuffer.mode ( QBuffer or Unbounded).
Depending on this mode, different methods are used for slicing and other operations inside the LineBuffer methods
Given that, it is important to keep the container type compatible with what LineBuffer logic is expecting IMHO