Multi core indicator processing

André Tavares

I am developing a strategy that tests cointegration for a list of 100 companies. I create an object of the indicator for each pair.
When the strategy is processing the method next for the indicator it processes each indicator at a time.
Is it possible to process them using all cores?

André Tavares

I seams the answer is no, as I have been looking at the code. How ever if I could multiprocess the lines below in lineiterator.py it would work.
Does anyone knows how I can multiprocess this to paralyze indicator analysis?

def _once(self):
            self.forward(size=self._clock.buflen())
            for indicator in self._lineiterators[LineIterator.IndType]:
               indicator._once()

@ab_trader I have seen you are active in this forum. Sorry to call you directly, but do you know how to solve this?

ab_trader

@André-Tavares I don't know

André Tavares

@backtrader any thoughts on how to speed up this part?

vladisld

I'm not sure the multiprocessor package is the way to go here because of possible data/indicators dependencies that need to be satisfied while calculating the indicator values (in case you are going to provide a fully generic solution).

Anyway, this could easily require relatively serious development and re-design (even if you are talking about completely independent indicator/data line calculations).

If you have some design in mind, please share - it could be an interesting proposal.

André Tavares

@vladisld I don't have it. I have been trying to change the code to allow multiprocessing but it did not work.

Another thing I could do to achieve my goal would be to instead of passing to the strategy a list of companies, I could tell it 2 companies and make an optimization of the strategy. This way I would achieve multiprocessing. However I would like to have an analysis of the whole strategy with all pairs together, and doing this I will have the results per pair.
Is it possible to merge Analyzers in order to see all pairs results together?