Is it possible to use multiprocessing during backtesting (not optimization)?

csibi.levente14

This backtrader article from 2015 explains the process of using multiprocessing very well but it only discusses its use for strategy optimization. I was wondering if there is support (or if it is even feasible in any way) to apply multiprocessing when running a single trading strategy across multiple assets.

The basic structure of my code is:

cerebro = bt.Cerebro(maxcpus = None)
cerebro.addstrategy(TestStrategy)
for csv_path in csv_file_list:
        if os.path.isfile(csv_path):
            data = GenericCSVData(dataname=csv_path)
            cerebro.adddata(data)

cerebro.run()

Changing the maxcpus parameter's value didn't cause any change in execution speed.

vladisld

@csibi-levente14 AFAIU, currently the multiprocessing module is used only for optimization.

Unfortunately even in multiple assets scenario, where one would expect each assets historical date could be pre-loaded simultaneously - it is still done serially:

https://github.com/mementum/backtrader/blob/0fa63ef4a35dc53cc7320813f8b15480c8f85517/backtrader/cerebro.py#L1166

csibi.levente14

@vladisld thanks a ton, that's definitely useful to know. Then I'll focus on other aspects of the code and will leave multiprocessing aside for the time being :)

run-out

@csibi-levente14 You can use multi processing but I believe you must run a full backtest for each run. @vladisld I believe is correct in saying you would have to load the data each time. However, if doing a lot of test, this still saves much time.