Out of memory error when running optstrategy

Dillonhao

New to the Backtrader. I got following error when running the optstrategy in linux server. all the memory and the swap drained then error.

Traceback (most recent call last):
  File "/home/dillonhao/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/dillonhao/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 412, in _handle_workers
    pool._maintain_pool()
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 248, in _maintain_pool
    self._repopulate_pool()
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory```

I dig a little bit with tracemalloc and looks like the below multiprocessing consume 50% of the memory.
```#1: /home/dillonhao/anaconda3/lib/python3.7/multiprocessing/connection.py:251: 
    return _ForkingPickler.loads(buf.getbuffer())

double check my bt code and have no clue of how to fix it without modify the bt code. Anyone can help? Thanks in advance

···
import sys
sys.path.append(r'/home/dillonhao/quant/lib/')
from DataPiple.DataCheck import checkMissingDate
import pickle
import logging
import backtrader as bt
from backtrader.analyzers import (SQN, AnnualReturn, TimeReturn, SharpeRatio_A, SharpeRatio,
TradeAnalyzer, drawdown)

startcash = 1000000

logging.basicConfig(level=loglevel)

cerebro = bt.Cerebro()
cerebro.broker.setcash(startcash)

cerebro.addstrategy(WLXStrategy)

cerebro.optstrategy(
WLXStrategy,
lmd=np.arange(0.8, 1, 0.01),
avg = np.arange(1,4),
dis = np.arange(0,0.02,0.002),

filename = timeline

)
file_location = r'/home/dillonhao/data/history_candle_data'
file_name = r'BTC-USDT_5m.csv'
btData = getLocalData(file_location, file_name)
timedic = timelist = getTimeSlot('2019-06-01', 60, 30, 4)[0]
data = bt.feeds.PandasData(dataname=btData, fromdate=timedic['stime'], todate=timedic['etime'],
timeframe=bt.TimeFrame.Minutes)
cerebro.adddata(data)
cerebro.addsizer(bt.sizers.PercentSizer, percents=100)
cerebro.broker.setcommission(commission=0.0005)
cerebro.addanalyzer(SharpeRatio, timeframe=bt.TimeFrame.Minutes, compression=30)
cerebro.addanalyzer(SQN)
cerebro.addanalyzer(AnnualReturn)
cerebro.addanalyzer(TradeAnalyzer)
result = cerebro.run(maxcpus=24)
···

run-out

I may be mistaken but my understanding is that the challenge with using the optimizer in backtrader is that it accumulates the results to return in a list. The more items being accumulated means more memory build up until released at the end.

Have a look here at @backtrader comments to try and solve this problem.

You can also consider breaking up your optimization into smaller runs.

Personally I found joy in not using the built in optimizer but using mutliprocessing outside of backtrader and calling each backtest individually as a function. My saving each result to a spreadsheet or postgres table I avoid the built up using memory altogether.

pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1)
            pool.map(st.backtest_controller_multi, scenarios)
            pool.close()
```
where scenarios is a list of dictionaries with all of the possible combinations of parameters.

vladisld

@Dillonhao said in Out of memory error when running optstrategy:

cerebro.addstrategy(WLXStrategy)
cerebro.optstrategy(

Probably a small note: no need to both addstrategy and optstrategy for strategy optimization - optstrategy is enough - otherwise, you will have two strategies that will be running at the same time.

Dillonhao

@run-out Thanks for your help. will give it a try.

Dillonhao

@vladisld Sorry. the "cerebro.addstrategy(WLXStrategy)" was commented out in my code. There is a mismatch in the ``` and the markup translated it to a format. my bad. But thank you for the reply.

vladisld

I'm usually using the following setup of Cerebro engine for the optimization:

cerebro = bt.Cerebro(maxcpus=args.maxcpus,
		                 live=False,
		                 runonce=True,
		                 exactbars=False,
		                 optdatas=True,
		                 optreturn=True,
		                 stdstats=False,
		                 quicknotify=True)

This setting will remove some standard analyzers which automatically get added to the Cerebro engine otherwise. It also optimizes the memory used for reporting the results (instead of returning the strategy instances - only the parameters and analyzers are returned).

Also, you may find it interesting to take a look at my recent post regarding the high memory consumption during strategy optimization using InfluxDB data feed - although it is not directly applicable to your case there could be some clues there as well:

https://community.backtrader.com/topic/2397/high-memory-consumption-while-optimizing-using-influxdb-data-feed

Dillonhao

update。
looks like the sharp ratio analyzer is the cause. I cancel out the following code and fixed the problem.

self.addanalyzer(SharpeRatio, timeframe=bt.TimeFrame.Minutes, compression=30)

chewbacca

Anyone figured out why bt goes OOM while optstrategy is running a strategy with a single analyzer?

littleking

@run-out This is a good way to sovle this problem, I am now learning how to use mutiprocessing package, can you show me a full example, Thank U.
And hope @backtrader can sovle this problem in later release.

littleking

@run-out thank u for your help