Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Out of memory error when running optstrategy

    General Code/Help
    5
    10
    1300
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      Dillonhao last edited by

      New to the Backtrader. I got following error when running the optstrategy in linux server. all the memory and the swap drained then error.

      Traceback (most recent call last):
        File "/home/dillonhao/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
          self.run()
        File "/home/dillonhao/anaconda3/lib/python3.7/threading.py", line 870, in run
          self._target(*self._args, **self._kwargs)
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 412, in _handle_workers
          pool._maintain_pool()
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 248, in _maintain_pool
          self._repopulate_pool()
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
          w.start()
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
          self._popen = self._Popen(self)
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
          return Popen(process_obj)
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
          self._launch(process_obj)
        File "/home/dillonhao/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
          self.pid = os.fork()
      OSError: [Errno 12] Cannot allocate memory```
      
      I dig a little bit with tracemalloc and looks like the below multiprocessing consume 50% of the memory.
      ```#1: /home/dillonhao/anaconda3/lib/python3.7/multiprocessing/connection.py:251: 
          return _ForkingPickler.loads(buf.getbuffer())
      

      double check my bt code and have no clue of how to fix it without modify the bt code. Anyone can help? Thanks in advance

      ···
      import sys
      sys.path.append(r'/home/dillonhao/quant/lib/')
      from DataPiple.DataCheck import checkMissingDate
      import pickle
      import logging
      import backtrader as bt
      from backtrader.analyzers import (SQN, AnnualReturn, TimeReturn, SharpeRatio_A, SharpeRatio,
      TradeAnalyzer, drawdown)

      startcash = 1000000

      logging.basicConfig(level=loglevel)

      cerebro = bt.Cerebro()
      cerebro.broker.setcash(startcash)

      cerebro.addstrategy(WLXStrategy)

      cerebro.optstrategy(
      WLXStrategy,
      lmd=np.arange(0.8, 1, 0.01),
      avg = np.arange(1,4),
      dis = np.arange(0,0.02,0.002),

      filename = timeline

      )
      file_location = r'/home/dillonhao/data/history_candle_data'
      file_name = r'BTC-USDT_5m.csv'
      btData = getLocalData(file_location, file_name)
      timedic = timelist = getTimeSlot('2019-06-01', 60, 30, 4)[0]
      data = bt.feeds.PandasData(dataname=btData, fromdate=timedic['stime'], todate=timedic['etime'],
      timeframe=bt.TimeFrame.Minutes)
      cerebro.adddata(data)
      cerebro.addsizer(bt.sizers.PercentSizer, percents=100)
      cerebro.broker.setcommission(commission=0.0005)
      cerebro.addanalyzer(SharpeRatio, timeframe=bt.TimeFrame.Minutes, compression=30)
      cerebro.addanalyzer(SQN)
      cerebro.addanalyzer(AnnualReturn)
      cerebro.addanalyzer(TradeAnalyzer)
      result = cerebro.run(maxcpus=24)
      ···

      1 Reply Last reply Reply Quote 0
      • run-out
        run-out last edited by

        I may be mistaken but my understanding is that the challenge with using the optimizer in backtrader is that it accumulates the results to return in a list. The more items being accumulated means more memory build up until released at the end.

        Have a look here at @backtrader comments to try and solve this problem.

        You can also consider breaking up your optimization into smaller runs.

        Personally I found joy in not using the built in optimizer but using mutliprocessing outside of backtrader and calling each backtest individually as a function. My saving each result to a spreadsheet or postgres table I avoid the built up using memory altogether.

        pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1)
                    pool.map(st.backtest_controller_multi, scenarios)
                    pool.close()
        ```
        where scenarios is a list of dictionaries with all of the possible combinations of parameters.

        RunBacktest.com

        D L 2 Replies Last reply Reply Quote 0
        • vladisld
          vladisld last edited by

          @Dillonhao said in Out of memory error when running optstrategy:

          cerebro.addstrategy(WLXStrategy)
          cerebro.optstrategy(

          Probably a small note: no need to both addstrategy and optstrategy for strategy optimization - optstrategy is enough - otherwise, you will have two strategies that will be running at the same time.

          D 1 Reply Last reply Reply Quote 1
          • D
            Dillonhao @run-out last edited by

            @run-out Thanks for your help. will give it a try.

            1 Reply Last reply Reply Quote 0
            • D
              Dillonhao @vladisld last edited by

              @vladisld Sorry. the "cerebro.addstrategy(WLXStrategy)" was commented out in my code. There is a mismatch in the ``` and the markup translated it to a format. my bad. But thank you for the reply.

              1 Reply Last reply Reply Quote 0
              • vladisld
                vladisld last edited by

                I'm usually using the following setup of Cerebro engine for the optimization:

                cerebro = bt.Cerebro(maxcpus=args.maxcpus,
                		                 live=False,
                		                 runonce=True,
                		                 exactbars=False,
                		                 optdatas=True,
                		                 optreturn=True,
                		                 stdstats=False,
                		                 quicknotify=True)
                

                This setting will remove some standard analyzers which automatically get added to the Cerebro engine otherwise. It also optimizes the memory used for reporting the results (instead of returning the strategy instances - only the parameters and analyzers are returned).

                Also, you may find it interesting to take a look at my recent post regarding the high memory consumption during strategy optimization using InfluxDB data feed - although it is not directly applicable to your case there could be some clues there as well:

                https://community.backtrader.com/topic/2397/high-memory-consumption-while-optimizing-using-influxdb-data-feed

                1 Reply Last reply Reply Quote 2
                • D
                  Dillonhao last edited by

                  update。
                  looks like the sharp ratio analyzer is the cause. I cancel out the following code and fixed the problem.

                  self.addanalyzer(SharpeRatio, timeframe=bt.TimeFrame.Minutes, compression=30)
                  
                  1 Reply Last reply Reply Quote 1
                  • C
                    chewbacca last edited by

                    Anyone figured out why bt goes OOM while optstrategy is running a strategy with a single analyzer?

                    1 Reply Last reply Reply Quote 0
                    • L
                      littleking @run-out last edited by

                      @run-out This is a good way to sovle this problem, I am now learning how to use mutiprocessing package, can you show me a full example, Thank U.
                      And hope @backtrader can sovle this problem in later release.

                      1 Reply Last reply Reply Quote 0
                      • L
                        littleking last edited by

                        @run-out thank u for your help

                        1 Reply Last reply Reply Quote 1
                        • 1 / 1
                        • First post
                          Last post
                        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors