For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Backtrader Multi-processing Issue.



  • Hi BT community,
    I am a huge fan of BT and using the platform from last 6 months.

    Recently I tried strategy optimisation using Muti-processors and It worked fine for smaller range of backtest.
    It works fine for around 100 counts but after it stops working.

    Starting Backtest
    Starting optimisation
    Killed
    (base) aadhunik@aadhunik:~/Desktop$ Process ForkPoolWorker-3:
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 127, in worker
        put((job, i, result))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 397, in _send_bytes
        self._send(header)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
        self._target(*self._args, **self._kwargs)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 132, in worker
        put((job, i, (False, wrapped)))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
        self._send(header + buf)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    Process ForkPoolWorker-2:
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 127, in worker
        put((job, i, result))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 397, in _send_bytes
        self._send(header)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
        self._target(*self._args, **self._kwargs)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 132, in worker
        put((job, i, (False, wrapped)))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
        self._send(header + buf)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    Process ForkPoolWorker-1:
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 127, in worker
        put((job, i, result))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 397, in _send_bytes
        self._send(header)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
        self._target(*self._args, **self._kwargs)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/pool.py", line 132, in worker
        put((job, i, (False, wrapped)))
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/queues.py", line 364, in put
        self._writer.send_bytes(obj)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
        self._send(header + buf)
      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    
    
    

    I have a 12 core I7 processor and Initially I though it can be resolved by lowering the cpu counts but it don't work even for

    cerebro = Cerebro(optreturn=False, maxcpus=2)
    cerebro.optstrategy(
        
            testStrategy,
             fast=8,
             slow=range(9, 21),
             dcperiod=range(10, 31),
            trperiod=12,
             volumep=range(5, 20),
        
         )
    

  • administrators

    @Dastaan-Sharma said in Backtrader Multi-processing Issue.:

    cerebro = Cerebro(optreturn=False, maxcpus=2)
    

    As long as you use more than 1 CPU it is obvious that the problem is going to show up.

    @Dastaan-Sharma said in Backtrader Multi-processing Issue.:

      File "/home/aadhunik/anaconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    

    No need to quote the entire chain of exceptions. All messages in the error trace point at the internals Python, in multiprocessing and specifically to pipe communication.

    This isn't even a pickle problem in which some of the Python objects cannot be conveyed over the pipes communicating processes. Here the pipe is broken and that usually means (not only in Python, in all multi-process environments)

    • THE PROCESS HAS DIED

    We are not in your code, but you can choose from these reasons (many other do exist)

    • Run out of memory ... and the process died
    • A non-recoverable exception happened ... and the process died
    • A non-handled exception ... and the process died
    • ...

    The problem is for sure not related to the number of CPUs.



  • @backtrader
    Thank you for helping me out in this.
    Actual problem is memory usage as you said.
    The memory usage keep rising while multiprocessing.
    Although this is not an issue in Backtrader, I would like to know your suggestion on how to deal with this.

    From my point of view there could be 2 solutions for these,
    1st ) Increasing Swap
    2nd ) Re-run the optimisation program every time after n number of simulations(200 in this case).


  • administrators

    You probably want to run less simulations each time.



  • @backtrader
    Thanks for your help but I found out that even for 1 cpu the memory usage keeps increasing.
    I think it keeps adding data in each and every step of simulation.


Log in to reply
 

});