For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Does Backtrader support Parallel Strategies?



  • I'm working on a design which involve working with many strategies. some are small and simple and other more complex. each strategy output its decision and on the end the 'master' strategy is making the final decision, which can operate on the market by placing orders or cancel them....

    Is there any limitation/reason not to run all/some strategies in parallel?


  • administrators

    @jacob said in Does Backtrader support Parallel Strategies?:

    Does Backtrader support Parallel Strategies?

    You can call addstrategy as many times as you want. And this has been discussed several times as is part of the described features

    @jacob said in Does Backtrader support Parallel Strategies?:

    Is there any limitation/reason not to run all/some strategies in parallel?

    What kind of limitation are you expecting?



  • let's say for example i have 100 strategies and some of them are calculated quickly and some take couple of minutes.
    all i'm asking is it possible to run them all together at the same time or would backtrader can only process them sequentially?


  • administrators

    Do you know what the GIL is?



  • Thanks @backtrader for pointing this out. I didn't know python had a problem with multithreading before.

    After reading quite a bit on the subject, I now understand python works best as a single thread. so now I also need to refine my question.

    A big part of the bottle neck of having a lot of strategies comes down to I/O requests from the broker, which can consume a lot of time.

    And the go to solution seems to be using asynchronous instead of multithreading.

    but would it work/make sense to run strategies async?
    may it require also changing the store?


  • administrators

    @jacob said in Does Backtrader support Parallel Strategies?:

    After reading quite a bit on the subject, I now understand python works best as a single thread

    Being new to the topic I fear you have only slightly grasped the situation. Multi-threading in Python has several scenarios in which mult-threading works best. Classical examples:

    • I/O Bound
    • GUI

    @jacob said in Does Backtrader support Parallel Strategies?:

    A big part of the bottle neck of having a lot of strategies comes down to I/O requests from the broke

    Sorry but this is wrong. Strategies don't make any live request to brokers that have to go the network. If this is the fall in some broker code you have, the design is simply wrong.

    @jacob said in Does Backtrader support Parallel Strategies?:

    And the go to solution seems to be using asynchronous instead of multithreading.

    You want to replace a model in which you can achieve concurrency with threaded I/O (which seems to be what troubles you with network requests) for a model in which you run in everything a single thread.

    Sorry but async was a very old black many years ago in for example C, long before Javascript and other things started using it. It wasn't even a language feature, because it made use of the OS (unix-like and even under windows) facilities.

    You probably have a background in Javascript and like many in that camp, you believe that async is the new black.

    Async was a design choice of Javascript to keep things simple and have a myriad of junior programmers under control by avoiding any need to avoid things like race conditions (which can happen when doing multi-threading programming and yes ... in Python too)

    But Async doesn't magically turn your single-thread in a parallel execution engine. Things are still run in a single thread and have single-thread performance (actually, the performance is slightly worse)

    @jacob said in Does Backtrader support Parallel Strategies?:

    but would it work/make sense to run strategies async?

    It doesn't make any sense. Read above.

    @jacob said in Does Backtrader support Parallel Strategies?:

    may it require also changing the store?

    As pointed out above: if the store you work with makes direct requests to the network as a result of an action initiated by the Strategy (hence ... blocking the main thread), the design is wrong.

    The stores included with backtrader all use threading and for a very good reason: they are all I/O bound, which should be the case in any network application.



  • @backtrader Thanks for your detailed reply.

    As pointed out above: if the store you work with makes direct requests to the network as a result of an action initiated by the Strategy (hence ... blocking the main thread), the design is wrong.

    My focus is crypto currency and I work with the community driven development that tries to bind between Backtrader and CCXT project (which play the broker roll).
    but as you mentioned broker and store doesn't include threading to keep all the data well updated, instead it use cache or direct network calls.

    which made me wonder how to run strategies in that setup.

    clearly for it to work as intended in Backtrader more work on both store and broker is needed.

    Ok, so just to see if I understand the concept, at the most simple level.
    we expect from the store to take control over updating some of the data independently of what is going on in backtrader using a thread for each thing we wanna keep updated. so for example, as the store get instantiated we will need to create a thread that keep account details up to date at all time. so when we call it from the strategy we get the latest values from the cache.

    You probably have a background in Javascript and like many in that camp, you believe that async is the new black.

    sorry to disappoint but no background in Javascript or async for that matter :)
    just trying to connect the dots with the information I read.


  • administrators

    @jacob said in Does Backtrader support Parallel Strategies?:

    clearly for it to work as intended in Backtrader more work on both store and broker is needed.

    Actually it is a general design practice. You shouldn't block the main thread of any running program. If you see the GUI of an application freezing, it is because things are being blocked in the main thread and the OS has no chance to keep the GUI updated.

    The design which backtrader has is also asynchronous (which is in many occasions what people take as event-based), because your code is called when it needs to be called

    • Order has been executed
    • next iteration
    • Others ...

    To keep things simple there are simplifications like:

    • notify_cash won't be called for each update, but only when your next iteration is going to be called

      The rationale is that you only care about your cash in the same timeframe that you are operating on.