Save cerebro data to file for later plotting it on another computer



  • Here is how I work in general

    I am working on a strategy which works on a bigger set of indicators and price action / swing trading setups. My approach is, to integrate several trading ideas into one strategy.

    Said that, there are particular branches for each idea. E.g.:

    • master branch holds stable status of strategy
    • feature_x branch holds one newly generated stop-rule
    • feature_y branch holds a new filter for ranging markets

    After developing a new stop-rule / entry / pattern etc I need to backtest it to see if it works for a lot of market data. The normal process here is:

    1. Checkout feature_x
    2. Backtest it on about 140 symbols for 10-17 years
    3. Find bugs / Find market conditions in backtest which I did not concider in before / ...
    4. Adjust it
    5. If not satisfied: Repeat at 1. If satisfied: Merge it into master branch

    So the step of backtesting a huge dataset is done very often. Therefore my interest in having the backtest process done quickly, automated and away from my laptop is very high.

    Due to some resource-consuming custom indicator (Support/Resistance) one backtest runs approx. 90 seconds per symbol. I outsourced the backtesting to a powerful linux server which runs 12 backtests in parallel.

    Problem

    So the problem right now is, that my log entries tell me, at which symbols on which dates the current feature have an impact on (e.g. Your new stop rule triggered on 2008-01-19 on EURUSD=X). But I need a chart to see if the program executes it in a situation which I created it for. So i started to use savefig option (cerebro.plot(style='candlestick', savefig=save_fig, dpi=900, width=32, height=18, numfigs=5)) which has been nicely provided by @xnox, but zooming into it looks like this (even with the provided settings):

    0_1499418575958_f4c26c51-43f4-44fb-b544-9c998699c349-grafik.png

    Proposal

    So my idea is to save all the cerebro data to file (e.g. by a linux server) and load the plotter later (on a normal computer) with the saved data. Do you think this is a reasonable way?


  • administrators

    A cerebro instance can be pickled. This is something one can assume as a fact or else it wouldn't be compatible with multiprocessing (there are obscure corners in the pickling methodology, which means that something could still fail)

    With that assumption in mind you could pickle the already executed cerebro instance, take it to a new computer and then execute the plotting. Saving to a file or sending it over a socket once the instance has been pickled shouldn't make a difference



  • Thanks for the reply. Just wrote a test:

            try:
                pickle.dump(cerebro, open(pickle_file_path, "wb"))
                logger.debug("Successfully pickled cerebro instance to file (%s)!" %pickle_file_path)
            except:
                logger.exception("Failed to pickle cerebro instance to file (%s)" %pickle_file_path)
    

    but it fails with

    File "/Users/g/PycharmProjects/g1trd/master.py", line 236, in runstrat
        pickle.dump(cerebro, open(pickle_file_path, "wb"))
    TypeError: can't pickle _thread.RLock objects
    

  • administrators

    That's a surprising error message, because backtrader doesn't (at least directly) instantiate an _thread.RLock object. Some things that could (behind the scenes):

    • threading.Lock objects, but these are only created for the live broker/data feeds
    • queue.Queue objects, but these are also only created for live broker/data feeds

Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.