Save cerebro data to file for later plotting it on another computer
Here is how I work in general
I am working on a strategy which works on a bigger set of indicators and price action / swing trading setups. My approach is, to integrate several trading ideas into one strategy.
Said that, there are particular branches for each idea. E.g.:
masterbranch holds stable status of strategy
feature_xbranch holds one newly generated stop-rule
feature_ybranch holds a new filter for ranging markets
After developing a new stop-rule / entry / pattern etc I need to backtest it to see if it works for a lot of market data. The normal process here is:
- Backtest it on about 140 symbols for 10-17 years
- Find bugs / Find market conditions in backtest which I did not concider in before / ...
- Adjust it
- If not satisfied: Repeat at 1. If satisfied: Merge it into
So the step of backtesting a huge dataset is done very often. Therefore my interest in having the backtest process done quickly, automated and away from my laptop is very high.
Due to some resource-consuming custom indicator (Support/Resistance) one backtest runs approx. 90 seconds per symbol. I outsourced the backtesting to a powerful linux server which runs 12 backtests in parallel.
So the problem right now is, that my log entries tell me, at which symbols on which dates the current feature have an impact on (e.g. Your new stop rule triggered on 2008-01-19 on EURUSD=X). But I need a chart to see if the program executes it in a situation which I created it for. So i started to use
cerebro.plot(style='candlestick', savefig=save_fig, dpi=900, width=32, height=18, numfigs=5)) which has been nicely provided by @xnox, but zooming into it looks like this (even with the provided settings):
So my idea is to save all the
cerebrodata to file (e.g. by a linux server) and load the plotter later (on a normal computer) with the saved data. Do you think this is a reasonable way?
cerebroinstance can be pickled. This is something one can assume as a fact or else it wouldn't be compatible with
multiprocessing(there are obscure corners in the pickling methodology, which means that something could still fail)
With that assumption in mind you could pickle the already executed
cerebroinstance, take it to a new computer and then execute the plotting. Saving to a file or sending it over a socket once the instance has been pickled shouldn't make a difference
Thanks for the reply. Just wrote a test:
try: pickle.dump(cerebro, open(pickle_file_path, "wb")) logger.debug("Successfully pickled cerebro instance to file (%s)!" %pickle_file_path) except: logger.exception("Failed to pickle cerebro instance to file (%s)" %pickle_file_path)
but it fails with
File "/Users/g/PycharmProjects/g1trd/master.py", line 236, in runstrat pickle.dump(cerebro, open(pickle_file_path, "wb")) TypeError: can't pickle _thread.RLock objects
That's a surprising error message, because backtrader doesn't (at least directly) instantiate an
_thread.RLockobject. Some things that could (behind the scenes):
threading.Lockobjects, but these are only created for the live broker/data feeds
queue.Queueobjects, but these are also only created for live broker/data feeds