Save cerebro data to file for later plotting it on another computer
cgi1 last edited by
Here is how I work in general
I am working on a strategy which works on a bigger set of indicators and price action / swing trading setups. My approach is, to integrate several trading ideas into one strategy.
Said that, there are particular branches for each idea. E.g.:
masterbranch holds stable status of strategy
feature_xbranch holds one newly generated stop-rule
feature_ybranch holds a new filter for ranging markets
After developing a new stop-rule / entry / pattern etc I need to backtest it to see if it works for a lot of market data. The normal process here is:
- Backtest it on about 140 symbols for 10-17 years
- Find bugs / Find market conditions in backtest which I did not concider in before / ...
- Adjust it
- If not satisfied: Repeat at 1. If satisfied: Merge it into
So the step of backtesting a huge dataset is done very often. Therefore my interest in having the backtest process done quickly, automated and away from my laptop is very high.
Due to some resource-consuming custom indicator (Support/Resistance) one backtest runs approx. 90 seconds per symbol. I outsourced the backtesting to a powerful linux server which runs 12 backtests in parallel.
So the problem right now is, that my log entries tell me, at which symbols on which dates the current feature have an impact on (e.g. Your new stop rule triggered on 2008-01-19 on EURUSD=X). But I need a chart to see if the program executes it in a situation which I created it for. So i started to use
cerebro.plot(style='candlestick', savefig=save_fig, dpi=900, width=32, height=18, numfigs=5)) which has been nicely provided by @xnox, but zooming into it looks like this (even with the provided settings):
So my idea is to save all the
cerebrodata to file (e.g. by a linux server) and load the plotter later (on a normal computer) with the saved data. Do you think this is a reasonable way?
cerebroinstance can be pickled. This is something one can assume as a fact or else it wouldn't be compatible with
multiprocessing(there are obscure corners in the pickling methodology, which means that something could still fail)
With that assumption in mind you could pickle the already executed
cerebroinstance, take it to a new computer and then execute the plotting. Saving to a file or sending it over a socket once the instance has been pickled shouldn't make a difference
cgi1 last edited by
Thanks for the reply. Just wrote a test:
try: pickle.dump(cerebro, open(pickle_file_path, "wb")) logger.debug("Successfully pickled cerebro instance to file (%s)!" %pickle_file_path) except: logger.exception("Failed to pickle cerebro instance to file (%s)" %pickle_file_path)
but it fails with
File "/Users/g/PycharmProjects/g1trd/master.py", line 236, in runstrat pickle.dump(cerebro, open(pickle_file_path, "wb")) TypeError: can't pickle _thread.RLock objects
That's a surprising error message, because backtrader doesn't (at least directly) instantiate an
_thread.RLockobject. Some things that could (behind the scenes):
threading.Lockobjects, but these are only created for the live broker/data feeds
queue.Queueobjects, but these are also only created for live broker/data feeds
Yelloww last edited by
Sorry to re-open this old thread.
I tried to dump and then load a cerebro object, then error occured when I plot this pickled cerebro.
Here is my code
with open('test.file','wb') as f: pickle.dump(cerebro,f,pickle.HIGHEST_PROTOCOL)
Then in another script:
with open('test.file','rb') as f: cere=pickle.load(f) #no problem with loading, but plotting cere.plot() /home/rr/anaconda2/lib/python2.7/site-packages/matplotlib/__init__.py:1350: UserWarning: This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time. warnings.warn(_use_error_msg) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/rr/anaconda2/lib/python2.7/site-packages/backtrader/cerebro.py", line 987, in plot for stratlist in self.runstrats: AttributeError: 'Cerebro' object has no attribute 'runstrats'
I think its a typical attribute error explained here:http://stefaanlippens.net/python-pickling-and-dealing-with-attributeerror-module-object-has-no-attribute-thing.html
But still I dont know how to fix it.
Thanks for help!