For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Save cerebro data to file for later plotting it on another computer



  • Here is how I work in general

    I am working on a strategy which works on a bigger set of indicators and price action / swing trading setups. My approach is, to integrate several trading ideas into one strategy.

    Said that, there are particular branches for each idea. E.g.:

    • master branch holds stable status of strategy
    • feature_x branch holds one newly generated stop-rule
    • feature_y branch holds a new filter for ranging markets

    After developing a new stop-rule / entry / pattern etc I need to backtest it to see if it works for a lot of market data. The normal process here is:

    1. Checkout feature_x
    2. Backtest it on about 140 symbols for 10-17 years
    3. Find bugs / Find market conditions in backtest which I did not concider in before / ...
    4. Adjust it
    5. If not satisfied: Repeat at 1. If satisfied: Merge it into master branch

    So the step of backtesting a huge dataset is done very often. Therefore my interest in having the backtest process done quickly, automated and away from my laptop is very high.

    Due to some resource-consuming custom indicator (Support/Resistance) one backtest runs approx. 90 seconds per symbol. I outsourced the backtesting to a powerful linux server which runs 12 backtests in parallel.

    Problem

    So the problem right now is, that my log entries tell me, at which symbols on which dates the current feature have an impact on (e.g. Your new stop rule triggered on 2008-01-19 on EURUSD=X). But I need a chart to see if the program executes it in a situation which I created it for. So i started to use savefig option (cerebro.plot(style='candlestick', savefig=save_fig, dpi=900, width=32, height=18, numfigs=5)) which has been nicely provided by @xnox, but zooming into it looks like this (even with the provided settings):

    0_1499418575958_f4c26c51-43f4-44fb-b544-9c998699c349-grafik.png

    Proposal

    So my idea is to save all the cerebro data to file (e.g. by a linux server) and load the plotter later (on a normal computer) with the saved data. Do you think this is a reasonable way?


  • administrators

    A cerebro instance can be pickled. This is something one can assume as a fact or else it wouldn't be compatible with multiprocessing (there are obscure corners in the pickling methodology, which means that something could still fail)

    With that assumption in mind you could pickle the already executed cerebro instance, take it to a new computer and then execute the plotting. Saving to a file or sending it over a socket once the instance has been pickled shouldn't make a difference



  • Thanks for the reply. Just wrote a test:

            try:
                pickle.dump(cerebro, open(pickle_file_path, "wb"))
                logger.debug("Successfully pickled cerebro instance to file (%s)!" %pickle_file_path)
            except:
                logger.exception("Failed to pickle cerebro instance to file (%s)" %pickle_file_path)
    

    but it fails with

    File "/Users/g/PycharmProjects/g1trd/master.py", line 236, in runstrat
        pickle.dump(cerebro, open(pickle_file_path, "wb"))
    TypeError: can't pickle _thread.RLock objects
    

  • administrators

    That's a surprising error message, because backtrader doesn't (at least directly) instantiate an _thread.RLock object. Some things that could (behind the scenes):

    • threading.Lock objects, but these are only created for the live broker/data feeds
    • queue.Queue objects, but these are also only created for live broker/data feeds


  • @backtrader

    Sorry to re-open this old thread.

    I tried to dump and then load a cerebro object, then error occured when I plot this pickled cerebro.
    Here is my code

    with open('test.file','wb') as f:
       pickle.dump(cerebro,f,pickle.HIGHEST_PROTOCOL)
    

    Then in another script:

    with open('test.file','rb') as f:
       cere=pickle.load(f) 
    #no problem with loading, but plotting
    
    cere.plot()
    /home/rr/anaconda2/lib/python2.7/site-packages/matplotlib/__init__.py:1350: UserWarning:  This call to matplotlib.use() has no effect
    because the backend has already been chosen;
    matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
    or matplotlib.backends is imported for the first time.
    
      warnings.warn(_use_error_msg)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/rr/anaconda2/lib/python2.7/site-packages/backtrader/cerebro.py", line 987, in plot
        for stratlist in self.runstrats:
    AttributeError: 'Cerebro' object has no attribute 'runstrats'
    
    

    I think its a typical attribute error explained here:http://stefaanlippens.net/python-pickling-and-dealing-with-attributeerror-module-object-has-no-attribute-thing.html

    But still I dont know how to fix it.

    Thanks for help!



  • Edit 2: actually, error below is only when I add more than one data. For one data, it works fine. I can deserialize and plot no problem..

    Edit: never mind, there's a problem with this method.. getting a different error:

    AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_ba0f06d924224f95b2bf7797aa7ce535'

    Backtrader skips pickling the 'runstrats' attribute, as seen in cerebro.py here:

    def __getstate__(self):
        '''
        Used during optimization to prevent optimization result `runstrats`
        from being pickled to subprocesses
        '''
    
        rv = vars(self).copy()
        if 'runstrats' in rv:
            del(rv['runstrats'])
        return rv
    

    I extended cerebro and overriden the method, so it pickles everything:

    class MyCerebro(Cerebro):
        def __getstate__(self):
            return vars(self)
    

    Just use "MyCerebro()" instead of bt.cerebro() and pickle that.



  • @ovix said in Save cerebro data to file for later plotting it on another computer:

    Edit 2: actually, error below is only when I add more than one data. For one data, it works fine. I can deserialize and plot no problem..

    Edit: never mind, there's a problem with this method.. getting a different error:

    AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_ba0f06d924224f95b2bf7797aa7ce535'

    Backtrader skips pickling the 'runstrats' attribute, as seen in cerebro.py here:

    def __getstate__(self):
        '''
        Used during optimization to prevent optimization result `runstrats`
        from being pickled to subprocesses
        '''
    
        rv = vars(self).copy()
        if 'runstrats' in rv:
            del(rv['runstrats'])
        return rv
    

    I extended cerebro and overriden the method, so it pickles everything:

    class MyCerebro(Cerebro):
        def __getstate__(self):
            return vars(self)
    

    Just use "MyCerebro()" instead of bt.cerebro() and pickle that.

    Thanks for your work on this, I'm also having issues pickling Cerebro.

    With:

    cerebro = bt.Cerebro()
    

    ...

    try:
        # open a file
        pickle_file = open('pickled_cerebro_{}'.format(run_name), 'wb')
    
        # dump information to that file
        pickle.dump(cerebro, pickle_file)
    
        # close the file
        pickle_file.close()
    
    except Exception as e:
        print("Couldn't pickle cerebro", e)
    

    ...

    Couldn't pickle cerebro cannot serialize '_io.TextIOWrapper' object
    

    Could you explain what you mean by:

    Just use "MyCerebro()" instead of bt.cerebro() and pickle that.

    ?



  • @ovix said in Save cerebro data to file for later plotting it on another computer:

    class MyCerebro(Cerebro):
    def getstate(self):
    return vars(self)

    I replaced my instantiation of cerebro:

    cerebro = bt.Cerebro()
    

    with this, per your message:

        class MyCerebro(bt.Cerebro):
            def __getstate__(self):
                return vars(self)
    
        cerebro = MyCerebro()
    

    Now, my error message is:

    Couldn't pickle cerebro Can't pickle local object 'run_the_strategy.<locals>.MyCerebro'



  • @yelloww said in Save cerebro data to file for later plotting it on another computer:

    pickle.HIGHEST_PROTOCOL

    Update:

    I found this package, dill which successfully saves a pickle of cerebro.

    However, upon trying to unpickle or load, it fails :(

    import dill
    import pickle
    import backtrader
    
    # open a file, where you stored the pickled data
    pickled_file = open('pickled_cerebro_willy-v0.07', 'rb')
    
    # pull information into the world
    eat_the_pickle = dill.load(pickled_file)
    
    # close the file
    pickled_file.close()        
    

    error:

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-8-1c94c425ea19> in <module>()
          4 
          5 # pull information into the world
    ----> 6 eat_the_pickle = dill.load(pickled_file)
          7 
          8 # close the file
    
    /anaconda3/lib/python3.6/site-packages/dill-0.2.9.dev0-py3.6.egg/dill/_dill.py in load(file, ignore)
        302     # apply kwd settings
        303     pik._ignore = bool(ignore)
    --> 304     obj = pik.load()
        305     if type(obj).__module__ == getattr(_main_module, '__name__', '__main__'):
        306         if not ignore:
    
    /anaconda3/lib/python3.6/site-packages/dill-0.2.9.dev0-py3.6.egg/dill/_dill.py in find_class(self, module, name)
        463             return type(None) #XXX: special case: NoneType missing
        464         if module == 'dill.dill': module = 'dill._dill'
    --> 465         return StockUnpickler.find_class(self, module, name)
        466 
        467     def __init__(self, *args, **kwds):
    
    AttributeError: Can't get attribute 'AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_BigWillyStyle' on <module 'backtrader.metabase' from '/anaconda3/lib/python3.6/site-packages/backtrader/metabase.py'>