Interpreting optimization results when using multiple strategies

vbs

Hello gentlemen,

I would like to integrate Bokeh plotting for optimization results but I have a problem interpreting optimization results when using multiple strategies.

Let's say I have this test code:

cerebro.optstrategy(TestStrategy, buydate=range(14, 16, 1))
cerebro.optstrategy(TestStrategy2, buydate=range(11, 13, 1))
cerebro.addanalyzer(bt.analyzers.TradeAnalyzer)
data = ...
cerebro.adddata(data)
res = cerebro.run()

Now I have a list of 4 entries in the variable res (all possible strategy combinations), which is fine for me. Each of these entities contain a list 2 OptReturn objects. One for each strategy involved.
But how can I know which of these OptReturn objects belongs to which of the two strategies? I mean: I have 2 OptReturn objects and I have 2 strategies. They only contain the set of optimization parameters and a list of analyzers.

0_1517560361811_4a3aea20-4080-412a-a12f-41a10170cc63-image.png

Thank you!

backtrader

They are returned in the same order in which they were inserted. You can also deactivate the return of the OptReturn objects and get the strategies by setting optreturn=False

Docs - Cerebro

vbs

@backtrader
Thanks for quick answer.

So I would need to have access to strategy list which means I need a reference to the cerebro object to be able to process the OptReturn objects programmatically, right?

As far as I can see the plotting interface (which is called when trying to plot the OptReturn) does not provide a reference to the the cerebro object in the call:
https://github.com/backtrader/backtrader/blob/development/backtrader/plot/plot.py#L116

So my idea would be to pass a reference to the cerebro object (or maybe just to the strategy list) in the constructor call to my Bokeh plotting object (only needed when planning to plot optimization results with optreturn=True). Then when plotting OptReturns the cerebro can be used to find the corresponding Strategy object and add, for example, the name of the strategy.

Does this sound reasonable or are there maybe better approaches to get the corresponding strategies?

EDIT:
Hm, another problem:
The plot function is called for every OptReturn object seperately, so I don't know the current index in the list which I need to access the Strategy list... hm...

backtrader

@vbs said in Interpreting optimization results when using multiple strategies:

So I would need to have access to strategy list

No. Your best bet is to use optreturn=False and get Strategy instances returned instead of OptReturn objects.

@vbs said in Interpreting optimization results when using multiple strategies:

The plot function is called for every OptReturn object seperately, so I don't know the current index in the list which I need to access the Strategy list... hm..

This because if you are optimizing you can end up with hundreds or thousands of Strategy instances. Plotting was deemed to be meaningless. Hence the optimization in the returned values.

vbs

I agree that plotting is not useful when having a certain amount of outputs. I just thought it would make sense to implement it since cerebro actually is trying to plot optimization results. However the integrated plotter seems to yield an exception when trying to plot opt results.

backtrader

End users are (sometimes) also responsible for their actions, i.e.: trying to plot an optimization run.

vbs

Sorry for coming back to this but I have a problem with this again:
I was able to get around my previous problem by getting the index of the current optresult and to use that index to access cerebro.runningstrats like this:

    def _get_strategy(self, opt_result):
        """Finds and returns the strategy the opt result is based upon"""
        optidx = self._find_optreturn_idx(opt_result)
        return self._cerebro.runningstrats[optidx]

This worked fine so far but it does not work anymore when using maxcpu with a value greater than 1. In that case the cerebro object does not contain the field runningstrats. I guess this field is only created in the subprocesses but is not available then in the parent process.

So currently I am not able to find the corresponding strategy (or strategy class) that belongs to an optresult object. Is the strategy object available somewhere else?
Maybe it would be possible to change backtrader so that a reference to the corresponding strategy is added to every optresult?
I am aware that full data is returned when using optresult=False. But I am afraid this would need alot of RAM when dealing with a huge number of results.

backtrader

@vbs said in Interpreting optimization results when using multiple strategies:

I guess this field is only created in the subprocesses but is not available then in the parent process.

Because the parent process is only responsible for distributing the work to subprocesses.

@vbs said in Interpreting optimization results when using multiple strategies:

Maybe it would be possible to change backtrader so that a reference to the corresponding strategy is added to every optresult?

If you add a reference to the Strategy you automatically pull in the data feeds, indicators and associated Observers, because they are referenced in the Strategy, in which case you have artificially recreated optreturn=False

vbs

If you add a reference to the Strategy you automatically pull in the data feeds, indicators and associated Observers, because they are referenced in the Strategy, in which case you have artificially recreated optreturn=False

How about not adding the strategy itself but a reference to the corresponding strategy class? That would help alot and should not pull in too much data?

vbs

As a proposal, it could be as simple as this?

cerebro.py, line 1314

        if self._dooptimize and self.p.optreturn:
            # Results can be optimized
            results = list()
            for strat in runstrats:
                for a in strat.analyzers:
                    a.strategy = None
                    a._parent = None
                    for attrname in dir(a):
                        if attrname.startswith('data'):
                            setattr(a, attrname, None)

                oreturn = OptReturn(strat.params, analyzers=strat.analyzers, strategycls=type(strat))
                results.append(oreturn)

Only added , strategycls=type(strat)

backtrader

I see where you want to go, but you can have the same Strategy class in the same optimization mix several times with different parameters. You could argue that you won't, but the solution has to be generic.

Solving the conumdrum should not add even more obscurity to the mix.

vbs

@backtrader
I totally agree that a non-generic solution that introduces limiting assumptions is bad and should not be added.

But I have a problem to understand your concerns in this case. The code iterates over runstrats, so in every iteration strat contains exactly one strategy object, right?

Then this line:

oreturn = OptReturn(strat.params, analyzers=strat.analyzers, strategycls=type(strat))

Builds an OptReturn object and captures the current strategy's parameters and it's analyzers. Now in addition it also captures the class of the current strategy.
So in the end, in the variable results there will be a list of OptReturn objects and every one of them carries the corresponding strategy class.
I mean it also carries a parameter object taken from a strategy object so the strategy class it belongs to has to be distinct anyway, no?

I hope I don't suffer from a bigger misconeption on my side here... :/

backtrader

The thread has probably gone too long and you seem to want something different now that you wanted at the beginning. Let's try to recapitatulate:

You wanted a reference to the Strategy which had been run.

But Python doesn't give references ... it gives you the entire object so to say, which would pull-in all other referenced objects like data feeds (which is what one tries to avoid with OptReturn instances
Now you want a reference to the Strategy class

But this would still not give you the actual instance which was run, even if OptReturn instances contain the Strategy class.

And you may have several OptReturn instances with the same Strategy class, which is by itself the problem when trying to recognize which strategy was run.

I may be missing something about your intent. Just let me know.

vbs

Ok you are right. I will try to re-explain what I have in mind and I will try to bring it more on-point.

The background this:
I am thinking about (and also coding) helper functions that are able to process cerebro results in different ways. Use cases are displaying, plotting, archiving, filtering etc.

One possible type of result you can get out of cerebro is when you run an optimization using optreturn=True. So it would be desirable (imo) to be able to process this type of result programmatically in a generic way. This means you have to be able to get all relevant information out of the result itself (or at least in conjunction with the used cerebro). In my opinion this is currently not the case.

To be sure we are on the same page I prepared a minimal example:

class GreatStrategy(bt.Strategy):
    params = (
        ('buydate', 2),
    )

    def next(self):
        pass

class BestStrategy(bt.Strategy):
    params = (
        ('period', 4),
    )

    def next(self):
        pass


if __name__ == '__main__':
    cerebro = bt.Cerebro(maxcpus=2, optreturn=True)

    cerebro.optstrategy(GreatStrategy, buydate=range(2, 4, 1))
    cerebro.optstrategy(BestStrategy, period=range(4, 6, 1))

    data = bt.feeds.YahooFinanceCSVData(dataname="datas/orcl-1995-2014.txt",
                                        fromdate=datetime.datetime(2000, 1, 1),
                                        todate=datetime.datetime(2001, 2, 28))
    cerebro.adddata(data)

    cerebro.addanalyzer(bt.analyzers.TradeAnalyzer)

    res = cerebro.run()

So there are 2 strategies involved. Both target of optimization. They have two different sets of parameters. Both only containing a single parameter.

Running this gives you list of 4 results in res. The first looks like this (removed field p cause duplication of params):

0 = {list} <class 'list'>: [<backtrader.cerebro.OptReturn object at 0x00000000058235F8>, <backtrader.cerebro.OptReturn object at 0x0000000005823EB8>]
 0 = {OptReturn} <backtrader.cerebro.OptReturn object at 0x00000000058235F8>
  analyzers = {ItemCollection} <backtrader.metabase.ItemCollection object at 0x0000000005823908>
  params = {AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_GreatStrategy} <backtrader.metabase.AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_GreatStrategy object at 0x00000000058235C0>
 1 = {OptReturn} <backtrader.cerebro.OptReturn object at 0x0000000005823EB8>
  analyzers = {ItemCollection} <backtrader.metabase.ItemCollection object at 0x0000000005823F60>
  params = {AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_BestStrategy} <backtrader.metabase.AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_BestStrategy object at 0x0000000005823EF0>

In my understanding one OptReturn object is the result of one distinct run with one distinct strategy object involved. The contained field params describes the "configuration" of the used strategy.
But my problem is that you only have the params object (e.g. the configuration) but you do not know to which strategy class this parameters belong to. So how to know if res[0][0] is the result of GreatStrategy or of BestStrategy?

In other words: a strategy instance can be described by a combination two things:

A strategy class
A parameter object describing the "configuration" of the strategy

But currently the OptReturn object only carries the parameter part.

I had a workaround that involved grabbing the strategy class from the used cerebro. But that does not work when using maxcpu>1 because then the cerebro does not has the field runningstrats.

So my proposal is to add a field strategycls to OptReturn that carries the used strategy class that belongs to that OptReturn object. In my understanding there is a 1:1 relation between an OptReturn object and a strategy class.

vbs

I am using this approach since a while now and I have to say that I am quite happy with it:
https://github.com/verybadsoldier/backtrader/commit/f03a0ed115338ed8f074a942f6520b31c630bcfb

Makes it possible for me to evaluate OptReturn objects without the need of any further input (like a cerebro object).