Create a tqdm object using the total runcount from cerebro

        opt_count_total = cerebro.get_opt_runcount()
        pbar = tqdm(smoothing=0.05, desc='Optimization', total=opt_count_total)
        res = cerebro.run()

And use the optcallback to call update() on the tqdm object:

c.optcallback(optimization_step)

def optimization_step(strategy: bt.Strategy):
    pbar.update()

@tw00000 said in progress bar / ETA during optimization:

Just bumping this again, it seems strange that there's no simple way to find out:

1. Estimate of how long your backtest will take

My workaround was to add another variable to backtrader so you can query the total run count for the optimization:
https://github.com/verybadsoldier/backtrader/commit/0018b38c65a5cb25b17c42913ec0838b49feb57d

@tw00000
As far as I can see it works fine when using cerebro.optcallback() to manually trigger tqdm.update().

Huh? I think that this is what I mean too. Based on the parameter that is part of the optimization, on every optimization iteration a different strategy might be instantiated (depends on the value of the parameter). While that is a very interesting approach I fail to see how I could use that with my problem.

Thank you, as I understand it, that approach can handle multiple strategies by interpreting them as one optimization parameter. But frankly I did not understand how to use that or modify it so it could be used to let a single strategy work on all datas in the mix in a single run (and get an aggregated result over all datas).

I thought about using it to instantiate always the same strategy so that every instance could trade on different data but then they would still be individual runs and I would not get aggregated results. Maybe I just fail to see the relation, sorry :(

Thanks for your input. Yes it seems to be the best way at the moment.

Also tried with some sort of intermediate strategy that basically acts as a dispatcher that instantiates the underlying "real" strategies and feeds them accordingly.

Would be used as this:

        cerebro_train.optstrategy(build_multidata_strategy(DipExploiter),
                                  stop_loss=np.arange(0.02, 0.15, 0.20),
                                  profit_target=np.arange(0.03, 0.12, 0.18),
                                  dip_thresh=np.arange(-0.01, -0.5, -0.2),
                                  adx_trend_thresh=np.arange(30, 60, 50),
                                  adx_period=np.arange(12, 16, 10)
                                  )

build_multidata_strategy would construct a strategy class at runtime. It would also "fake" self.datas for the underlying strategies so each strategy would only see one data and could act upon self.data as usual. But I could not get this to work (indicators were not called). I guess the problem is related to the metaclasses that are used by the strategies. I guess constructing strategy objects within other strategies is not supported.

Let's say I wrote a simple strategy that uses indicators and is meant to trade on a single data (first data in the mix: self.data). So the normal use case is a mix that contains this one strategy and one data.
But now I want to let the strategy act upon a whole bunch of datas, let's say, 30. Important for me is that I can run the optimization in an efficient way also.

What would be the best way to do this?

Two ideas that I had myself but I did not like alot:

• Add the same strategy multiple times: Add a parameter named data to the strategy so I can pass an instance of the data I want each strategy instance to trade on. Then I can add the strategy once for every data I want it to trade using regular addstrategy calls (everytime with a different value for data).
  Downsides:
  1 - The strategy code has to be modified to support the external data parameter. I would prefer it if "vanilla" strategies could be used without the need to add special behavior.
  2 - when optimizing the parameter combinations will not be "aggregated" as each instance of the strategy will be treated independently. So the number of results/calculations will go through the roof. I would like that every optimization run uses the same parameter set for every strategy instance. So it gives the same number of result when trading on 1 data or on 10 data.

• Modify the strategy so that it acts/trades on every data in the mix. The strategy has only to be added to the mix once.
  Pros:
  Optimization will work well as we have only a single strategy in the mix and cerebro can run the optimization as usual
  Cons:
  The strategy will need to be heavily modified. I would like to avoid pulling the logic of handling multples datas into the strategy. It would be great if the strategy would only have to know about how to trade on a single data.

Well, I hope I was able to make myself clear. :)
So what would be the best and cleanest way to let a strategy act upon all datas in the mix in parallel and also have the optimization work like there was only a single data?

Ok, so it is intended behavior. I already assumed it would be related to the code path. I think it would be good if optcallback would be called regardless of the parameter maxcpus.
I tried simply adding it myself, but I am not sure if it will introduce other unexpected behaviors or side effects. So could it be as simple as this? Any reason to not have it?

        if not self._dooptimize or self.p.maxcpus == 1:
            # If no optimmization is wished ... or 1 core is to be used
            # let's skip process "spawning"
            for iterstrat in iterstrats:
                runstrat = self.runstrategies(iterstrat)
                self.runstrats.append(runstrat)
                if self._dooptimize:
                    for cb in self.optcbs:
                        cb(runstrat)  # callback receives finished strategy

(last 3 lines added)

Yes, I understand it disables multiprocessing when using maxcpus=1 but it is still executing optimization and gives me optimization results, no?

My point is that the optcallback is not called during the optimization process when maxcpus=1. It is called though when maxcpus is greater than 1.

@backtrader said in progress bar / ETA during optimization:

It seems you want to add Optimization Callbacks.

See: Docs - Cerebro and look for optcallback

For me this callback does not work when using maxcpu=1. But it works fine for example with maxcpu=8. Is that intended or a bug possibly?
"Does not work" means that it never gets called.

The documentation at least does not mention any limitations:
https://www.backtrader.com/docu/cerebro.html#backtrader.Cerebro.optcallback