Smart optimizations and performance

leecallen

I am struggling with an issue that has been discussed here before, in threads "Genetic Optimization" and "Smart optimizations and Backtrader". I am optimizing multiple parameters.

Optimizing a single parameters via cerebro.optstrategy runs pretty fast due to multiprocessing & parallelization (thank you Backtrader). But each variable requires another set of passes through optstrategy, which must reload the entire data feed, a CSV file, and I think this takes a significant amount of time. I am looking for a way to reduce or eliminate that time.

I understand I could simply pass optstrategy the entire 'grid' -- all permutations of all my variables -- but that would be huge. I am running an intelligent optimization that does far fewer passes.

Is there some pseudo magical way to avoid reloading my data feed each time?

Barring that, is there a data feed format that is considerably faster to load than CSV? I am sure I could convert my CSV data to another format.

Thanks.

backtrader

You probably want to elaborate a bit more, because Cerebro does only load the data feed once and only once before optimizing, unless you explicitly disable preload or the data feed reports itself as live (which is not the case for a CSV), in which case it cannot obviously be preloaded.

leecallen

Sure.

In each call to optstrategy my program passes a list of values for a single variable. If there are 10 values, cerebro loads the data feed only 1 time.

But I am optimizing many variables, so I must call optstrategy many times, and each time it must load the data feed.

backtrader

You seem to imply that you actually run in a loop and in every loop you reinstantiate Cerebro (which is what has to be done).

In this case no. The new cerebro instance doesn't know that a previous one with data feeds existed. Nothing would in any case prevent that you create a data feed which uses a statically preloaded array (called it shared if you wish) as the source for the data.

That would achieve your intended effect.

leecallen

That would be ideal.

Can you give me any hints on how to accomplish that? I know how to read a CSV file into a data structure, but how can I tell Cerebro to use that as a data feed?

leecallen

Actually I think all the information I need is here:

https://www.backtrader.com/docu/datafeed-develop-general/datafeed-develop-general.html

I will work on that. If you can pass on any additional tips I would sure appreciate them.

leecallen

This worked out quite well, it reduced my execution time by 55%. Thank you.