How to change asset dynamically?

Yacc Don

Hi, I'm trying to implement some strategy which might change asset from time to time.

As illustration above, I might want to select stock at some interval, the asset might change, or might not change like Asset2.

And as the recommendation in this thread:
https://community.backtrader.com/topic/1559/indicator-with-dynamically-changing-data
Quote from thread: "In any case I feel you are using the wrong approach imho. Instead of changing the data feeds, you should change what the data feed delivers, which at the end of the day delivers the same result, but is compatible with any way an indicator can be written."

So I extend PandasData as follow, to change data feed when the asset is changed.

class ReplaceablePandasData(bt.feeds.PandasData):
    def changeDf(self, df):
        self._idx = df.index.get_loc( self.p.dataname.iloc[self._idx]["date"], method="backfill")
        self.p.dataname = df.reset_index()

But there is a problem: the indicator like SMA, might caculate the result from data in LineBuffer array, which contain both old data and new data:

How could I fix this problem, or is there any other better way to achieve dynamical asset?

Thanks
@ab_trader @backtrader

vladisld

@yacc-don

AFAIU the full list of assets is known to the strategy ( defined in the df).

In this case why not just use standard data feeds for all known assets and only work with the "active" data feeds inside the strategy ( "active" means the one that the strategy is actually trading while ignoring others ).

The "active" set could be updated according to the logic of your algorithm ( being it date based or any other logic ).

This way there is no need to force the Backtrader components to work the way it wasn't designed to work in the first place.

Please correct me if I'm missing something.

Yacc Don

@vladisld Thank for your reply, I have some question about your solution, would you please clarify it for me? Thanks for you time.

If add all the asset (which could be 4000+) to the strategy, how to make other data feed inactive, and don't need to load them and process them? Otherwise it will be very bad for performance and memory.
Once stratgey start to run, if I want to change some asset, how to make the new selected asset active?
Could I initize the indicator with new data feed in the strategy.next() function, then start to use these indicator? for example:

class UserStockStrategy(bt.Strategy):
    def next(self):
          if time_is_right == True:
               new_asset_idx = self.pick_some_asset()
               self.sma = SMA(self.datas[new_asset_idx], period=5)

          if self.sma > self.datas[new_asset_idx].close:
               self.buy()

Thanks

vladisld

So we talking about runtime/resource optimizations right ? And not about hypothetical problem of dynamic data changes from the outside source.

Few question are in place then:

What is the maximal number of asserts your hardware allows to comfortably work with ?
Does the resources used increase linearly with the number of assets?
Have you already tried to optimize the memory/time taken by your strategy?
Will it be easier/cheaper to upgrade your resources instead of investing time (and money) to tweak the framework?
Does your strategy investigate the correlations between different assets or each asset could be independently traded/scanned?
Will it be possible to run your strategy multiple times on smaller set of assets serially without hurting the logic of your strategy (at expense of more time spent )

@yacc-don said in How to change asset dynamically?:

Otherwise it will be very bad for performance and memory.

It depends on your resources, timeframe/compression used and time you wish to spend optimizing (if we are talking about backtesting): https://medium.com/@danjrod/on-backtesting-performance-and-out-of-core-memory-execution-f3fdb0c9fdf8

Another question is whether you really would like to run the strategy blindly on 4000+ assets without any offline screening first ?

@yacc-don said in How to change asset dynamically?:

how to make the new selected asset active

I would just maintain the list/set of the active datas and add/remove to/from it when necessary - so in the next method (for example) only this list of assert will be traded.

@yacc-don said in How to change asset dynamically?:

Could I initize the indicator with new data feed in the strategy.next() function, then start to use these indicator?

I'm not sure it will work this way out of the box without violating some assumptions inside the Cerebro engine about proper sequence of calls to start,prenext,etc methods to be called as well as proper minperiod handling. However it is probably worth investigating in more details if one is willing to go this path.

run-out

@vladisld Normally I just upvote, but that's an awesome answer.

ethan

@yacc-don I actually have the exact same question as I'm trying out Backtrader. based on @vladisld answer, it doesn't look like something Backtrader supports and it's gonna be a pain to set this up if even feasible. It makes no sense to me to even have to think about runtime optimization. Why should you even grab an extra data feed if the asset is not in your portfolio? If data feed is not something that can be added to Cerebro in a runtime fashion, I believe it's not gonna work. And if my statement is true, other platforms like QuantConnect will serve your need with 0 code, and you just need to focus on the actual logic to select your asset. I like Backtrader, and I hope that I'm wrong on this so please enlighten me here.

vladisld

@ethan Just found the old post that was discussing exactly this issue of working with large 'universe'. Please take a look:

https://community.backtrader.com/topic/399/downselect-from-a-large-universe-trade-6-tickers/6?_=1612681210185

AFAIU from this post, there is no other way (at least in current design) in Backtrader to select the data feed dynamically if it wasn't added before starting the Cerebro engine.

Frankly I'm not very familiar with QuantConnect - so could be wrong here - the advantage of QuantConnect (which is also its disadvantage) is that your code is actually running in the QuantConnect cloud where all the data feeds are already available (loaded). This is not the case with Backtrader, where its engine is designed to run locally - so all the heavy lifting of loading the data feeds is left to the developer to handle (giving him more flexibility btw).

yacc2000

@vladisld Thank you very much. I forgot my password, so I have to register this account to come back online to express my thanks.

And I would like to try this path mentioned in your answer:
"I'm not sure it will work this way out of the box without violating some assumptions inside the Cerebro engine about proper sequence of calls to start,prenext,etc methods to be called as well as proper minperiod handling. However it is probably worth investigating in more details if one is willing to go this path."

Won't be a easy thing, but worth a try.