Multiple Live Data Feeds
-
I read through Docs/Forum, but could not find a definitive answer/solution.
Is it possible to monitor multiple live data feeds in backtrader? Let's say I want to look for an entry based on an indicator (eg RSI) - I would like to parse live data feeds of 10s of instuments and choose the best 3.
I assume there is no easy workaround, so I would be grateful if someone could contribute which of the following solutions seem to be most reasonable:
-
running x algos in parallel with x single live data feeds - this becomes very impractical if you want to monitor many instruments. If I want to only choose 3 intruments out of 50 I monitor, this becomes very unefficient as well unless there is a shared DB (or sth similar) that signals once 3 assets have been already chosen (so other scripts can stop parsing looking for an entry). Also this would still probably require some flow of information between individual algos for proper risk management etc.
-
trying to utilize pararelization and shared/ditributed memory with multiprocessing module or similar - I dont know too much about this so I cant even tell if this has a potential to solve my problem. Thanks for your input.
-
-
@totoro said in Multiple Live Data Feeds:
Is it possible to monitor multiple live data feeds in backtrader?
@totoro said in Multiple Live Data Feeds:
I read through Docs/Forum, but could not find a definitive answer/solution.
The docs and features description clearly indicate that you can work with multiple data feeds. The platform doesn't make distinctions between live and non-live. The limitations usually arise from data providers. Interactive Brokers is a well known example due to the limitations it imposes on backfilling with pacing violations.
@totoro said in Multiple Live Data Feeds:
- trying to utilize pararelization and shared/ditributed memory with multiprocessing module or similar - I dont know too much about this so I cant even tell if this has a potential to solve my problem. Thanks for your input.
backtrader only uses
multiprocessingfor optimization. Nothing prevents you of course from delegating work to sub-processes which can run in the background whilst the engine is waiting for data and/or processing that data for indicator calculations, but all this will reside and run in a single process.@totoro said in Multiple Live Data Feeds:
- running x algos in parallel with x single live data feeds
You probably have to define what you mean by "running in parallel". Python has a
GILlimitation and things don't run in parallel. You can still run multiple strategies in backtrader. How they communicate with each other is up to you (simplequeue.queuedo actually suffice) -
@backtrader Thanks for a prompt response!
Well, I guess my confusion comes from the point that working with multiple data feeds in parallel would require multiprocessing or similar, right? Otherwise it is more like looping through data feeds and introduces lag/latency (or do you assume that the delay is essentialy negligible even if we have 100s of feeds?)
@backtrader said in Multiple Live Data Feeds:
running x algos in parallel with x single live data feeds
You probably have to define what you mean by "running in parallel". Python has a GIL limitation and things don't run in parallel. You can still run multiple strategies in backtrader. How they communicate with each other is up to you (simple queue.queue do actually suffice)
Here what I meant was to simply run each algo with single data feed independently (as a seprate program/script) and create some sort of communication channel - I would say this is a straight-forward "brute-force" approach. This way I could ensure that multiple data feeds are parsed simultaneously (altough in separate programs/scripts) without multiprocessing or similar.
-
In this Blog - Multi Example post, you can see multiple data feeds simultaneously.
You can also add as many strategies as needed and if you wish restrict each strategy to work on a single data feed (choose it by name, index ...)
-
@backtrader Thanks, I have read the post, however I still struggle to understand how this would work in my scenario - in the code above you seem to loop through data feeds (self.datas) one by one. If we deal with hundreds of live feeds where latency matters (ideally no delay between 1st and 100th data feed), this can spoil the results. Am I missing sth or you just assume that delay would be negligible? Thanks for your time again.
-
@totoro You would have to process each data feed separately and there is no workaround that in any language or using any framework. Even using pandas you have to filter the DataFrame according to the symbol being processed in this round.
If your strategy relies on microseconds precision then you are entering the realm of HFT and in that case, you have to evaluate if Python is really the correct tool of your choice.
-
@kausality Thanks, do you have any idea about the delay magnitude if we talk about hundreds of live data feeds?. And no, obviously I am not looking for microseconds precision, however 10s of seconds of delay might matter, thus my question. Thanks for your time.
-
@totoro It depends on what you are doing inside the loop. If you are making some calculations like std devs, atr etc then I don't think that even with 100's of data feed your lag could reach 10s. If you are running some neural network based complex calculations then it can though.
You can use something like the time command to get an idea about your execution time.