Navigation

    Backtrader Community
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Multiple Live Data Feeds

    General Discussion
    3
    8
    1833
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      TOTORO last edited by

      I read through Docs/Forum, but could not find a definitive answer/solution.

      Is it possible to monitor multiple live data feeds in backtrader? Let's say I want to look for an entry based on an indicator (eg RSI) - I would like to parse live data feeds of 10s of instuments and choose the best 3.

      I assume there is no easy workaround, so I would be grateful if someone could contribute which of the following solutions seem to be most reasonable:

      • running x algos in parallel with x single live data feeds - this becomes very impractical if you want to monitor many instruments. If I want to only choose 3 intruments out of 50 I monitor, this becomes very unefficient as well unless there is a shared DB (or sth similar) that signals once 3 assets have been already chosen (so other scripts can stop parsing looking for an entry). Also this would still probably require some flow of information between individual algos for proper risk management etc.

      • trying to utilize pararelization and shared/ditributed memory with multiprocessing module or similar - I dont know too much about this so I cant even tell if this has a potential to solve my problem. Thanks for your input.

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by backtrader

        @totoro said in Multiple Live Data Feeds:

        Is it possible to monitor multiple live data feeds in backtrader?

        @totoro said in Multiple Live Data Feeds:

        I read through Docs/Forum, but could not find a definitive answer/solution.

        The docs and features description clearly indicate that you can work with multiple data feeds. The platform doesn't make distinctions between live and non-live. The limitations usually arise from data providers. Interactive Brokers is a well known example due to the limitations it imposes on backfilling with pacing violations.

        @totoro said in Multiple Live Data Feeds:

        • trying to utilize pararelization and shared/ditributed memory with multiprocessing module or similar - I dont know too much about this so I cant even tell if this has a potential to solve my problem. Thanks for your input.

        backtrader only uses multiprocessing for optimization. Nothing prevents you of course from delegating work to sub-processes which can run in the background whilst the engine is waiting for data and/or processing that data for indicator calculations, but all this will reside and run in a single process.

        @totoro said in Multiple Live Data Feeds:

        • running x algos in parallel with x single live data feeds

        You probably have to define what you mean by "running in parallel". Python has a GIL limitation and things don't run in parallel. You can still run multiple strategies in backtrader. How they communicate with each other is up to you (simple queue.queue do actually suffice)

        T 1 Reply Last reply Reply Quote 0
        • T
          TOTORO @backtrader last edited by

          @backtrader Thanks for a prompt response!

          Well, I guess my confusion comes from the point that working with multiple data feeds in parallel would require multiprocessing or similar, right? Otherwise it is more like looping through data feeds and introduces lag/latency (or do you assume that the delay is essentialy negligible even if we have 100s of feeds?)

          @backtrader said in Multiple Live Data Feeds:

          running x algos in parallel with x single live data feeds

          You probably have to define what you mean by "running in parallel". Python has a GIL limitation and things don't run in parallel. You can still run multiple strategies in backtrader. How they communicate with each other is up to you (simple queue.queue do actually suffice)

          Here what I meant was to simply run each algo with single data feed independently (as a seprate program/script) and create some sort of communication channel - I would say this is a straight-forward "brute-force" approach. This way I could ensure that multiple data feeds are parsed simultaneously (altough in separate programs/scripts) without multiprocessing or similar.

          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            In this Blog - Multi Example post, you can see multiple data feeds simultaneously.

            You can also add as many strategies as needed and if you wish restrict each strategy to work on a single data feed (choose it by name, index ...)

            T 1 Reply Last reply Reply Quote 0
            • T
              TOTORO @backtrader last edited by

              @backtrader Thanks, I have read the post, however I still struggle to understand how this would work in my scenario - in the code above you seem to loop through data feeds (self.datas) one by one. If we deal with hundreds of live feeds where latency matters (ideally no delay between 1st and 100th data feed), this can spoil the results. Am I missing sth or you just assume that delay would be negligible? Thanks for your time again.

              K 1 Reply Last reply Reply Quote 0
              • K
                kausality @TOTORO last edited by

                @totoro You would have to process each data feed separately and there is no workaround that in any language or using any framework. Even using pandas you have to filter the DataFrame according to the symbol being processed in this round.

                If your strategy relies on microseconds precision then you are entering the realm of HFT and in that case, you have to evaluate if Python is really the correct tool of your choice.

                T 1 Reply Last reply Reply Quote 0
                • T
                  TOTORO @kausality last edited by

                  @kausality Thanks, do you have any idea about the delay magnitude if we talk about hundreds of live data feeds?. And no, obviously I am not looking for microseconds precision, however 10s of seconds of delay might matter, thus my question. Thanks for your time.

                  K 1 Reply Last reply Reply Quote 0
                  • K
                    kausality @TOTORO last edited by

                    @totoro It depends on what you are doing inside the loop. If you are making some calculations like std devs, atr etc then I don't think that even with 100's of data feed your lag could reach 10s. If you are running some neural network based complex calculations then it can though.
                    You can use something like the time command to get an idea about your execution time.

                    1 Reply Last reply Reply Quote 1
                    • 1 / 1
                    • First post
                      Last post
                    Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                    $(document).ready(function () { app.coldLoad(); }); }