Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Compression on IB Live data

    General Code/Help
    4
    17
    2340
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bb2
      bb2 last edited by bb2

      Hello,

      I am struggling with data collection from IB under some conditions. I want to receive data from IB. I have tried the following:

      data = ibstore.getdata(dataname=instrumentName, timeframe=bt.TimeFrame.Seconds, compression=30)
      cerebro.adddata(data)
      cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=1)
      cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=15)
      cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=30)
      cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=60)
      

      The code above fetches the data from IB in 30 Seconds bars as defined just for historical data. However, once the data is live, it arrives like tick data(under 1 Sec).
      I need both of the 30 Sec bars and replayed ones.
      Is there any way to let backtrader fetch 30 Seconds bars only?
      The real issue here is that this strategy runs on 100-150 products and the strategy doesn't need to wake up every few milliseconds.

      Thanks

      1 Reply Last reply Reply Quote 0
      • A
        ab_trader last edited by

        Probably (I am not familiar with live trading deep enough, but based on the example provided in the docs) cerebro.resampledata(...) need to be used instead of cerebro.replaydata(...).

        • If my answer helped, hit reputation up arrow at lower right corner of the post.
        • Python Debugging With Pdb
        • New to python and bt - check this out
        B 1 Reply Last reply Reply Quote 1
        • B
          backtrader administrators @ab_trader last edited by

          @ab_trader said in Compression on IB Live data:

          Probably (I am not familiar with live trading deep enough, but based on the example provided in the docs) cerebro.resampledata(...) need to be used instead of cerebro.replaydata(...).

          Indeed.

          replaydata is called so, because it does replay what the data does.

          Documentation:

          • Docs - Data Resampling
          • Docs - Data Replaying

          @bb2 said in Compression on IB Live data:

          Is there any way to let backtrader fetch 30 Seconds bars only?

          Nothing to do with backtrader. Interactive Brokers doesn't provide that service for live data. That's only for the historical part (and backfilling where possible)

          @bb2 said in Compression on IB Live data:

          The real issue here is that this strategy runs on 100-150 products and the strategy doesn't need to wake up every few milliseconds.

          Don't replay the data then.

          bb2 1 Reply Last reply Reply Quote 0
          • bb2
            bb2 last edited by

            @backtrader Frankly I do not have any performance issues when I do backtesting on a csv data. But when I run the same strategy on live IB data, after 1-2 minutes the data starts to stall and thus I am not able to do 1-min trades anymore. Have you tried getting live data of around 20-30 products from IB and just printing them out? Even without replaying or resampling, the 'live' data is out of sync and delayed about a minute and goes off 'live' status for a while. I would be happy to help to resolve this problem if this persists for others as well.

            Best

            B 1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators @bb2 last edited by

              @bb2 said in Compression on IB Live data:

              Have you tried getting live data of around 20-30 products from IB and just printing them out?

              No. For such a feat I would use a different data provider/broker, given the many limitations present in the data service offered by Interactive. For example: https://interactivebrokers.github.io/tws-api/historical_limitations.html

              bb2 1 Reply Last reply Reply Quote 0
              • bb2
                bb2 @backtrader last edited by

                @backtrader which one would you suggest for such an operation? (both for live data acquisition and making orders)

                1 Reply Last reply Reply Quote 0
                • bb2
                  bb2 @backtrader last edited by bb2

                  @backtrader not replaying is unfortunately not an option. However, the following would solve my problem, I am not sure if it is possible to do it though:

                  tick data is resampled to 30 seconds.
                  replay data is called on the resampled data ( 30 sec bars).

                  in this case replayed data can only wake up every 30 secs instead of every tick. How can I make this happen?

                  The strategy needs to have access to live data (doesn't need to be tick, but 20-30 secs is enough) on high timeframes, because it relies on indicator's live values in high timeframes, too. As far as I know, resampling only takes into account the closed bar, thus live values wouldn't be considered. Please correct me if I am wrong.

                  Best

                  B 1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators @bb2 last edited by

                    @bb2 said in Compression on IB Live data:

                    replay data is called on the resampled data ( 30 sec bars).
                    in this case replayed data can only wake up every 30 secs instead of every tick. How can I make this happen?

                    You seem to fail to understand that replaydata has the prefix replay because it delivers each and every tick received to actually replay what the market does.

                    If you only want the complete 30 seconds every 30 seconds, you have to resample.

                    Please the read the documentation linked above.

                    bb2 1 Reply Last reply Reply Quote 0
                    • bb2
                      bb2 @backtrader last edited by bb2

                      @backtrader seems I was not clear enough. how can I replay on the resampled data? Let's assume the IB tick data arrives every 250ms. Instead of directly replaying on this tick data, I want to first resample the tick data to 30 secs and run replay on 30 sec data. How would that be possible?

                      B 1 Reply Last reply Reply Quote 0
                      • B
                        backtrader administrators @bb2 last edited by

                        Have you tried replaydata(resampledata(...))? It may of course work or bring the system down to its knees ...

                        bb2 1 Reply Last reply Reply Quote 0
                        • bb2
                          bb2 @backtrader last edited by bb2

                          @backtrader ... seriously? my approach could be wrong. though do you have anything else in mind as a solution? resampling everything for sure wouldn't work cos recent data plays important role. Is this doable with backtrader or am I simply wasting my time here?

                          1 Reply Last reply Reply Quote 0
                          • bb2
                            bb2 last edited by bb2

                            if still not clear, the strategy has to work on open bar ( which just needs to be updated every 20-30 secs) in low and high timeframes, how would you make it happen with 250ms tick data on backtrader?

                            B 1 Reply Last reply Reply Quote 0
                            • bb2
                              bb2 last edited by

                              I have just noticed if I resample tick data to 30 secs first and then replaydata on higher timeframes, the replayed data only updates with 30 secs intervals together with resampled data instead of updating with tick data. Is this a bug or a feature? :D

                              1 Reply Last reply Reply Quote 0
                              • B
                                backtrader administrators @bb2 last edited by

                                @bb2 said in Compression on IB Live data:

                                if still not clear,

                                No it's not clear.

                                You are again describing resampling but you say you want to replay.

                                @bb2 said in Compression on IB Live data:

                                I have just noticed if I resample tick data to 30 secs first and then replaydata on higher timeframes, the replayed data only updates with 30 secs intervals together with resampled data instead of updating with tick data. Is this a bug or a feature?

                                It is the intended behavior. A bar resampled to 30 seconds, will be delivered every 30 seconds. If you replay that 30 seconds bar, the lowest possible replay frequency is obviously 30 seconds.

                                Incredibly, this is what was recommended:

                                @backtrader said in Compression on IB Live data:

                                Have you tried replaydata(resampledata(...))?

                                @bb2 said in Compression on IB Live data:

                                @backtrader ... seriously?

                                Yes. Apparently this is what you want in spite of the obscurity of the posts.

                                1 Reply Last reply Reply Quote 1
                                • bb2
                                  bb2 last edited by bb2

                                  referring to the code:

                                  data = ibstore.getdata(dataname=instrumentName)
                                  cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1)
                                  cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=5)
                                  

                                  it is not obvious to the coder that the replay is done on the resampled data. I thought resampling and replaying were done independently on the tick data. You could probably update the doc to point this out. Thank you for your help.

                                  A 1 Reply Last reply Reply Quote 0
                                  • A
                                    ab_trader @bb2 last edited by

                                    @bb2 I might be wrong, but looks like your baseline data obtained from ib is resampled to 30 sec bars during data delivery. So there are no tick data available in the system.

                                    • If my answer helped, hit reputation up arrow at lower right corner of the post.
                                    • Python Debugging With Pdb
                                    • New to python and bt - check this out
                                    1 Reply Last reply Reply Quote 0
                                    • N
                                      NothingButHands last edited by

                                      @bb2 It makes sense if you realize that resampledata() performs the resampling in-place; something not specified very clearly in the docs

                                      1 Reply Last reply Reply Quote -1
                                      • 1 / 1
                                      • First post
                                        Last post
                                      Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors