Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Example of adding live-data to static in strategy?

    General Code/Help
    2
    51
    20830
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RandyT
      RandyT last edited by

      Trying to get my head around the following set of requirements. I'm taking the system that I have coded that is succesfully backtesting against static data and now trying to add in a live data source to do more realtime calculation of signals.

      Data0 = Live data for traded instrument 'BAR'
      Data1 = Live data for signaling instrument 'FOO' used to signal entry/exit for the current day
      Data2 =  Years of static daily timeframe data for signaling instrument 'FOO'
      

      I need to add the close value for probably a minute timeframe off of Data1 to the value from yesterday and beyond found in daily timeframe data of Data2.

      I've looked at most of the samples and am not finding anything that is augmenting the Data2 to calculate the indicator values that need lookback of multiple years.

      This all works when dealing with daily static data provided by Data2. I'm just a bit confused as to where to take the value of the minute close of Data1 as if it was the Daily close value for today. I am really only interested in triggering a signal off of the actual close for today to execute a trade on the next bar of Data0.

      As a side note, I don't really trust the IB historical data enough to use it for the multi-year lookback needed here, so I use a different static source for the historical daily data and really only want to add today's values to that other static data source to calculate the signal.

      Advance thanks for your help.

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        The topic seems to summarize itself as: *How to calculate an indicator which is based in data from different timeframes? *

        This is supported through a mechanism that was named: Coupling

        See here:

        • https://www.backtrader.com/docu/mixing-timeframes/indicators-mixing-timeframes.html

        or here

        • https://www.backtrader.com/blog/posts/2016-05-05-indicators-mixing-timeframes/indicators-mixing-timeframes.html

        (The content should be basically the same)

        Note: the mechanism works but cross-plotting doesn't. The changes involved in releasing the 1.9.x series for a new data synchronization mechanism, meant changes in plotting and the mechanism originally used for the cross-plotting no longer works.

        1 Reply Last reply Reply Quote 0
        • RandyT
          RandyT last edited by

          Still struggling with this concept a bit.

          How do I specifically couple data2 and data1 from my example above?

          Also, looking at the example code you referenced, does the resample implicitly create data1?

          I'm doing something like the following:

              # Parse static ES data file
              data0 = bt.feeds.PandasData()
              cerebro.adddata(data0, name='ES')
          
              # Parse SPY data file
              data1 = bt.feeds.PandasData()
              cerebro.adddata(data1, name='SPY')
          
          if live:
                  IBDataFactory = ibstore.getdata
                  data2 = IBDataFactory(dataname=symbol)
                  cerebro.adddata(data2, name='ES-live')
          
                  # SPY Live Data
                  data3 = IBDataFactory(dataname=symbol)
                  cerebro.adddata(data3, name='SPY-live')
          

          I've obviously left out a lot of the details in that code but the connection works and am getting data based on the log info of the IBgateway.

          I'm still not clear on how I couple data1 and data3 to get live SPY data to pass to my indicators. I've spent a lot of time in the debugger trying to sort this out and am not finding the magic.

          Thanks for your help.

          1 Reply Last reply Reply Quote 0
          • RandyT
            RandyT last edited by

            @backtrader I also wonder if we are not talking about different issues here.

            In Live data mode, I have 4 datas. 2 of those datas sources are static data read from file in order to give me enough backdata to feed my indicators. The other 2 datas are live streaming data that I wish to execute trades against one of them and to continue to calculate the indicator values based on the new live data coming in to provide today's values for the static data from yesterday and beyond.

            I am not resampling to generate this data.

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              This was completely misunderstood. It seems you are simply looking for backfilling to warm up the indicators.

              The IBData supportf backfilling from existing data sources with:

              • backfill_from (default: None)

                An additional data source can be passed to do an initial layer of backfilling. Once the data source is depleted and if requested, backfilling from IB will take place. This is ideally meant to backfill from already stored sources like a file on disk, but not limited to.

                  # Parse static ES data file
                  data0 = bt.feeds.PandasData()
              
                  # Parse SPY data file
                  data1 = bt.feeds.PandasData()
              
                  IBDataFactory = ibstore.getdata
                  data2 = IBDataFactory(dataname=symbol)
                  cerebro.adddata(data2, name='ES-live', backfill_from=data0)
              
                  # SPY Live Data
                  data3 = IBDataFactory(dataname=symbol, backfill_from=data1)
                  cerebro.adddata(data3, name='SPY-live')
              
              1 Reply Last reply Reply Quote 0
              • RandyT
                RandyT last edited by

                Excellent. I will give this a try.

                As I wrote above, I do not trust the IB historical data. I have another source of very solid historical data that I prefer to use and would then like to augment that with the current day's data for making trading decisions.

                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators last edited by

                  That's the point of passing your data0 which you have loaded from a file as backfill_from. Anything which is in that file will be considered, and only the delta between the end of the file and the start of the live feed will be pulled from IB.

                  1 Reply Last reply Reply Quote 0
                  • RandyT
                    RandyT last edited by RandyT

                    Just a bit of a clarification for anyone running across this thread in the future, the backfill_from parameter should be used with the call to IBDataFactory() instead of the .adddata() object as show in one of the examples above.

                    That said, I'm running into the following error when I attempt to take this approach:

                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 809, in run
                        runstrat = self.runstrategies(iterstrat)
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 933, in runstrategies
                        self._runnext(runstrats)
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1152, in _runnext
                        drets = [d.next(ticks=False) for d in datas]
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1152, in <listcomp>
                        drets = [d.next(ticks=False) for d in datas]
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feed.py", line 339, in next
                        ret = self.load()
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feed.py", line 411, in load
                        _loadret = self._load()
                      File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feeds/ibdata.py", line 528, in _load
                        for alias in self.lines.getaliases():
                    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'getaliases'
                    
                    1 Reply Last reply Reply Quote 0
                    • B
                      backtrader administrators last edited by

                      Indeed. That's the problem with snippet typing ... one of the backfill_from got wrongly positioned (luckily the 2nd was where it had to be) The corrected snippet.

                        # Parse static ES data file
                          data0 = bt.feeds.PandasData(dataname='my-es-file')
                      
                          # Parse SPY data file
                          data1 = bt.feeds.PandasData(dataname='my-spy-file')
                      
                          IBDataFactory = ibstore.getdata
                          data2 = IBDataFactory(dataname=symbol, backfill_from=data0)
                          cerebro.adddata(data2, name='ES-live')
                      
                          # SPY Live Data
                          data3 = IBDataFactory(dataname=symbol, backfill_from=data1)
                          cerebro.adddata(data3, name='SPY-live')
                      

                      The getaliases seems to be one of the points where a refactoring went wrong. The correct call is getlinealiases and has been pushed to the master.

                      1 Reply Last reply Reply Quote 0
                      • RandyT
                        RandyT last edited by RandyT

                        With this change, when running in live mode now with the backfill_from , I don't seem to have an initialized self.position in next() of strategy.

                        Not sure yet if this is something that I have failed to do or issue.

                        This works when not running with live data with local static data only.

                        Looking at this in the debugger:

                        (Pdb) foo = self.broker.getposition(data=self.data_es)
                        (Pdb) foo.size
                        0
                        

                        So the data is there, just not initialized to self.position.

                        1 Reply Last reply Reply Quote 0
                        • B
                          backtrader administrators last edited by

                          The status/size reported by self.position (which is a Position instance with size and price attributes) is not related to what's being used to fill the value in the self.data.

                          In your original example there are 4 data feeds and now the system should be down to 2. self.position is a shortcut to get the actual position on self.datas[0] (aka self.data or self.data0).

                          Additionally and if code has been used from ibtest.py, any buy/sell action will first take place when the LIVE notification is received (the initial status whilst backfilling is DELAYED

                          1 Reply Last reply Reply Quote 0
                          • RandyT
                            RandyT last edited by

                            Thanks for the hint. That seems to have resolved my issue.

                            Count me as one who is officially running backtrader fully auto on paper account.

                            @backtrader thanks again for all of your help and a great set of tools that you have made available to us all.

                            1 Reply Last reply Reply Quote 0
                            • RandyT
                              RandyT last edited by RandyT

                              Continuing to have some trouble here.

                              System happily waited for live data to start flowing. However, as soon as the market opened this evening, the system crashed with the following backtrace.

                              Server Version: 76
                              TWS Time at connection:20170102 19:10:03 MST
                              Traceback (most recent call last):
                              File "systems/system.py", line 335, in <module> runstrategy()
                              File "systems/system.py", line 149, in runstrategy
                                  results = cerebro.run(runonce=False, tradehistory=True, exactbars=args.exactbars)
                              File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 809, in run
                                  runstrat = self.runstrategies(iterstrat)
                              File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 933, in runstrategies
                                  self._runnext(runstrats)
                              File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1166, in _runnext                                        
                                  dt0 = min((d for i, d in enumerate(dts)
                              
                              ValueError: min() arg is an empty sequence
                              

                              I've set a break in notify_data() as follows and am never getting there. Status is never data.LIVE.
                              The market data is available as I have another system that is able to pull the requested data.

                              def notify_data(self, data, status, *args, **kwargs):
                                      if self.p.printout:
                                          print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
                              
                                      if status == data.LIVE:
                                          self.datastatus = True
                                          import pdb; pdb.set_trace()
                              

                              Also getting the following prints:

                              ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
                              ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
                              ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
                              ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
                              ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
                              

                              I can seemingly resolve this issue by first doing .adddata() for the live data stream before the .resample(). But I was under the impression that the .adddata() was not required if doing a .resample() as seen in the ibtest.py code.

                              I never get LIVE data status though with this configuration.

                              1 Reply Last reply Reply Quote 0
                              • B
                                backtrader administrators last edited by backtrader

                                The <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm> notifications are OK. IB chose to notify that the status of the different data server through errors 21xx. The text in errMsg is what's actually makes the diffference, but in this case it says OK.

                                There shouldn't be any need to use adddata, as you have already seen in the sample.

                                From the trace, the error can be tracked down to:

                                dt0 = min((d for i, d in enumerate(dts)
                                
                                ValueError: min() arg is an empty sequence
                                

                                The entire code fragment which is in play:

                                                # Get index to minimum datetime
                                                if onlyresample or noresample:
                                                    dt0 = min((d for d in dts if d is not None))
                                                else:
                                                    dt0 = min((d for i, d in enumerate(dts)
                                                               if d is not None and i not in rsonly))
                                

                                Your code fails in the 2nd part which is for cases in which no resampling takes place or a mix of adddata and resample/replay takes place.

                                In any case, one can only be in that code fragment if one of the existing data feeds has reported to have something to deliver.

                                This is only a guess, because there is no description in your post as to what you are actually doing (why you don't get LIVE is part of the unknown configuration)

                                1 Reply Last reply Reply Quote 0
                                • RandyT
                                  RandyT last edited by

                                  One suspicion that I would first ask, is it possible to use backfill_from with data in different timeframe (Daily in my case) than the data timeframe used for the live feed?

                                  To give a bit more detail as I try to sort through where this might be failing... I am doing the following:

                                   # Parse static ES data file of daily OHLCV
                                      data0 = bt.feeds.PandasData(dataname='my-es-file') 
                                  
                                      # Parse SPY data file of daily OHLCV
                                      data1 = bt.feeds.PandasData(dataname='my-spy-file')
                                  
                                      IBDataFactory = ibstore.getdata
                                      # ES Live Data
                                      data2 = IBDataFactory(dataname=symbol, backfill_from=data0, 
                                                                               timeframe=bt.Timeframe.Seconds,
                                                                               fromdate=None,
                                                                               historical=False,
                                                                               rtbar=False,
                                                                               qcheck=0.5,
                                                                               what=None,
                                                                               latethrough=False,
                                                                               tz=None,
                                                                               backfill_start=True,
                                                                               backfill=True,
                                                                                compression=5)
                                      cerebro.resampledata(data3, timeframe=bt.TimeFrame.Minutes, compression=1, 
                                                                               bar2edge=True,
                                                                               adjbartime=True,
                                                                               rightedge=True,
                                                                               takelate=True)
                                  
                                      # SPY Live Data
                                      data3 = IBDataFactory(dataname=symbol, backfill_from=data1,
                                                                               timeframe=bt.TimeFrame.Minutes,
                                                                               fromdate=None,
                                                                               historical=False,
                                                                               rtbar=False,
                                                                               qcheck=0.5,
                                                                               what=None,
                                                                               latethrough=False,
                                                                               tz=None,
                                                                               backfill_start=True,
                                                                               backfill=True)
                                      cerebro.resampledata(data3, timeframe=bt.TimeFrame.Days, compression=1, 
                                                                               bar2edge=True,
                                                                               adjbartime=True,
                                                                               rightedge=True,
                                                                               takelate=True)
                                  

                                  With the above setup, I now error out in my indicator with the following error:

                                  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/back
                                  trader/linebuffer.py", line 182, in get                                           return list(islice(self.array, start, end))
                                  ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys.max
                                  size.
                                  

                                  In the debugger, I see:

                                  (Pdb) len(self)
                                  756
                                  (Pdb) len (self.dval8)
                                  756
                                  (Pdb) self.dval8[0]
                                  0.49771632773232516
                                  (Pdb) self.dval8.get(size=252, ago=0)
                                  *** ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys
                                  .maxsize. 
                                  (Pdb) self.dval8.get(size=8, ago=0)
                                  [0.49987104505171959, 0.5011817035953996, 0.50303547679268779, 0.5033757977972
                                  1131, 0.5014079085533496, 0.4985198983481765, 0.49832603424557598, 0.49771632773232516]  
                                  

                                  Requesting a slice bigger than 8 items fails with the above error.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    backtrader administrators last edited by backtrader

                                    is it possible to use backfill_from with data in different timeframe (Daily in my case) than the data timeframe used for the live feed?

                                    The platform will NOT prohibit that you do that, but the outcome is clearly undefined. Indicators (if working at all) will be calculating values based on completely different bar sizes (for example) and the outcome will make no sense.

                                    Requesting a slice bigger than 8 items fails with the above error.

                                    The default ibtest.py applies exactbars=1 to cerebro.run(...). Although not shown above, this is for sure the problem. This switches all objects into only buffering the minimum needed amount of data, as reported by the dependencies recognized during creation.

                                    Your indicator has reported to only need a lookback period of 8 positions. Looking back 252 positions breaks it.

                                    1 Reply Last reply Reply Quote 0
                                    • RandyT
                                      RandyT last edited by RandyT

                                      If I run with exactbars=0, I get the error previously reported:

                                        File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/back
                                      trader/cerebro.py", line 1166, in _runnext                                        
                                           dt0 = min((d for i, d in enumerate(dts)
                                      ValueError: min() arg is an empty sequence
                                      

                                      exactbars set to 1 or -1 produce the error in the indicator.

                                      In my indicator, I have the following in __init__():

                                         def __init__(self):
                                              self.addminperiod(756)
                                      

                                      This is why I see a size of 756 when we hit next(): for the first time in the indicator. How can I have a size of 756 yet not have 756 values available to the .get?

                                      To clarify the first question about different sizes, I am resampling the live data to the same timeframe as the static local data, so I am assuming that resolves any issue there. I am only using the SPY data for indicator calculations, so I would not expect that to be an issue.

                                      1 Reply Last reply Reply Quote 0
                                      • B
                                        backtrader administrators last edited by

                                        Because there is always a minimum period of 1 which is transmitted by data feeds. Try adding 757

                                        1 Reply Last reply Reply Quote 0
                                        • RandyT
                                          RandyT last edited by

                                          Adding 757 as minperiod has the same error:

                                          (Pdb) len(self)
                                          757
                                          

                                          This does work with static data only. It is the addition of the live data and backfill_from that is producing this issue.

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            backtrader administrators last edited by

                                            (Correction added above to remark that "The platform will NOT ... when mixing different timeframes with backfill_from)

                                            It may be good to try to summarize the situation. (Please correct any misinterpretation):

                                            • Above the indicator was looking to get 252 items, but it failed because it only had 8 in the buffer, because the code is working with exactbars=1 which reduces the buffers of all objects to the minimum required by the dependencies.

                                            • 756 was not working in spite of having self.addminperiod(756). This was interpreted, probably wrong, as you not having the full 756 values but probably 755 due to the overlapping nature of the datas.

                                              May it be that you still had only the 8 items from the post above?

                                            Summary of exactbars:

                                                    Possible values:
                                                      - ``True`` or ``1``: all "lines" objects reduce memory usage to the
                                                        automatically calculated minimum period.
                                            
                                                        If a Simple Moving Average has a period of 30, the underlying data
                                                        will have always a running buffer of 30 bars to allow the
                                                        calculation of the Simple Moving Average
                                            
                                                        - This setting will deactivate ``preload`` and ``runonce``
                                                        - Using this setting also deactivates **plotting**
                                            
                                                      - ``-1``: data feeds and indicators/operations at strategy level will
                                                        keep all data in memory.
                                            
                                                        For example: a ``RSI`` internally uses the indicator ``UpDay`` to
                                                        make calculations. This subindicator will not keep all data in
                                                        memory
                                            
                                                        - This allows to keep ``plotting`` and ``preloading`` active.
                                            
                                                        - ``runonce`` will be deactivated
                                            
                                                      - ``-2``: data feeds and indicators kept as attributes of the
                                                        strategy will keep all points in memory.
                                            
                                                        For example: a ``RSI`` internally uses the indicator ``UpDay`` to
                                                        make calculations. This subindicator will not keep all data in
                                                        memory
                                            
                                                        If in the ``__init__`` something like
                                                        ``a = self.data.close - self.data.high`` is defined, then ``a``
                                                        will not keep all data in memory
                                            
                                                        - This allows to keep ``plotting`` and ``preloading`` active.
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 1 / 3
                                            • First post
                                              Last post
                                            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors