Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Programmatically extending a datafeed

    General Code/Help
    data data feed pandas
    5
    11
    2677
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tmorgan4 last edited by tmorgan4

      This issue has had me stumped for at 2+ weeks and I've decided to reach out for help.

      A common question I see about backtrader is how to extend a datafeed to include additional columns in the input data file (e.g. indicators calculated outside of backtrader). This has been thoroughly documented and a good example is shown in pandas-data-optix.py which is partially shown below:

      class PandasDataOptix(btfeeds.PandasData):
      
          lines = ('optix_close', 'optix_pess', 'optix_opt',)
          params = (('optix_close', -1),
                    ('optix_pess', -1),
                    ('optix_opt', -1))
      
          datafields = btfeeds.PandasData.datafields + (
              ['optix_close', 'optix_pess', 'optix_opt'])
      

      What if we didn't know the names of the additional columns (or there were hundreds of columns that are unique to each data file) and want to automatically add every column not in the standard list (below)?

      ['datetime', 'open', 'high', 'low', 'close', 'volume', 'openinterest']
      

      The heavy use of Metaclasses is beyond my knowledge level at the moment and using the debugger to follow how the PandasDataOptix class is created step-by-step is difficult.

      The issue is that the lines, params, and datafields variables must be populated when the PandasDataOptix class is created. The MetaBase.__call__() method calls Lines.__init__() which cycles through the supplied lines and creates a LineBuffer for each. Since this all happens before PandasDataOptix.__init__() no arguments can be passed to the class (such as lines, params, and datafields).

      I got it working by manually re-running several *._derive() methods to overwrite some of the PandasDataOptix class attributes after it was created but this is very hacky and surely not the best way:

      PandasDataOptix.linealias = la = newcls.linealias._derive('la_' + name_cls, newlalias, oblalias)
      PandasDataOptix.lines = newcls.lines._derive(name_cls, newlines, extralines, morebaseslines, linesoverride, lalias=la)
      PandasDataOptix.plotlines = newcls.plotlines._derive('pl_' + name_cls, newplotlines, morebasesplotlines, recurse=True)
      

      It seems that some of the steps which initialize the PandasData class could be manually called afterward to include additional arguments but so far I'm stumped. Thanks in advance.

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by backtrader

        May have you gone too deep into the details?

        datafields is actually a relic and should be removed at some point in time (or at least deprecated and not needed)

        Extending the data feed in real-time should be possible without having to call the internal _derive methods.

        thenewlines = ['newline1', 'newline2']
        
        mydict = dict(
            lines=tuple(thenewlines),
            datafieds=bt.feeds.PandasData.datafields + thenewlines,
        )
        
        MyPandasClass = type('mypandasname', (bt.feeds.PandasData,), mydict)
        

        This is 100% Python idiomatic and the usual way to create classes dynamically.

        You can of course add a plotinfo and plotlines dictionaries (with the same syntax as in manual class declaration) to mydict and they will be taken into account.

        T B 2 Replies Last reply Reply Quote 1
        • T
          tmorgan4 @backtrader last edited by

          You are absolutely correct! Despite hours of searching for how to pass arguments to a class as it's created I never came across this alternative method for creating a class dynamically. It works perfectly.

          You have taught me so much about Python just from snooping through backtrader code. Thank you Daniel!

          In case anyone else finds this useful here is the data-pandas-optix.py file (backtrader/samples/data-pandas) with all parameters defined externally from the PandasData class. The original class creation is included but commented out.

          #!/usr/bin/env python
          # -*- coding: utf-8; py-indent-offset:4 -*-
          ###############################################################################
          #
          # Copyright (C) 2015, 2016 Daniel Rodriguez
          #
          # This program is free software: you can redistribute it and/or modify
          # it under the terms of the GNU General Public License as published by
          # the Free Software Foundation, either version 3 of the License, or
          # (at your option) any later version.
          #
          # This program is distributed in the hope that it will be useful,
          # but WITHOUT ANY WARRANTY; without even the implied warranty of
          # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
          # GNU General Public License for more details.
          #
          # You should have received a copy of the GNU General Public License
          # along with this program.  If not, see <http://www.gnu.org/licenses/>.
          #
          ###############################################################################
          from __future__ import (absolute_import, division, print_function,
                                  unicode_literals)
          import argparse
          import backtrader as bt
          import backtrader.feeds as btfeeds
          import pandas
          
          
          # class PandasDataOptix(btfeeds.PandasData):
          #
          #     lines = ('optix_close', 'optix_pess', 'optix_opt',)
          #     params = (('optix_close', -1),
          #               ('optix_pess', -1),
          #               ('optix_opt', -1))
          #
          #     datafields = btfeeds.PandasData.datafields + (
          #         ['optix_close', 'optix_pess', 'optix_opt'])
          
          
          ######## Dynamic class creation ##########
          
          lines = ('optix_close', 'optix_pess', 'optix_opt',)
          params = (('optix_close', -1),
                    ('optix_pess', -1),
                    ('optix_opt', -1))
          
          datafields = btfeeds.PandasData.datafields + (
              ['optix_close', 'optix_pess', 'optix_opt'])
          
          mydict = dict(
              lines=tuple(lines),
              params=params,
              datafields=bt.feeds.PandasData.datafields + list(lines),
              )
          
          PandasDataOptix = type('PandasDataOptix', (btfeeds.PandasData,), mydict)
          
          
          class StrategyOptix(bt.Strategy):
          
              def next(self):
                  print('%03d %f %f, %f' % (
                      len(self),
                      self.data.optix_close[0],
                      self.data.lines.optix_pess[0],
                      self.data.optix_opt[0],))
          
          
          def runstrat():
              args = parse_args()
          
              # Create a cerebro entity
              cerebro = bt.Cerebro(stdstats=False)
          
              # Add a strategy
              cerebro.addstrategy(StrategyOptix)
          
              # Get a pandas dataframe
              datapath = ('../../datas/2006-day-001-optix.txt')
          
              # Simulate the header row isn't there if noheaders requested
              skiprows = 1 if args.noheaders else 0
              header = None if args.noheaders else 0
          
              dataframe = pandas.read_csv(datapath,
                                          skiprows=skiprows,
                                          header=header,
                                          parse_dates=True,
                                          index_col=0)
          
              if not args.noprint:
                  print('--------------------------------------------------')
                  print(dataframe)
                  print('--------------------------------------------------')
          
              # Pass it to the backtrader datafeed and add it to the cerebro
              data = PandasDataOptix(dataname=dataframe)
          
              cerebro.adddata(data)
          
              # Run over everything
              cerebro.run()
          
              # Plot the result
              if not args.noplot:
                  cerebro.plot(style='bar')
          
          
          def parse_args():
              parser = argparse.ArgumentParser(
                  description='Pandas test script')
          
              parser.add_argument('--noheaders', action='store_true', default=False,
                                  required=False,
                                  help='Do not use header rows')
          
              parser.add_argument('--noprint', action='store_true', default=False,
                                  help='Print the dataframe')
          
              parser.add_argument('--noplot', action='store_true', default=False,
                                  help='Do not plot the chart')
          
              return parser.parse_args()
          
          
          if __name__ == '__main__':
              runstrat()
          
          1 Reply Last reply Reply Quote 2
          • L
            LAbrk last edited by

            Hi all.
            Im new on BT, im researching the code to know if is compatible with mi live data. I'm seeing that bt is an incredible tool!

            @backtrader my doubt, is this example valid for live datafeed?
            Or do I have to add the indicators in another way?

            Thanks a lot!

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              The extension mechanism is generic. How to fill the values is what's data feed specific.

              1 Reply Last reply Reply Quote 0
              • B
                backtrader administrators @backtrader last edited by

                @backtrader said in Programmatically extending a datafeed:

                datafields is actually a relic and should be removed at some point in time (or at least deprecated and not needed)

                The need to use datafields for the PandasData was already removed some versions ago

                L 1 Reply Last reply Reply Quote 0
                • L
                  LAbrk @backtrader last edited by

                  @backtrader
                  Great. Thank you for the reply.
                  Good to have an active community.
                  I'm still trying.

                  1 Reply Last reply Reply Quote 0
                  • B
                    btr last edited by

                    @backtrader @tmorgan4 Hey, I am trying to understand this and the question is about programatically extending the data feed when you do not know the additional column names of df. After the discussion, @tmorgan4 posted a code where ('optix_close', 'optix_pess', 'optix_opt',) lines are already known. How is this created programatically? Am I missing something?

                    thanks you in advance for the response

                    run-out 1 Reply Last reply Reply Quote 0
                    • run-out
                      run-out @btr last edited by

                      @btr What part do you feel like you are missing?

                      B 1 Reply Last reply Reply Quote 0
                      • B
                        btr @run-out last edited by

                        @run-out I needed a break and coffee. It was silly

                        run-out 1 Reply Last reply Reply Quote 1
                        • run-out
                          run-out @btr last edited by

                          @btr said in Programmatically extending a datafeed:

                          I needed a break and coffee. It was silly

                          Hilarious! Hope you had a nice break!

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post
                          Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                          $(document).ready(function () { app.coldLoad(); }); }