Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    'frompackages' directive functionality seems to be broken when using inheritance

    General Code/Help
    1
    2
    157
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • vladisld
      vladisld last edited by

      Background:

      'frompackages' directive was added to backtrader back in 2017 (starting from release 1.9.30.x). It allows specifying the external packages to be imported only during the instantiation of the class (usually indicator). It comes very handily during the optimizations, reducing the serialization size of the objects. More on this here

      Usage example:

      class MyIndicator(bt.Indicator):
          frompackages = (('pandas', 'SomeFunction'),)
          lines = ('myline',)
          params = (
              ('period', 50),
          )
      
          def next(self):
              print('mylines[0]:', SomeFunction(self.lines.myline[0]))
      

      Here the SomeFunction will be imported from pandas package during the instantiation of MyIndicator and not earlier.

      Testcase:

      In the same article, it was also claimed that "Both packages and frompackages support (multiple) inheritance". However, it seems to be not the case. Here a short test case:

      import os
      import backtrader as bt
      
      class HurstExponentEx(bt.indicators.HurstExponent):
          def __init__(self):
              super(HurstExponentEx, self).__init__()
      
          def next(self):
              super(HurstExponentEx, self).__init__()
              print('test')
      
      class TheStrategy(bt.Strategy):
          def __init__(self):
              self.hurst = HurstExponentEx(self.data, lag_start=10,lag_end=500)
      
          def next(self):
              print('next')
      
      def runstrat():
          cerebro = bt.Cerebro()
          cerebro.broker.set_cash(1000000)
          data_path = os.path.join(bt.__file__, '../../datas/yhoo-1996-2014.txt')
          data0 = bt.feeds.YahooFinanceCSVData(dataname=data_path)
          cerebro.adddata(data0)
          cerebro.addstrategy(TheStrategy)
          cerebro.run()
          cerebro.plot()
      
      if __name__ == '__main__':
          runstrat()
      

      where the HustExponent class is defined in backtrader as:

      class HurstExponent(PeriodN):
          frompackages = (
              ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')),
          )
          ...
      

      Unexpected behavior:

      trying to run it (using python 3.6 in my case) will produce:

      Traceback (most recent call last):
        File "test_frompackage.py", line 42, in <module>
          runstrat()
        File "test_frompackage.py", line 38, in runstrat
          cerebro.run()
        File "W:\backtrader\backtrader\cerebro.py", line 1182, in run
          runstrat = self.runstrategies(iterstrat)
        File "W:\backtrader\backtrader\cerebro.py", line 1275, in runstrategies
          strat = stratcls(*sargs, **skwargs)
        File "W:\backtrader\backtrader\metabase.py", line 88, in __call__
          _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
        File "W:\backtrader\backtrader\metabase.py", line 78, in doinit
          _obj.__init__(*args, **kwargs)
        File "test_frompackage.py", line 17, in __init__
          lag_end=500)
        File "W:\backtrader\backtrader\indicator.py", line 53, in __call__
          return super(MetaIndicator, cls).__call__(*args, **kwargs)
        File "W:\backtrader\backtrader\metabase.py", line 88, in __call__
          _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
        File "W:\backtrader\backtrader\metabase.py", line 78, in doinit
          _obj.__init__(*args, **kwargs)
        File "test_frompackage.py", line 6, in __init__
          super(HurstExponentEx, self).__init__()
        File "W:\backtrader\backtrader\indicators\hurst.py", line 82, in __init__
          self.lags = asarray(range(lag_start, lag_end))
      NameError: name 'asarray' is not defined
      

      as could be seen asarray is a method that should have been imported from numpy package upon instantiation of HurstExponent class.

      If we will directly use the HurstExponent class instead of our HurstExponentEx (which inherits from HurstExponent) - everything will work just fine.

      Analysis - TL;DR

      Exploring this is a little bit exposes the problem with the implementation of frompackages inside backtrader.

      The magic code responsible for 'frompackages' directive handling could be found in backtrader\metabase.py file inside the MetaParams.__new__ and MetaParams.donew methods. Here the 'frompackages' directive is first examined recursively (in __new__ method ) and the appropriate packages are imported using __import__ function ( in donew method)

      The problem is with the following code inside the donew method:

          def donew(cls, *args, **kwargs):
              clsmod = sys.modules[cls.__module__]
              .
              . <removed for clarity>
              .
              # import from specified packages - the 2nd part is a string or iterable
              for p, frompackage in cls.frompackages:
                  if isinstance(frompackage, string_types):
                      frompackage = (frompackage,)  # make it a tuple
      
                  for fp in frompackage:
                      if isinstance(fp, (tuple, list)):
                          fp, falias = fp
                      else:
                          fp, falias = fp, fp  # assumed is string
      
                      # complain "not string" without fp (unicode vs bytes)
                      pmod = __import__(p, fromlist=[str(fp)])
                      pattr = getattr(pmod, fp)
                      setattr(clsmod, falias, pattr)
      

      The cls parameter to this function is the class that needs to be instantiated. In our case, this is our inherited class HustExponentEx.

      So the clsmod variable will contain the module of our class - obviously the file that HurstExponentEx was defined in.

      The problem is with the last line of the above code:

      setattr(clsmod, falias, pattr)
      

      Here the setattr will introduce the imported names to the module - our module with inherited class - not the module the original base class HurstExponent is defined in.

      And it is a problem!

      Once the HurstExponent class will start executing and calling the supposedly imported functions - those will be looked in the module the HurstExponent class is defined in - and will not be found, since those names are introduced in the module of our inherited class instead!

      FIX

      The fix seems to be obvious. Introduce the imported names to the original base class module.

      The Issue was opened in backtrader2 repo:

      https://github.com/backtrader2/backtrader/issues/24

      1 Reply Last reply Reply Quote 2
      • vladisld
        vladisld last edited by

        PR submitted: https://github.com/mementum/backtrader/pull/411

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors