'frompackages' directive functionality seems to be broken when using inheritance
-
Background:
'frompackages' directive was added to backtrader back in 2017 (starting from release 1.9.30.x). It allows specifying the external packages to be imported only during the instantiation of the class (usually indicator). It comes very handily during the optimizations, reducing the serialization size of the objects. More on this here
Usage example:
class MyIndicator(bt.Indicator): frompackages = (('pandas', 'SomeFunction'),) lines = ('myline',) params = ( ('period', 50), ) def next(self): print('mylines[0]:', SomeFunction(self.lines.myline[0]))
Here the
SomeFunction
will be imported frompandas
package during the instantiation ofMyIndicator
and not earlier.Testcase:
In the same article, it was also claimed that "Both packages and frompackages support (multiple) inheritance". However, it seems to be not the case. Here a short test case:
import os import backtrader as bt class HurstExponentEx(bt.indicators.HurstExponent): def __init__(self): super(HurstExponentEx, self).__init__() def next(self): super(HurstExponentEx, self).__init__() print('test') class TheStrategy(bt.Strategy): def __init__(self): self.hurst = HurstExponentEx(self.data, lag_start=10,lag_end=500) def next(self): print('next') def runstrat(): cerebro = bt.Cerebro() cerebro.broker.set_cash(1000000) data_path = os.path.join(bt.__file__, '../../datas/yhoo-1996-2014.txt') data0 = bt.feeds.YahooFinanceCSVData(dataname=data_path) cerebro.adddata(data0) cerebro.addstrategy(TheStrategy) cerebro.run() cerebro.plot() if __name__ == '__main__': runstrat()
where the
HustExponent
class is defined in backtrader as:class HurstExponent(PeriodN): frompackages = ( ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')), ) ...
Unexpected behavior:
trying to run it (using python 3.6 in my case) will produce:
Traceback (most recent call last): File "test_frompackage.py", line 42, in <module> runstrat() File "test_frompackage.py", line 38, in runstrat cerebro.run() File "W:\backtrader\backtrader\cerebro.py", line 1182, in run runstrat = self.runstrategies(iterstrat) File "W:\backtrader\backtrader\cerebro.py", line 1275, in runstrategies strat = stratcls(*sargs, **skwargs) File "W:\backtrader\backtrader\metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "test_frompackage.py", line 17, in __init__ lag_end=500) File "W:\backtrader\backtrader\indicator.py", line 53, in __call__ return super(MetaIndicator, cls).__call__(*args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "test_frompackage.py", line 6, in __init__ super(HurstExponentEx, self).__init__() File "W:\backtrader\backtrader\indicators\hurst.py", line 82, in __init__ self.lags = asarray(range(lag_start, lag_end)) NameError: name 'asarray' is not defined
as could be seen
asarray
is a method that should have been imported fromnumpy
package upon instantiation ofHurstExponent
class.If we will directly use the
HurstExponent
class instead of ourHurstExponentEx
(which inherits fromHurstExponent
) - everything will work just fine.Analysis - TL;DR
Exploring this is a little bit exposes the problem with the implementation of
frompackages
inside backtrader.The magic code responsible for 'frompackages' directive handling could be found in backtrader\metabase.py file inside the
MetaParams.__new__
andMetaParams.donew
methods. Here the 'frompackages' directive is first examined recursively (in__new__
method ) and the appropriate packages are imported using__import__
function ( indonew
method)The problem is with the following code inside the
donew
method:def donew(cls, *args, **kwargs): clsmod = sys.modules[cls.__module__] . . <removed for clarity> . # import from specified packages - the 2nd part is a string or iterable for p, frompackage in cls.frompackages: if isinstance(frompackage, string_types): frompackage = (frompackage,) # make it a tuple for fp in frompackage: if isinstance(fp, (tuple, list)): fp, falias = fp else: fp, falias = fp, fp # assumed is string # complain "not string" without fp (unicode vs bytes) pmod = __import__(p, fromlist=[str(fp)]) pattr = getattr(pmod, fp) setattr(clsmod, falias, pattr)
The
cls
parameter to this function is the class that needs to be instantiated. In our case, this is our inherited classHustExponentEx
.So the
clsmod
variable will contain the module of our class - obviously the file thatHurstExponentEx
was defined in.The problem is with the last line of the above code:
setattr(clsmod, falias, pattr)
Here the
setattr
will introduce the imported names to the module - our module with inherited class - not the module the original base classHurstExponent
is defined in.And it is a problem!
Once the
HurstExponent
class will start executing and calling the supposedly imported functions - those will be looked in the module theHurstExponent
class is defined in - and will not be found, since those names are introduced in the module of our inherited class instead!FIX
The fix seems to be obvious. Introduce the imported names to the original base class module.
The Issue was opened in backtrader2 repo:
-
PR submitted: https://github.com/mementum/backtrader/pull/411