'frompackages' directive functionality seems to be broken when using inheritance
vladisld last edited by
'frompackages' directive was added to backtrader back in 2017 (starting from release 1.9.30.x). It allows specifying the external packages to be imported only during the instantiation of the class (usually indicator). It comes very handily during the optimizations, reducing the serialization size of the objects. More on this here
class MyIndicator(bt.Indicator): frompackages = (('pandas', 'SomeFunction'),) lines = ('myline',) params = ( ('period', 50), ) def next(self): print('mylines:', SomeFunction(self.lines.myline))
SomeFunctionwill be imported from
pandaspackage during the instantiation of
MyIndicatorand not earlier.
In the same article, it was also claimed that "Both packages and frompackages support (multiple) inheritance". However, it seems to be not the case. Here a short test case:
import os import backtrader as bt class HurstExponentEx(bt.indicators.HurstExponent): def __init__(self): super(HurstExponentEx, self).__init__() def next(self): super(HurstExponentEx, self).__init__() print('test') class TheStrategy(bt.Strategy): def __init__(self): self.hurst = HurstExponentEx(self.data, lag_start=10,lag_end=500) def next(self): print('next') def runstrat(): cerebro = bt.Cerebro() cerebro.broker.set_cash(1000000) data_path = os.path.join(bt.__file__, '../../datas/yhoo-1996-2014.txt') data0 = bt.feeds.YahooFinanceCSVData(dataname=data_path) cerebro.adddata(data0) cerebro.addstrategy(TheStrategy) cerebro.run() cerebro.plot() if __name__ == '__main__': runstrat()
HustExponentclass is defined in backtrader as:
class HurstExponent(PeriodN): frompackages = ( ('numpy', ('asarray', 'log10', 'polyfit', 'sqrt', 'std', 'subtract')), ) ...
trying to run it (using python 3.6 in my case) will produce:
Traceback (most recent call last): File "test_frompackage.py", line 42, in <module> runstrat() File "test_frompackage.py", line 38, in runstrat cerebro.run() File "W:\backtrader\backtrader\cerebro.py", line 1182, in run runstrat = self.runstrategies(iterstrat) File "W:\backtrader\backtrader\cerebro.py", line 1275, in runstrategies strat = stratcls(*sargs, **skwargs) File "W:\backtrader\backtrader\metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "test_frompackage.py", line 17, in __init__ lag_end=500) File "W:\backtrader\backtrader\indicator.py", line 53, in __call__ return super(MetaIndicator, cls).__call__(*args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "W:\backtrader\backtrader\metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "test_frompackage.py", line 6, in __init__ super(HurstExponentEx, self).__init__() File "W:\backtrader\backtrader\indicators\hurst.py", line 82, in __init__ self.lags = asarray(range(lag_start, lag_end)) NameError: name 'asarray' is not defined
as could be seen
asarrayis a method that should have been imported from
numpypackage upon instantiation of
If we will directly use the
HurstExponentclass instead of our
HurstExponentEx(which inherits from
HurstExponent) - everything will work just fine.
Analysis - TL;DR
Exploring this is a little bit exposes the problem with the implementation of
The magic code responsible for 'frompackages' directive handling could be found in backtrader\metabase.py file inside the
MetaParams.donewmethods. Here the 'frompackages' directive is first examined recursively (in
__new__method ) and the appropriate packages are imported using
__import__function ( in
The problem is with the following code inside the
def donew(cls, *args, **kwargs): clsmod = sys.modules[cls.__module__] . . <removed for clarity> . # import from specified packages - the 2nd part is a string or iterable for p, frompackage in cls.frompackages: if isinstance(frompackage, string_types): frompackage = (frompackage,) # make it a tuple for fp in frompackage: if isinstance(fp, (tuple, list)): fp, falias = fp else: fp, falias = fp, fp # assumed is string # complain "not string" without fp (unicode vs bytes) pmod = __import__(p, fromlist=[str(fp)]) pattr = getattr(pmod, fp) setattr(clsmod, falias, pattr)
clsparameter to this function is the class that needs to be instantiated. In our case, this is our inherited class
clsmodvariable will contain the module of our class - obviously the file that
HurstExponentExwas defined in.
The problem is with the last line of the above code:
setattr(clsmod, falias, pattr)
setattrwill introduce the imported names to the module - our module with inherited class - not the module the original base class
HurstExponentis defined in.
And it is a problem!
HurstExponentclass will start executing and calling the supposedly imported functions - those will be looked in the module the
HurstExponentclass is defined in - and will not be found, since those names are introduced in the module of our inherited class instead!
The fix seems to be obvious. Introduce the imported names to the original base class module.
The Issue was opened in backtrader2 repo:
vladisld last edited by
PR submitted: https://github.com/mementum/backtrader/pull/411