For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Multiprocessing error when extending bt.feed.DataBase



  • I've implemented a custom Pandas data feed class. I extend this class per dataframe, and use make_type() to dynamically creates new subclasses.

    This has worked fine in a single process, but when I use cerebro.optstrategy, it fails when pickling for multiprocessing:
    _pickle.PicklingError: Can't pickle <class 'backtrader.metabase.Price'>: attribute lookup Price on backtrader.metabase failed

    When I print the Feed object returned by Price.read() below, it actually says backtrader.metabase.Price. But obviously there's something going on in the meta classes that I don't understand.

    Any idea how to fix this?

    class PandasData(bt.feed.DataBase):
    
        @classmethod
        def read(cls, some_arg):
            raise NotImplementedError()
    
        @classmethod
        def make_type(cls, df, attrs):
            cols = ['datetime'] + df.columns.str.lower().tolist()
            attrs['params'] = (('name', cls.name), ('dataname', df)) + (tuple((c, i) for i, c in enumerate(cols)))
            attrs['lines'] = tuple(p for p, _ in attrs['params'])
            return type(cls.name, (cls,), attrs)
    
        def start(self):
            super(PandasData, self).start()
            self._rows = self.p.dataname.itertuples()
    
        def _load(self):
            try:
                row = next(self._rows)
            except StopIteration:
                return False
    
            # Set lines values here.
    
            return True
    
    
    class Price(PandasData):
        name = 'Price'
    
        @classmethod
        def read(cls, some_arg):
            df = read_price_df(some_arg)
            Feed = cls.make_type(df, {'some': 'attrs'})
            return Feed()
    

  • administrators

    @capitalglitch said in Multiprocessing error when extending bt.feed.DataBase:

    _pickle.PicklingError: Can't pickle <class 'backtrader.metabase.Price'>: attribute lookup Price on backtrader.metabase failed

    Pickling is known to fail if the class is not defined at the right module level expected by the pickling technology (which has additional limitations). That's why some people have developed some alternatives but these cannot be forced on the existing multiprocessing module.

    You will have to manually add your extended data feeds to the module reported by pickle (i.e.: where the lookup fails)



  • @backtrader Thanks. Pickling does indeed have its limitations, but this is some unfortunate entanglement of pickling and liberal metaclass usage.

    (Pdb) feed
    <backtrader.metabase.Price object at 0x7f5cb4138b70>
    (Pdb) getattr(bt.metabase, 'Price')
    *** AttributeError: module 'backtrader.metabase' has no attribute 'Price'
    

    Is there some other metabase somewhere?

    Setting it explicitly doesn't work either. Adding setattr(bt.metabase, feed.name, feed) results in:

    _pickle.PicklingError: Can't pickle <class 'backtrader.metabase.Price'>: it's not the same object as backtrader.metabase.Price
    

    Hmm. Thoughts?


  • administrators

    Seeing the errors doesn't suffice. See for example:

    @capitalglitch said in Multiprocessing error when extending bt.feed.DataBase:

        @classmethod
        def make_type(cls, df, attrs):
            cols = ['datetime'] + df.columns.str.lower().tolist()
            attrs['params'] = (('name', cls.name), ('dataname', df)) + (tuple((c, i) for i, c in enumerate(cols)))
            attrs['lines'] = tuple(p for p, _ in attrs['params'])
            return type(cls.name, (cls,), attrs)
    

    The dynamically generated classes are not being added to any module. And it is a mystery which actual name is being given to the class, because it seems derived from cls.name which is probably a pre-existing name.

    And it can't be told if class Price has been added to any module.

    It's all down to pickling and not related to the metaclasses.



  • If anyone else encounters this, my earlier attempt to add it to the metabase class was setattr(bt.metabase, feed.name, feed).

    Turns out, you need to add the Feed class and not the instance. So, my Price.read() method would return Feed and not return Feed(), and I added setattr(bt.metabase, Feed.name, Feed).

    I also encountered a pickling error when using Pandas's itertuples (which anyone using the Pandas direct feed will also encounter). Per this issue, the solution was to use .itertuples(name=None).


  • administrators

    @capitalglitch said in Multiprocessing error when extending bt.feed.DataBase:

    Turns out, you need to add the Feed class and not the instance

    Pickling looks into modules for classes definitions. Instances are not defined anywhere, they are instantiated.