For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
2019-10-02: The community is currently in read-only mode

self.getposition(d) not equal to self.getpositionbyname(d._name)



  • I appreciate that a complete code example would be useful, but I am not sure I can extract one easily, though I will try to produce a MWE. Anyway, in the meantime, is there any obvious plausible explanation for the following (literally executed line after after in next()):

    self.getposition(d)
    --- Position Begin
    - Size: -2469643.0
    - Price: 7.51399993896484
    - Price orig: 0.0
    - Closed: 0
    - Opened: -2469643.0
    - Adjbase: 7.550000190734861
    --- Position End
    self.getpositionbyname(d._name)
    --- Position Begin
    - Size: 0
    - Price: 0.0
    - Price orig: 0.0
    - Closed: 0
    - Opened: 0
    - Adjbase: None
    --- Position End
    

    There are 3 data feeds in this example. We loop through them in next(). The first of them has a legitimate position. The second one no position. The third one, should have no position, but self.getposition(d) returns the position for the first one, as shown above. But then, when using self.getpositionbyname(d._name) returns a 0 position as expected.

    Any thoughts appreciated, as I am pretty baffled.


  • administrators

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

    is there any obvious plausible explanation for the following

    Yes. You have a bug.

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

    I appreciate that a complete code example would be useful,

    It's the only thing that matters.

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

    but self.getposition(d) returns the position for the first one

    With no code, we don't know what d is holding, neither do you.

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

    but I am not sure I can extract one easily

    It's not about extracting anything. If you really think there is a bug in backtrader it will take around 20 lines to proof it. If those 20 lines produce the expected (good) result, you will have proof that you have a bug.



  • With no code, we don't know what d is holding, neither do you.

    I understand; this should be somewhat more informative:

        def next(self):
    
            for i, d in enumerate(self.datas):
    
                if self.order[d._name]:
                    continue
    
                print(self.getposition(d))
                print(self.getpositionbyname(d._name))
    

    If you really think there is a bug in backtrader it will take around 20 lines to proof it. If those 20 lines produce the expected (good) result, you will have proof that you have a bug.

    OK, sure, I will try to create a MWE. Of course, it's far from inconceivable that I have inadvertently introduced a bug, but since I have extended backtrader.Strategy and not extended/changed any code that should affect self.datas (beyond populating them of course), I am at a loss as to how the code I have written could create this phenomenon. Will keep digging, but any suggestions would be great.

    And thanks! Really great platform that I have found extremely useful. Your work is much appreciated.



  • @backtrader

    Here's an MWE:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import datetime  
    import os.path 
    import sys  
    import backtrader as bt 
    from backtrader.feeds import GenericCSVData
    
    def create_dynamic_data_class():
        """
        Dynamic class creation 
        see: https://community.backtrader.com/topic/837/programmatically-extending-a-datafeed/3
        """
        # Add a lines to the inherited ones from the base class
        strategy_cols = ['IND']
        lines = tuple(strategy_cols)
        # add parameters starting from column 6 (date, OHLCV = 012345 come before)
        # NOTE: openinterest currently excluded
        params = tuple((name,6+i) for name,i in zip(strategy_cols,range(0,len(strategy_cols))))
        mydict = dict(lines=lines, params=params,)
        return type('DrCSVData', (GenericCSVData,), mydict)
    
    def add_data(cerebro, symbols):
    
        DrCSVData = create_dynamic_data_class()
    
        datadir ='DATA' # relative reference to location of csv files
        csv_files = os.listdir(datadir)
        files = [s + '.csv' for s in symbols]
    
        for f in csv_files:
            if f not in files:
                continue
    
            fpath = os.path.join(datadir, f)
    
            # Create a Data Feed
            data = DrCSVData(dataname=fpath,
                dtformat='%Y-%m-%d',
                openinterest=-1,
                reverse=False)
    
            cerebro.adddata(data)
    
    class DrStrategy(bt.Strategy):
    
        def __init__(self):
    
            self.order = {}
    
            for i, d in enumerate(self.datas):
                self.order[d._name] = None
    
        def notify_order(self, order):
            if order.status not in [order.Submitted, order.Accepted]:
                self.order[order.data._name] = None
    
        def next(self):
    
            for i, d in enumerate(self.datas):
                if self.order[d._name]:
    
                    continue
    
                pos1 = self.getposition(d) 
                pos2 = self.getpositionbyname(d._name) 
    
                if pos1.size != pos2.size:
                    print("PROBLEM FOUND: d._name %s" % d._name)
                    print("self.getposition(d):\n", pos1)
                    print("self.getposition(d._name):\n", pos2)
                    sys.exit()
    
                if d.close[0] == 0:
                    continue
    
                pos = self.getpositionbyname(d._name)
                if not pos:
                    if d.IND[0] >= 0.99:
                        self.order[d._name] = self.buy(data=d)
                    elif d.IND[0] <= 0.01:
                        self.order[d._name] = self.sell(data=d)
                else:
                    self.order[d._name] = self.close(data=d)
    
    if __name__ == "__main__":
    
        cerebro = bt.Cerebro(cheat_on_open=True, stdstats=False)
        cerebro.addstrategy(DrStrategy)
        add_data(cerebro, symbols=['S.1', 'S.2'])
        #add_data(cerebro, symbols=['S1', 'S2'])
        start_cash = 1000000000
        cerebro.broker.setcash(start_cash)
        results = cerebro.run()
        print("pnl:", cerebro.broker.getvalue() - start_cash)
    

    I can provide the input data if useful, which are csv files lying in ./DATA,

     $ ls DATA
    S.1.csv S.2.csv S1.csv  S2.csv
    

    In creating this example, I spotted a crucial requirement for the problem, namely that the data name has to contain a dot, so the version with:

    add_data(cerebro, symbols=['S.1', 'S.2'])
    

    gives the problem, the version with

    add_data(cerebro, symbols=['S1', 'S2'])
    

    does not.

    Sorry if I missed the restriction to not name datas with any dots in them; I have been extensively reading the docs but may have missed that.

    P.S. obviously "S.1" looks like a silly name, but actually these were a large number of crosses, e.g., "USD.EUR" etc.


  • administrators

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

    In creating this example, I spotted a crucial requirement for the problem, namely that the data name has to contain a dot, so the version with:

    Nothing within backtrader analyzes the name (it may contain dots, dashes, hashes ... anything ...)

    @rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

            # Create a Data Feed
            data = DrCSVData(dataname=fpath,
                dtformat='%Y-%m-%d',
                openinterest=-1,
                reverse=False)
    
            cerebro.adddata(data)
    

    In the lines above lies your crucial problem

    See the reference for cerebro.adddata

    You are assigning no name to any of the data feeds.

    My very humble opinion: when one is creating a short snippet to test a bug/behavior, one creates something simple.



  • You are assigning no name to any of the data feeds.

    OK, thanks. That's easy to fix of course. But then what I don't understand:

    In the code above we have not sett the name via adddata`` but he data feeds inself.datas**do** have names, and those names are used in the code above (e.g., withself.getpositionbyname(d._name)). The names returned byd._name``` correspond to the names of the csv files (without the ".csv" ending). I presumed that, having seen these names set automatically, everything was in order.

    How are the names apparently set in the above code, but are somehow set differently (so as to not see the behaviour above) if I add the name as an explicit argument to adddata?

    My very humble opinion: when one is creating a short snippet to test a bug/behavior, one creates something simple.

    Yes, you are right of course. The reason for this not so simple code is that I started from something much much more complicated and chose to strip back to get something simpler (incrementally, while keeping the behaviour) rather than starting with something totally simple and trying to recreate the behaviour (since I didn't know exactly what was needed to create it).

    Thanks!



  • In case it wasn't clear, my claim is that there is a bug because in my original code I made no explicit use of the _name property; I only used this when investigating the problem. The problem is that self.getposition(d) returned the position for the wrong data feed. I will do my best to come up with an MWE for that.

    In the meantime, on my question in the last post, essentially: "How do I even have names if, as you point out I have not assigned them, and they are not assigned automatically?", the following (shorter ;-) example shows that _name is set automatically:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import os.path 
    import sys  
    import backtrader as bt 
    
    def add_data(cerebro):
        files = ['DATA/S1.csv', 'DATA/S2.csv']
        for f in files:
            data = bt.feeds.GenericCSVData(
                    dataname=f,
                    dtformat=('%Y-%m-%d'),
                    datetime=0,
                    open=1,
                    high=2,
                    low=3,
                    close=4,
                    volume=5,
                    openinterest=-1
            )
            cerebro.adddata(data)
    
    class DrStrategy(bt.Strategy):
    
        def __init__(self):
            pass
    
        def notify_order(self, order):
            pass
    
        def next(self):
            for i, d in enumerate(self.datas):
                print(d._name)
            sys.exit()
    
    if __name__ == "__main__":
        cerebro = bt.Cerebro()
        cerebro.addstrategy(DrStrategy)
        add_data(cerebro)
        results = cerebro.run()
    

    which when run gives:

     $ python main_auto_names.py
    S1
    S2
    

    Delving into the source by adding a print at the start of ```adddata`` as follows:

        def adddata(self, data, name=None):
            '''
            Adds a ``Data Feed`` instance to the mix.
    
            If ``name`` is not None it will be put into ``data._name`` which is
            meant for decoration/plotting purposes.
            '''
            print(data._name)
    
            if name is not None:
                data._name = name
    

    we see that data already has its name set when passed to adddata. Indeed the auto-naming happens here:

    class MetaCSVDataBase(DataBase.__class__):
        def dopostinit(cls, _obj, *args, **kwargs):
            # Before going to the base class to make sure it overrides the default
            if not _obj.p.name and not _obj._name:
                _obj._name, _ = os.path.splitext(os.path.basename(_obj.p.dataname))
    
            _obj, args, kwargs = \
                super(MetaCSVDataBase, cls).dopostinit(_obj, *args, **kwargs)
    
            return _obj, args, kwargs
    

    In particular, the actual thing that drops the ".csv" is:

    _obj._name, _ = os.path.splitext(os.path.basename(_obj.p.dataname))
    

    It still doesn't explain the behaviour I was experiencing with getposition, so I will try to get a clean MWE for that.

    Thanks, --Rahul.



  • So managed to get an MWE. The problem relates to 0 prices. The problem goes away if those prices as empty in the csv and treated by NaNs by backtrader. This is really as simple as I can get it, hopefully you agree:

    Contents of input file S1.csv:

    date,OPEN,CLOSE
    1996-01-01,1.0,1.0
    1996-01-02,1.0,1.0
    1996-01-03,1.0,1.0

    Contents of input file S2.csv:

    date,OPEN,CLOSE
    1996-01-01,0.0,0.0
    1996-01-02,0.0,0.0
    1996-01-03,0.0,0.0

    Contents of input file S3.csv:

    date,OPEN,CLOSE
    1996-01-01,,
    1996-01-02,,
    1996-01-03,,

    Source code for MWE:

    import backtrader as bt 
    
    def add_data(cerebro, symbols):
    
        for symbol in symbols:
            fname = 'DATA_S/%s.csv' % symbol
            data = bt.feeds.GenericCSVData(
                    dataname=fname,
                    dtformat=('%Y-%m-%d'),
                    datetime=0,
                    open=1,
                    close=2,
                    high=-1,
                    low=-1,
                    volume=-1,
                    openinterest=-1
            )
            cerebro.adddata(data)
    
    class DrStrategy(bt.Strategy):
    
        def next(self):
            print(self.data.datetime.date(0))
            for i, d in enumerate(self.datas):
                if i == 0:
                    self.buy(data=d)
                print("Datas %s, position size: %s" % (i, self.getposition(d).size))
    
    if __name__ == "__main__":
    
        cerebro = bt.Cerebro(cheat_on_open=True)
    
        cerebro.addstrategy(DrStrategy)
    
        # position reported for datas 1 erroneously same as for datas 0
        add_data(cerebro, symbols=['S1', 'S2'])
    
        # looks ok (missing entries in S.3 instead of 0s)
        # add_data(cerebro, symbols=['S1', 'S3']) 
    
        results = cerebro.run()
    

    Output from running it with add_data(cerebro, symbols=['S1', 'S2']), where we get the problem:

    1996-01-01
    Datas 0, position size: 0
    Datas 1, position size: 0
    1996-01-02
    Datas 0, position size: 1
    Datas 1, position size: 1
    1996-01-03
    Datas 0, position size: 2
    Datas 1, position size: 2
    

    Output from running it with add_data(cerebro, symbols=['S1', 'S3']), where we do not get the problem:

    1996-01-01
    Datas 0, position size: 0
    Datas 1, position size: 0
    1996-01-02
    Datas 0, position size: 1
    Datas 1, position size: 0
    1996-01-03
    Datas 0, position size: 2
    Datas 1, position size: 0
    

    Any feedback, comments, suggestions welcome. Happy to investigate anything related that might be useful. Of course, I will personally be careful to avoid 0 prices.

    Thanks, --Rahul.


Log in to reply
 

});