self.getposition(d) not equal to self.getpositionbyname(d._name)

Rahul Savani

I appreciate that a complete code example would be useful, but I am not sure I can extract one easily, though I will try to produce a MWE. Anyway, in the meantime, is there any obvious plausible explanation for the following (literally executed line after after in next()):

self.getposition(d)
--- Position Begin
- Size: -2469643.0
- Price: 7.51399993896484
- Price orig: 0.0
- Closed: 0
- Opened: -2469643.0
- Adjbase: 7.550000190734861
--- Position End
self.getpositionbyname(d._name)
--- Position Begin
- Size: 0
- Price: 0.0
- Price orig: 0.0
- Closed: 0
- Opened: 0
- Adjbase: None
--- Position End

There are 3 data feeds in this example. We loop through them in next(). The first of them has a legitimate position. The second one no position. The third one, should have no position, but self.getposition(d) returns the position for the first one, as shown above. But then, when using self.getpositionbyname(d._name) returns a 0 position as expected.

Any thoughts appreciated, as I am pretty baffled.

backtrader

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

is there any obvious plausible explanation for the following

Yes. You have a bug.

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

I appreciate that a complete code example would be useful,

It's the only thing that matters.

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

but self.getposition(d) returns the position for the first one

With no code, we don't know what d is holding, neither do you.

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

but I am not sure I can extract one easily

It's not about extracting anything. If you really think there is a bug in backtrader it will take around 20 lines to proof it. If those 20 lines produce the expected (good) result, you will have proof that you have a bug.

Rahul Savani

With no code, we don't know what d is holding, neither do you.

I understand; this should be somewhat more informative:

    def next(self):

        for i, d in enumerate(self.datas):

            if self.order[d._name]:
                continue

            print(self.getposition(d))
            print(self.getpositionbyname(d._name))

If you really think there is a bug in backtrader it will take around 20 lines to proof it. If those 20 lines produce the expected (good) result, you will have proof that you have a bug.

OK, sure, I will try to create a MWE. Of course, it's far from inconceivable that I have inadvertently introduced a bug, but since I have extended backtrader.Strategy and not extended/changed any code that should affect self.datas (beyond populating them of course), I am at a loss as to how the code I have written could create this phenomenon. Will keep digging, but any suggestions would be great.

And thanks! Really great platform that I have found extremely useful. Your work is much appreciated.

Rahul Savani

@backtrader

Here's an MWE:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import datetime  
import os.path 
import sys  
import backtrader as bt 
from backtrader.feeds import GenericCSVData

def create_dynamic_data_class():
    """
    Dynamic class creation 
    see: https://community.backtrader.com/topic/837/programmatically-extending-a-datafeed/3
    """
    # Add a lines to the inherited ones from the base class
    strategy_cols = ['IND']
    lines = tuple(strategy_cols)
    # add parameters starting from column 6 (date, OHLCV = 012345 come before)
    # NOTE: openinterest currently excluded
    params = tuple((name,6+i) for name,i in zip(strategy_cols,range(0,len(strategy_cols))))
    mydict = dict(lines=lines, params=params,)
    return type('DrCSVData', (GenericCSVData,), mydict)

def add_data(cerebro, symbols):

    DrCSVData = create_dynamic_data_class()

    datadir ='DATA' # relative reference to location of csv files
    csv_files = os.listdir(datadir)
    files = [s + '.csv' for s in symbols]

    for f in csv_files:
        if f not in files:
            continue

        fpath = os.path.join(datadir, f)

        # Create a Data Feed
        data = DrCSVData(dataname=fpath,
            dtformat='%Y-%m-%d',
            openinterest=-1,
            reverse=False)

        cerebro.adddata(data)

class DrStrategy(bt.Strategy):

    def __init__(self):

        self.order = {}

        for i, d in enumerate(self.datas):
            self.order[d._name] = None

    def notify_order(self, order):
        if order.status not in [order.Submitted, order.Accepted]:
            self.order[order.data._name] = None

    def next(self):

        for i, d in enumerate(self.datas):
            if self.order[d._name]:

                continue

            pos1 = self.getposition(d) 
            pos2 = self.getpositionbyname(d._name) 

            if pos1.size != pos2.size:
                print("PROBLEM FOUND: d._name %s" % d._name)
                print("self.getposition(d):\n", pos1)
                print("self.getposition(d._name):\n", pos2)
                sys.exit()

            if d.close[0] == 0:
                continue

            pos = self.getpositionbyname(d._name)
            if not pos:
                if d.IND[0] >= 0.99:
                    self.order[d._name] = self.buy(data=d)
                elif d.IND[0] <= 0.01:
                    self.order[d._name] = self.sell(data=d)
            else:
                self.order[d._name] = self.close(data=d)

if __name__ == "__main__":

    cerebro = bt.Cerebro(cheat_on_open=True, stdstats=False)
    cerebro.addstrategy(DrStrategy)
    add_data(cerebro, symbols=['S.1', 'S.2'])
    #add_data(cerebro, symbols=['S1', 'S2'])
    start_cash = 1000000000
    cerebro.broker.setcash(start_cash)
    results = cerebro.run()
    print("pnl:", cerebro.broker.getvalue() - start_cash)

I can provide the input data if useful, which are csv files lying in ./DATA,

 $ ls DATA
S.1.csv S.2.csv S1.csv  S2.csv

In creating this example, I spotted a crucial requirement for the problem, namely that the data name has to contain a dot, so the version with:

add_data(cerebro, symbols=['S.1', 'S.2'])

gives the problem, the version with

add_data(cerebro, symbols=['S1', 'S2'])

does not.

Sorry if I missed the restriction to not name datas with any dots in them; I have been extensively reading the docs but may have missed that.

P.S. obviously "S.1" looks like a silly name, but actually these were a large number of crosses, e.g., "USD.EUR" etc.

backtrader

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

In creating this example, I spotted a crucial requirement for the problem, namely that the data name has to contain a dot, so the version with:

Nothing within backtrader analyzes the name (it may contain dots, dashes, hashes ... anything ...)

@rahul-savani said in self.getposition(d) not equal to self.getpositionbyname(d._name):

        # Create a Data Feed
        data = DrCSVData(dataname=fpath,
            dtformat='%Y-%m-%d',
            openinterest=-1,
            reverse=False)

        cerebro.adddata(data)

In the lines above lies your crucial problem

See the reference for cerebro.adddata

Docs - Strategy

You are assigning no name to any of the data feeds.

My very humble opinion: when one is creating a short snippet to test a bug/behavior, one creates something simple.

Rahul Savani

You are assigning no name to any of the data feeds.

OK, thanks. That's easy to fix of course. But then what I don't understand:

In the code above we have not sett the name via adddata`` but he data feeds in self.datas **do** have names, and those names are used in the code above (e.g., withself.getpositionbyname(d._name)). The names returned by d._name``` correspond to the names of the csv files (without the ".csv" ending). I presumed that, having seen these names set automatically, everything was in order.

How are the names apparently set in the above code, but are somehow set differently (so as to not see the behaviour above) if I add the name as an explicit argument to adddata?

My very humble opinion: when one is creating a short snippet to test a bug/behavior, one creates something simple.

Yes, you are right of course. The reason for this not so simple code is that I started from something much much more complicated and chose to strip back to get something simpler (incrementally, while keeping the behaviour) rather than starting with something totally simple and trying to recreate the behaviour (since I didn't know exactly what was needed to create it).

Thanks!

Rahul Savani

In case it wasn't clear, my claim is that there is a bug because in my original code I made no explicit use of the _name property; I only used this when investigating the problem. The problem is that self.getposition(d) returned the position for the wrong data feed. I will do my best to come up with an MWE for that.

In the meantime, on my question in the last post, essentially: "How do I even have names if, as you point out I have not assigned them, and they are not assigned automatically?", the following (shorter ;-) example shows that _name is set automatically:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import os.path 
import sys  
import backtrader as bt 

def add_data(cerebro):
    files = ['DATA/S1.csv', 'DATA/S2.csv']
    for f in files:
        data = bt.feeds.GenericCSVData(
                dataname=f,
                dtformat=('%Y-%m-%d'),
                datetime=0,
                open=1,
                high=2,
                low=3,
                close=4,
                volume=5,
                openinterest=-1
        )
        cerebro.adddata(data)

class DrStrategy(bt.Strategy):

    def __init__(self):
        pass

    def notify_order(self, order):
        pass

    def next(self):
        for i, d in enumerate(self.datas):
            print(d._name)
        sys.exit()

if __name__ == "__main__":
    cerebro = bt.Cerebro()
    cerebro.addstrategy(DrStrategy)
    add_data(cerebro)
    results = cerebro.run()

which when run gives:

 $ python main_auto_names.py
S1
S2

Delving into the source by adding a print at the start of ```adddata`` as follows:

    def adddata(self, data, name=None):
        '''
        Adds a ``Data Feed`` instance to the mix.

        If ``name`` is not None it will be put into ``data._name`` which is
        meant for decoration/plotting purposes.
        '''
        print(data._name)

        if name is not None:
            data._name = name

we see that data already has its name set when passed to adddata. Indeed the auto-naming happens here:

class MetaCSVDataBase(DataBase.__class__):
    def dopostinit(cls, _obj, *args, **kwargs):
        # Before going to the base class to make sure it overrides the default
        if not _obj.p.name and not _obj._name:
            _obj._name, _ = os.path.splitext(os.path.basename(_obj.p.dataname))

        _obj, args, kwargs = \
            super(MetaCSVDataBase, cls).dopostinit(_obj, *args, **kwargs)

        return _obj, args, kwargs

In particular, the actual thing that drops the ".csv" is:

_obj._name, _ = os.path.splitext(os.path.basename(_obj.p.dataname))

It still doesn't explain the behaviour I was experiencing with getposition, so I will try to get a clean MWE for that.

Thanks, --Rahul.

Rahul Savani

So managed to get an MWE. The problem relates to 0 prices. The problem goes away if those prices as empty in the csv and treated by NaNs by backtrader. This is really as simple as I can get it, hopefully you agree:

Contents of input file S1.csv:

date,OPEN,CLOSE
1996-01-01,1.0,1.0
1996-01-02,1.0,1.0
1996-01-03,1.0,1.0

Contents of input file S2.csv:

date,OPEN,CLOSE
1996-01-01,0.0,0.0
1996-01-02,0.0,0.0
1996-01-03,0.0,0.0

Contents of input file S3.csv:

date,OPEN,CLOSE
1996-01-01,,
1996-01-02,,
1996-01-03,,

Source code for MWE:

import backtrader as bt 

def add_data(cerebro, symbols):

    for symbol in symbols:
        fname = 'DATA_S/%s.csv' % symbol
        data = bt.feeds.GenericCSVData(
                dataname=fname,
                dtformat=('%Y-%m-%d'),
                datetime=0,
                open=1,
                close=2,
                high=-1,
                low=-1,
                volume=-1,
                openinterest=-1
        )
        cerebro.adddata(data)

class DrStrategy(bt.Strategy):

    def next(self):
        print(self.data.datetime.date(0))
        for i, d in enumerate(self.datas):
            if i == 0:
                self.buy(data=d)
            print("Datas %s, position size: %s" % (i, self.getposition(d).size))

if __name__ == "__main__":

    cerebro = bt.Cerebro(cheat_on_open=True)

    cerebro.addstrategy(DrStrategy)

    # position reported for datas 1 erroneously same as for datas 0
    add_data(cerebro, symbols=['S1', 'S2'])

    # looks ok (missing entries in S.3 instead of 0s)
    # add_data(cerebro, symbols=['S1', 'S3']) 

    results = cerebro.run()

Output from running it with add_data(cerebro, symbols=['S1', 'S2']), where we get the problem:

1996-01-01
Datas 0, position size: 0
Datas 1, position size: 0
1996-01-02
Datas 0, position size: 1
Datas 1, position size: 1
1996-01-03
Datas 0, position size: 2
Datas 1, position size: 2

Output from running it with add_data(cerebro, symbols=['S1', 'S3']) , where we do not get the problem:

1996-01-01
Datas 0, position size: 0
Datas 1, position size: 0
1996-01-02
Datas 0, position size: 1
Datas 1, position size: 0
1996-01-03
Datas 0, position size: 2
Datas 1, position size: 0

Any feedback, comments, suggestions welcome. Happy to investigate anything related that might be useful. Of course, I will personally be careful to avoid 0 prices.

Thanks, --Rahul.