How to add extra column to lines?



  • Hi, all,

    I can successfully add one column, for example, PE, to YahooCSVData via using openinterest. However, there is no way for me to add another column to linetoken.

    Can anyone help me out here?

    Thank you in advance.


  • administrators

    However, there is no way for me to add another column to linetoken.

    This is unclear. linetokens is just a variable inside the generic functionality of csv derived data feeds which is passed to _loadline (the overriden method in subclasses) to let it parse and assign the parsed values to the tokens.

    If the question is understood, you would have to override the _loadline method of YahooCSVData to add your pe information.

    If wrongly understood, the best thing would be to share a code snippet here of what you are trying.



  • Hi,

    Thanks for your reply. I should have enclosed my log.

    First, I cloned backtrader/feeds/yahoo.py to be myyahoo.py as the following:

    class My_YahooFinanceCSVData(feed.CSVDataBase):
        '''
        Parses pre-downloaded Yahoo CSV Data Feeds (or locally generated if they
        comply to the Yahoo format)
    
        Specific parameters:
    
          - ``dataname``: The filename to parse or a file-like object
    
          - ``reverse`` (default: ``False``)
    
            It is assumed that locally stored files have already been reversed
            during the download process
    
          - ``adjclose`` (default: ``True``)
    
            Whether to use the dividend/split adjusted close and adjust all
            values according to it.
    
          - ``round`` (default: ``True``)
    
            Whether to round the values to a specific number of decimals after
            having adjusted the close
    
          - ``decimals`` (default: ``2``)
    
            Number of decimals to round to
        '''
        params = (
            ('reverse', False),
            ('adjclose', True),
            ('round', True),
            ('decimals', 2),
        )
    
        def start(self):
            super(My_YahooFinanceCSVData, self).start()
    
            if not self.params.reverse:
                return
    
            # Yahoo sends data in reverse order and the file is still unreversed
            dq = collections.deque()
            for line in self.f:
                dq.appendleft(line)
    
            f = io.StringIO(newline=None)
            f.writelines(dq)
            f.seek(0)
            self.f.close()
            self.f = f
    
        def _loadline(self, linetokens):
            i = itertools.count(0)
    
            dttxt = linetokens[next(i)]
            dt = date(int(dttxt[0:4]), int(dttxt[5:7]), int(dttxt[8:10]))
            dtnum = date2num(datetime.combine(dt, self.p.sessionend))
    
            self.lines.datetime[0] = dtnum
            o = float(linetokens[next(i)])
            h = float(linetokens[next(i)])
            l = float(linetokens[next(i)])
            c = float(linetokens[next(i)])
            v = float(linetokens[next(i)])
            #self.lines.openinterest[0] = 0.0
    
            if self.params.adjclose:
                adjustedclose = float(linetokens[next(i)])
                adjfactor = c / adjustedclose
                d = float(linetokens[next(i)]) # added for dividend
                pe = float(linetokens[next(i)]) # added for PE
    
    
                o /= adjfactor
                h /= adjfactor
                l /= adjfactor
                c = adjustedclose
                v /= adjfactor
    
                if self.p.round:
                    decimals = self.p.decimals
                    o = round(o, decimals)
                    h = round(h, decimals)
                    l = round(l, decimals)
                    c = round(c, decimals)
                    v = round(v, decimals)
                    d = round(d, decimals) 
                    pe = round(pe, decimals) 
    
            self.lines.open[0] = o
            self.lines.high[0] = h
            self.lines.low[0] = l
            self.lines.close[0] = c
            self.lines.volume[0] = v
            self.lines.openinterest[0] = d # new added
            self.lines.pe[0] = pe # new added
    
            return True
    

    Then, in my code, I add data to the strategy using:

        data = My_YahooFinanceCSVData(
            dataname=symbol + '.dat',
            fromdate=datetime.datetime(2015, 1, 1), 
            todate=datetime.datetime(2017, 3, 3),
            #nullvalue=0.0,
            )
        cerebro.adddata(data)
    

    Unfortunately, I got this error:

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-4-ed69e268d408> in <module>()
        124 cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
        125
    --> 126 results = cerebro.run()
    
    D:\Anaconda3\lib\site-packages\backtrader\cerebro.py in run(self, **kwargs)
        808             # let's skip process "spawning"
        809             for iterstrat in iterstrats:
    --> 810                 runstrat = self.runstrategies(iterstrat)
        811                 self.runstrats.append(runstrat)
        812         else:
    
    D:\Anaconda3\lib\site-packages\backtrader\cerebro.py in runstrategies(self, iterstrat, predata)
        871                 data._start()
        872                 if self._dopreload:
    --> 873                     data.preload()
        874
        875         for stratcls, sargs, skwargs in iterstrat:
    
    D:\Anaconda3\lib\site-packages\backtrader\feed.py in preload(self)
        657
        658     def preload(self):
    --> 659         while self.load():
        660             pass
        661
    
    D:\Anaconda3\lib\site-packages\backtrader\feed.py in load(self)
        452
        453             if not self._fromstack(stash=True):
    --> 454                 _loadret = self._load()
        455                 if not _loadret:  # no bar use force to make sure in exactbars
        456                     # the pointer is undone this covers especially (but not
    
    D:\Anaconda3\lib\site-packages\backtrader\feed.py in _load(self)
        679         line = line.rstrip('\n')
        680         linetokens = line.split(self.separator)
    --> 681         return self._loadline(linetokens)
        682
        683
    
    D:\Anaconda3\lib\site-packages\backtrader\feeds\myyahoo_2.py in _loadline(self, linetokens)
        130         self.lines.volume[0] = v
        131         self.lines.openinterest[0] = d # new added
    --> 132         self.lines.pe[0] = pe # new added
        133
        134         return True
    
    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'pe'.
    

    My data file looks like this:

    Date,Open,High,Low,Close,Volume,Adj Close,Yield,PE
    2014-07-11,9.39,9.42,9.38,9.41,168200.0,7.0631520000000005,0.09277364505844846,0.09277364505844846
    2014-07-14,9.42,9.43,9.4,9.4,168300.0,7.055646,0.09287234042553193,0.09287234042553193
    2014-07-15,9.42,9.42,9.39,9.4,150900.0,7.055646,0.09287234042553193,0.09287234042553193
    2014-07-16,9.37,9.39,9.34,9.34,228500.0,7.062452,0.10085653104925052,0.10085653104925052
    2014-07-17,9.36,9.36,9.29,9.31,181200.0,7.0397669999999986,0.09312567132116004,0.09312567132116004
    2014-07-18,9.35,9.36,9.28,9.28,238600.0,7.017082,0.09342672413793104,0.09342672413793104
    2014-07-21,9.28,9.32,9.24,9.24,202800.0,6.986836,0.09383116883116883,0.09383116883116883
    

    Hope this will clarify the issue.


  • administrators

    You have not added a pe line to the hierarchy. See Docs - Extending a Datafeed

    That's why it cannot be found as the attribute of the instance holding the lines (self.lines)



  • @backtrader Thank you alot.

    Following your doc, I can successfully overcame the line issue. However, when I use the same dataset to fetch new added PE information, I got nan value.

    Here is the extending datafeed part:

    class YahooFinanceCSVData_New(YahooFinanceCSVData):
    
        # Add a 'pe' line to the inherited ones from the base class
        lines = ('pe',
            'dividend',
            )
    
        # openinterest in GenericCSVData has index 7 ... add 1
        # add the parameter to the parameters inherited from the base class
        params = (('pe', 8),
            ('dividend', 9),
            )
    

    And in my strategy, it simply print something:

                print(
                    self.datas[i]._name, #ticker name 
                    current_position, # current postion size
                    round(position_weight, 2),  # current percentage of total portfolio
                    self.datas[i].close[0], # close price
                    self.datas[i].volume[0],
                    self.datas[i].openinterest[0],
                    self.datas[i].pe[0],
                    self.datas[i].dividend[0]
                    ) 
    

    However, I got nan for both PE and dividend part like this:

    0001 - 2016-09-19 - Position Size:     16932 - Value 1098268.15
    HIX 16932 0.1 6.71 163985.36 0.0 nan nan
    2016-09-19, Rebalanced portfolio to target weights at 2016-09-19
    0001 - 2016-12-19 - Position Size:     16204 - Value 1100391.60
    HIX 16204 0.1 6.84 335312.19 0.0 nan nan
    

    I can't figure out how to debug. Please shed some colors. Thank you very much.


  • administrators

    @littledou said in How to add extra column to lines?:

    # openinterest in GenericCSVData has index 7 ... add 1
    # add the parameter to the parameters inherited from the base class
    params = (('pe', 8),
        ('dividend', 9),
        )
    

    This is good for GenericCSVData. It will have no effect on YahooFinanceCSVData unless you subclass it, override _loadline and fill the values.

    That's the reason for the NaN: nothing is filling the values in pe and dividend



  • @backtrader Thank you very much. I can successfully add extra column to lines using GenericData.

    This is really great platform which I prefer to again zipline.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.