AttributeError: 'numpy.int64' object has no attribute 'lower' when using pandas with the --noheaders argument

weibudda

When I try the post "Pandas DataFeed Support" Link

When type in the command:

>(backtrader37) D:\Python\Jupyter\BackTrader\Test>python ./data-pandas.py --noheaders

It reports:

> (backtrader37) D:\Python\Jupyter\BackTrader\Test>python ./data-pandas.py --noheaders
--------------------------------------------------
                 1        2        3        4  5  6
0
2006-01-02  3578.73  3605.95  3578.73  3604.33  0  0
2006-01-03  3604.08  3638.42  3601.84  3614.34  0  0
2006-01-04  3615.23  3652.46  3615.23  3652.46  0  0
2006-01-05  3652.19  3661.65  3643.17  3650.24  0  0
2006-01-06  3650.54  3666.99  3647.66  3666.99  0  0
...             ...      ...      ...      ... .. ..
2006-12-21  4111.85  4125.27  4104.46  4112.10  0  0
2006-12-22  4109.86  4109.86  4072.62  4073.50  0  0
2006-12-27  4079.70  4134.86  4079.70  4134.86  0  0
2006-12-28  4137.44  4142.06  4125.14  4130.66  0  0
2006-12-29  4130.12  4142.01  4119.94  4119.94  0  0
[255 rows x 6 columns]
--------------------------------------------------
Traceback (most recent call last):
  File "./data-pandas.py", line 92, in <module>
    runstrat()
  File "./data-pandas.py", line 71, in runstrat
    cerebro.run()
  File "C:\Users\WEI\.conda\envs\backtrader37\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "C:\Users\WEI\.conda\envs\backtrader37\lib\site-packages\backtrader\cerebro.py", line 1210, in runstrategies
    data._start()
  File "C:\Users\WEI\.conda\envs\backtrader37\lib\site-packages\backtrader\feed.py", line 203, in _start
    self.start()
  File "C:\Users\WEI\.conda\envs\backtrader37\lib\site-packages\backtrader\feeds\pandafeed.py", line 212, in start
    colnames = [x.lower() for x in self.p.dataname.columns.values]
  File "C:\Users\WEI\.conda\envs\backtrader37\lib\site-packages\backtrader\feeds\pandafeed.py", line 212, in <listcomp>
    colnames = [x.lower() for x in self.p.dataname.columns.values]
AttributeError: 'numpy.int64' object has no attribute 'lower'

Without the argument of --noheaders：

>(backtrader37) D:\Python\Jupyter\BackTrader\Test>python ./data-pandas.py
--------------------------------------------------
               Open     High      Low    Close  Volume  OpenInterest
Date
2006-01-02  3578.73  3605.95  3578.73  3604.33       0             0
2006-01-03  3604.08  3638.42  3601.84  3614.34       0             0
2006-01-04  3615.23  3652.46  3615.23  3652.46       0             0
2006-01-05  3652.19  3661.65  3643.17  3650.24       0             0
2006-01-06  3650.54  3666.99  3647.66  3666.99       0             0
...             ...      ...      ...      ...     ...           ...
2006-12-21  4111.85  4125.27  4104.46  4112.10       0             0
2006-12-22  4109.86  4109.86  4072.62  4073.50       0             0
2006-12-27  4079.70  4134.86  4079.70  4134.86       0             0
2006-12-28  4137.44  4142.06  4125.14  4130.66       0             0
2006-12-29  4130.12  4142.01  4119.94  4119.94       0             0

[255 rows x 6 columns]
--------------------------------------------------

Everything is OK! But from the output, we can see that, with the --noheaders argument, the pandas data is successfully loaded, but Cerebro cannot deal with it.

What's the problem?

Appendix: data-pandas.py

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import argparse
import backtrader as bt
import backtrader.feeds as btfeeds
import pandas
def runstrat():
    args = parse_args()

    # Create a cerebro entity
    cerebro = bt.Cerebro(stdstats=False)

    # Add a strategy
    cerebro.addstrategy(bt.Strategy)

    # Get a pandas dataframe
    datapath = ('../../datas/2006-day-001.txt')

    # Simulate the header row isn't there if noheaders requested
    skiprows = 1 if args.noheaders else 0
    header = None if args.noheaders else 0

    dataframe = pandas.read_csv(
        datapath,
        skiprows=skiprows,
        header=header,
        # parse_dates=[0],
        parse_dates=True,
        index_col=0,
    )

    if not args.noprint:
        print('--------------------------------------------------')
        print(dataframe)
        print('--------------------------------------------------')

    # Pass it to the backtrader datafeed and add it to the cerebro
    data = bt.feeds.PandasData(dataname=dataframe,
                               # datetime='Date',
                               nocase=True,
                               )

    cerebro.adddata(data)

    # Run over everything
    cerebro.run()

    # Plot the result
    cerebro.plot(style='bar')


def parse_args():
    parser = argparse.ArgumentParser(
        description='Pandas test script')

    parser.add_argument('--noheaders', action='store_true', default=False,
                        required=False,
                        help='Do not use header rows')

    parser.add_argument('--noprint', action='store_true', default=False,
                        help='Print the dataframe')

    return parser.parse_args()


if __name__ == '__main__':
    runstrat()

vladisld

It seems that PandasData column index auto-detection mechanism is broken in case the input dataframe contains no column names ( using indexes instead).

I suspect the following commit has introduced the regression:

05051890efe527ee22919d354038b4dfc1ffe7ca: Rework ix -> iloc pull request and autodetection algorithm in PandasData

Opening the issue to track this:

https://github.com/backtrader2/backtrader/issues/43