plot() not working when adding a second (Pandas) data feed

RandyT

Struggling recently to get plot to work. This seems to happen as soon as I add a second PandasData source.

The recreated this using code taken directly from the example code on https://www.backtrader.com/docu/quickstart/quickstart.html

The code taken from the example works fine. As soon as I add the second data source, I get the following error.

I've added the following bit of code to import this CSV formatted data:

    # Parse CSI ES data file
    data1path = os.path.join(modpath, '../datas/ES_0_I0B.CSV')
    data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
    data1 = bt.feeds.PandasData(dataname=data1frame)
    cerebro.adddata(data1, name='ES')

    # Add the Data Feed to Cerebro
    cerebro.adddata(data)
    cerebro.adddata(data1)

The file format of this data is as follows:

Date, Open, High, Low, Close, UnAdjusted Close, Volume, Open Interest, Total Volume, Total Open Interest^M
19980901,1069.50000000,1122.50000000,1052.75000000,1117.25000000,999.50000000,   29568,   14514,   30842,   16026^M
19980902,1117.50000000,1132.75000000,1103.75000000,1108.50000000,990.80000000,   22276,   14718,   22790,   16158^M
19980903,1109.00000000,1109.50000000,1082.75000000,1103.25000000,985.50000000,   22137,   14960,   22598,   16420^M
19980904,1102.75000000,1112.00000000,1072.25000000,1093.75000000,976.00000000,   18208,   15005,   18617,   16521^M

Below is the stacktrace I am getting:

Traceback (most recent call last):
  File "systems/stock-test.py", line 163, in <module>
    cerebro.plot()
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
    useplotly=useplotly)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
    self.plotdata(data, self.dplotsover[data])
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 656, in plotdata
    self.plotind(data, ind, subinds=self.dplotsover[ind], masterax=ax)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 407, in plotind
    plottedline = pltmethod(self.pinf.xdata, lplot, **plotkwargs)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/__init__.py", line 1819, in inner
    return func(ax, *args, **kwargs)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 1382, in plot
    for line in self._get_lines(*args, **kwargs):
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 381, in _grab_next_args
    for seg in self._plot_args(remaining, kwargs):
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 359, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 219, in _xy_from_xy
    raise ValueError("x and y must have same first dimension")
ValueError: x and y must have same first dimension

I've found mention on the web about the need to pass an numpy.array to matplotlib when seeing this error.

FWIW, if I add the following bit of code which as I understand tells backtrader not to plot this data, I get the following error:

data1.plotinfo.plot = False

Traceback (most recent call last):
  File "systems/stock-test.py", line 163, in <module>
    cerebro.plot()
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
    useplotly=useplotly)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
    self.plotdata(data, self.dplotsover[data])
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 578, in plotdata
    data, opens, highs, lows, closes, volumes, vollabel)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 504, in plotvolume
    ax = self.newaxis(data.volume, rowspan=rowspan)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 327, in newaxis
    rowspan=rowspan, sharex=self.pinf.sharex)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/pyplot.py", line 1275, in subplot2grid
    a = fig.add_subplot(subplotspec, **kwargs)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/figure.py", line 1005, in add_subplot
    a = subplot_class_factory(projection_class)(self, *args, **kwargs)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 70, in __init__
    self.update_params()
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 114, in update_params
    return_all=True)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/gridspec.py", line 429, in get_position
    figBottom = figBottoms[rowNum]
IndexError: list index out of range

backtrader

The second error should be corrected in the latest version.

The first is due to the out of sync bars between data0 and data1. In the default mode cerebro ruuning mode, runonce=True, the buffers are precreated when the datas are pre-loaded and the indicators are precalculated. If the original data sets are out of sync the length of the buffers will not match and matplotlib is very sensitive.

Try switching to runonce=False.

RandyT

runonce=False does in fact solve the plot() problem.

I'm running with version 1.9.19.105 and the error persists when setting plotinfo.plot = False for the dataset I want to suppress.

backtrader

The latest version was issued to exactly correct the problem shown there which shows up with plotvolume.

The question here would be if the data in question has volume (some data feeds have the field, but it is empty or set to zero)

RandyT

The datafeeds I am working with both have volume. However there is some other data there which may be confusing the discovery? There is a snippet above of the ES data I am using and I will provide a snippet below of the SPY data I am looking at. Which brings up another issue.

I've tried, using the following bit of code, to tell PandasData where to find the volume info in the feed and get a KeyError both using a numbered index or a name. Perhaps you can clear this up for me.

    # Parse ES data file
    data0path = os.path.join(modpath, args.data0)
    data0frame = pd.read_csv(data0path, skiprows=1, parse_dates=True, index_col=0)
    data0 = bt.feeds.PandasData(dataname=data0frame, volume=7, **dkwargs)
    cerebro.adddata(data0, name='ES')

    # Parse SPY data file
    data1path = os.path.join(modpath, args.data1)
    data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
    data1 = bt.feeds.PandasData(dataname=data1frame, volume=5, openinterest=None, **dkwargs)
    cerebro.adddata(data1, name='SPY')

Traceback (most recent call last):
  File "systems/system-dv-swing.py", line 215, in <module>
    runstrat()
  File "systems/system-dv-swing.py", line 104, in runstrat
    results = cerebro.run(runonce=False)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 809, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 870, in runstrategies
    data.preload()
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 370, in preload
    while self.load():
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 411, in load
    _loadret = self._load()
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feeds/pandafeed.py", line 221, in _load
    line[0] = self.p.dataname[colindex][self._idx]
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2059, in __getitem__
    return self._getitem_column(key)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2066, in _getitem_column
    return self._get_item_cache(key)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/generic.py", line 1386, in _get_item_cache
    values = self._data.get(item)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/internals.py", line 3541, in get
    loc = self.items.get_loc(item)
  File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/indexes/base.py", line 2136, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc (pandas/index.c:4164)
  File "pandas/index.pyx", line 161, in pandas.index.IndexEngine.get_loc (pandas/index.c:4028)
  File "pandas/src/hashtable_class_helper.pxi", line 732, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)
  File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)
KeyError: 7

SPY data:

Date, Open, High, Low, Close, Volume, UnAdjusted Close^M
20000403,150.13000000,151.25000000,148.69000000,151.25000000,   85082,151.25000000^M
20000404,151.75000000,153.00000000,141.39000000,150.13000000,  195855,150.12500000^M
20000405,147.88000000,150.81000000,147.63000000,149.19000000,   83872,149.18750000^M
20000406,150.25000000,151.69000000,149.00000000,150.48000000,   63785,150.48437500^M
20000407,151.56000000,152.13000000,150.50000000,151.44000000,   60236,151.43750000^M
20000410,151.75000000,153.11000000,150.31000000,150.84000000,   96242,150.84375000^M

I ultimately want to suppress the plot of the SPY data which I have tried by setting plotinfo which is where I am running into this other error.

backtrader

If you use the integer the right one would be: volume=4, because datetime is the index and doesn't count as a column in pandas. From the class:
- Values possible for datetime
  - None: the index contains the datetime
  - -1: no index, autodetect column
  - >= 0 or string: specific colum identifier
The same applies to volume=7 further above, which should be volume=6

In any case you may use: volume=' Volume' in both cases. Notice the leading whitespace in the name. This is because pandas doesn't trim/strip the whitespace when assigning names to the columns.

Note: stripping the names when comparing for autodetection could be considered.
There is in any case an error in PandasData, which should apply ix, when an integer is used like above. A patch has been pushed into the development to address this.
The tests for the new plotting functionality (which is actually no-plotting of datas) concentrated too much on plotting on another data. A patch has also been pushed into the development brach to address the no-plotting

RandyT

Great. I can confirm that the changes work with exception that using the string ' Volume' does not accomplish our goal. It still errors with KeyError: u' Volume'

Will work with the integer indexes for now.

Thanks again

backtrader

It didn't work for you because you are using skip_rows=1, not giving pandas a chance to process the headers.

Try the following arguments:

skiprows=0, parse_dates=True, index_col=0, header=0

And volume=' Volume' will work

RandyT

You are absolutely correct. Thanks for your help.