For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

plot() not working when adding a second (Pandas) data feed



  • Struggling recently to get plot to work. This seems to happen as soon as I add a second PandasData source.

    The recreated this using code taken directly from the example code on https://www.backtrader.com/docu/quickstart/quickstart.html

    The code taken from the example works fine. As soon as I add the second data source, I get the following error.

    I've added the following bit of code to import this CSV formatted data:

        # Parse CSI ES data file
        data1path = os.path.join(modpath, '../datas/ES_0_I0B.CSV')
        data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
        data1 = bt.feeds.PandasData(dataname=data1frame)
        cerebro.adddata(data1, name='ES')
    
        # Add the Data Feed to Cerebro
        cerebro.adddata(data)
        cerebro.adddata(data1)
    

    The file format of this data is as follows:

    Date, Open, High, Low, Close, UnAdjusted Close, Volume, Open Interest, Total Volume, Total Open Interest^M
    19980901,1069.50000000,1122.50000000,1052.75000000,1117.25000000,999.50000000,   29568,   14514,   30842,   16026^M
    19980902,1117.50000000,1132.75000000,1103.75000000,1108.50000000,990.80000000,   22276,   14718,   22790,   16158^M
    19980903,1109.00000000,1109.50000000,1082.75000000,1103.25000000,985.50000000,   22137,   14960,   22598,   16420^M
    19980904,1102.75000000,1112.00000000,1072.25000000,1093.75000000,976.00000000,   18208,   15005,   18617,   16521^M
    

    Below is the stacktrace I am getting:

    Traceback (most recent call last):
      File "systems/stock-test.py", line 163, in <module>
        cerebro.plot()
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
        useplotly=useplotly)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
        self.plotdata(data, self.dplotsover[data])
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 656, in plotdata
        self.plotind(data, ind, subinds=self.dplotsover[ind], masterax=ax)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 407, in plotind
        plottedline = pltmethod(self.pinf.xdata, lplot, **plotkwargs)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/__init__.py", line 1819, in inner
        return func(ax, *args, **kwargs)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 1382, in plot
        for line in self._get_lines(*args, **kwargs):
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 381, in _grab_next_args
        for seg in self._plot_args(remaining, kwargs):
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 359, in _plot_args
        x, y = self._xy_from_xy(x, y)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 219, in _xy_from_xy
        raise ValueError("x and y must have same first dimension")
    ValueError: x and y must have same first dimension
    
    

    I've found mention on the web about the need to pass an numpy.array to matplotlib when seeing this error.

    FWIW, if I add the following bit of code which as I understand tells backtrader not to plot this data, I get the following error:

    data1.plotinfo.plot = False
    
    Traceback (most recent call last):
      File "systems/stock-test.py", line 163, in <module>
        cerebro.plot()
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
        useplotly=useplotly)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
        self.plotdata(data, self.dplotsover[data])
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 578, in plotdata
        data, opens, highs, lows, closes, volumes, vollabel)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 504, in plotvolume
        ax = self.newaxis(data.volume, rowspan=rowspan)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 327, in newaxis
        rowspan=rowspan, sharex=self.pinf.sharex)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/pyplot.py", line 1275, in subplot2grid
        a = fig.add_subplot(subplotspec, **kwargs)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/figure.py", line 1005, in add_subplot
        a = subplot_class_factory(projection_class)(self, *args, **kwargs)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 70, in __init__
        self.update_params()
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 114, in update_params
        return_all=True)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/gridspec.py", line 429, in get_position
        figBottom = figBottoms[rowNum]
    IndexError: list index out of range
    

  • administrators

    The second error should be corrected in the latest version.

    The first is due to the out of sync bars between data0 and data1. In the default mode cerebro ruuning mode, runonce=True, the buffers are precreated when the datas are pre-loaded and the indicators are precalculated. If the original data sets are out of sync the length of the buffers will not match and matplotlib is very sensitive.

    Try switching to runonce=False.



  • runonce=False does in fact solve the plot() problem.

    I'm running with version 1.9.19.105 and the error persists when setting plotinfo.plot = False for the dataset I want to suppress.


  • administrators

    The latest version was issued to exactly correct the problem shown there which shows up with plotvolume.

    The question here would be if the data in question has volume (some data feeds have the field, but it is empty or set to zero)



  • The datafeeds I am working with both have volume. However there is some other data there which may be confusing the discovery? There is a snippet above of the ES data I am using and I will provide a snippet below of the SPY data I am looking at. Which brings up another issue.

    I've tried, using the following bit of code, to tell PandasData where to find the volume info in the feed and get a KeyError both using a numbered index or a name. Perhaps you can clear this up for me.

        # Parse ES data file
        data0path = os.path.join(modpath, args.data0)
        data0frame = pd.read_csv(data0path, skiprows=1, parse_dates=True, index_col=0)
        data0 = bt.feeds.PandasData(dataname=data0frame, volume=7, **dkwargs)
        cerebro.adddata(data0, name='ES')
    
        # Parse SPY data file
        data1path = os.path.join(modpath, args.data1)
        data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
        data1 = bt.feeds.PandasData(dataname=data1frame, volume=5, openinterest=None, **dkwargs)
        cerebro.adddata(data1, name='SPY')
    
    Traceback (most recent call last):
      File "systems/system-dv-swing.py", line 215, in <module>
        runstrat()
      File "systems/system-dv-swing.py", line 104, in runstrat
        results = cerebro.run(runonce=False)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 809, in run
        runstrat = self.runstrategies(iterstrat)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 870, in runstrategies
        data.preload()
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 370, in preload
        while self.load():
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 411, in load
        _loadret = self._load()
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feeds/pandafeed.py", line 221, in _load
        line[0] = self.p.dataname[colindex][self._idx]
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2059, in __getitem__
        return self._getitem_column(key)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2066, in _getitem_column
        return self._get_item_cache(key)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/generic.py", line 1386, in _get_item_cache
        values = self._data.get(item)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/internals.py", line 3541, in get
        loc = self.items.get_loc(item)
      File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/indexes/base.py", line 2136, in get_loc
        return self._engine.get_loc(self._maybe_cast_indexer(key))
      File "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc (pandas/index.c:4164)
      File "pandas/index.pyx", line 161, in pandas.index.IndexEngine.get_loc (pandas/index.c:4028)
      File "pandas/src/hashtable_class_helper.pxi", line 732, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)
      File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)
    KeyError: 7
    

    SPY data:

    Date, Open, High, Low, Close, Volume, UnAdjusted Close^M
    20000403,150.13000000,151.25000000,148.69000000,151.25000000,   85082,151.25000000^M
    20000404,151.75000000,153.00000000,141.39000000,150.13000000,  195855,150.12500000^M
    20000405,147.88000000,150.81000000,147.63000000,149.19000000,   83872,149.18750000^M
    20000406,150.25000000,151.69000000,149.00000000,150.48000000,   63785,150.48437500^M
    20000407,151.56000000,152.13000000,150.50000000,151.44000000,   60236,151.43750000^M
    20000410,151.75000000,153.11000000,150.31000000,150.84000000,   96242,150.84375000^M
    

    I ultimately want to suppress the plot of the SPY data which I have tried by setting plotinfo which is where I am running into this other error.


  • administrators

    1. If you use the integer the right one would be: volume=4, because datetime is the index and doesn't count as a column in pandas. From the class:

      • Values possible for datetime

        • None: the index contains the datetime
        • -1: no index, autodetect column
        • >= 0 or string: specific colum identifier

      The same applies to volume=7 further above, which should be volume=6

      In any case you may use: volume=' Volume' in both cases. Notice the leading whitespace in the name. This is because pandas doesn't trim/strip the whitespace when assigning names to the columns.

      Note: stripping the names when comparing for autodetection could be considered.

    2. There is in any case an error in PandasData, which should apply ix, when an integer is used like above. A patch has been pushed into the development to address this.

    3. The tests for the new plotting functionality (which is actually no-plotting of datas) concentrated too much on plotting on another data. A patch has also been pushed into the development brach to address the no-plotting



  • Great. I can confirm that the changes work with exception that using the string ' Volume' does not accomplish our goal. It still errors with KeyError: u' Volume'

    Will work with the integer indexes for now.

    Thanks again


  • administrators

    It didn't work for you because you are using skip_rows=1, not giving pandas a chance to process the headers.

    Try the following arguments:

    skiprows=0, parse_dates=True, index_col=0, header=0
    

    And volume=' Volume' will work



  • You are absolutely correct. Thanks for your help.