Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    plot() not working when adding a second (Pandas) data feed

    General Code/Help
    4
    9
    3947
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RandyT
      RandyT last edited by

      Struggling recently to get plot to work. This seems to happen as soon as I add a second PandasData source.

      The recreated this using code taken directly from the example code on https://www.backtrader.com/docu/quickstart/quickstart.html

      The code taken from the example works fine. As soon as I add the second data source, I get the following error.

      I've added the following bit of code to import this CSV formatted data:

          # Parse CSI ES data file
          data1path = os.path.join(modpath, '../datas/ES_0_I0B.CSV')
          data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
          data1 = bt.feeds.PandasData(dataname=data1frame)
          cerebro.adddata(data1, name='ES')
      
          # Add the Data Feed to Cerebro
          cerebro.adddata(data)
          cerebro.adddata(data1)
      

      The file format of this data is as follows:

      Date, Open, High, Low, Close, UnAdjusted Close, Volume, Open Interest, Total Volume, Total Open Interest^M
      19980901,1069.50000000,1122.50000000,1052.75000000,1117.25000000,999.50000000,   29568,   14514,   30842,   16026^M
      19980902,1117.50000000,1132.75000000,1103.75000000,1108.50000000,990.80000000,   22276,   14718,   22790,   16158^M
      19980903,1109.00000000,1109.50000000,1082.75000000,1103.25000000,985.50000000,   22137,   14960,   22598,   16420^M
      19980904,1102.75000000,1112.00000000,1072.25000000,1093.75000000,976.00000000,   18208,   15005,   18617,   16521^M
      

      Below is the stacktrace I am getting:

      Traceback (most recent call last):
        File "systems/stock-test.py", line 163, in <module>
          cerebro.plot()
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
          useplotly=useplotly)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
          self.plotdata(data, self.dplotsover[data])
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 656, in plotdata
          self.plotind(data, ind, subinds=self.dplotsover[ind], masterax=ax)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 407, in plotind
          plottedline = pltmethod(self.pinf.xdata, lplot, **plotkwargs)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/__init__.py", line 1819, in inner
          return func(ax, *args, **kwargs)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 1382, in plot
          for line in self._get_lines(*args, **kwargs):
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 381, in _grab_next_args
          for seg in self._plot_args(remaining, kwargs):
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 359, in _plot_args
          x, y = self._xy_from_xy(x, y)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 219, in _xy_from_xy
          raise ValueError("x and y must have same first dimension")
      ValueError: x and y must have same first dimension
      
      

      I've found mention on the web about the need to pass an numpy.array to matplotlib when seeing this error.

      FWIW, if I add the following bit of code which as I understand tells backtrader not to plot this data, I get the following error:

      data1.plotinfo.plot = False
      
      Traceback (most recent call last):
        File "systems/stock-test.py", line 163, in <module>
          cerebro.plot()
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 659, in plot
          useplotly=useplotly)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 193, in plot
          self.plotdata(data, self.dplotsover[data])
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 578, in plotdata
          data, opens, highs, lows, closes, volumes, vollabel)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 504, in plotvolume
          ax = self.newaxis(data.volume, rowspan=rowspan)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/plot/plot.py", line 327, in newaxis
          rowspan=rowspan, sharex=self.pinf.sharex)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/pyplot.py", line 1275, in subplot2grid
          a = fig.add_subplot(subplotspec, **kwargs)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/figure.py", line 1005, in add_subplot
          a = subplot_class_factory(projection_class)(self, *args, **kwargs)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 70, in __init__
          self.update_params()
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 114, in update_params
          return_all=True)
        File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/matplotlib/gridspec.py", line 429, in get_position
          figBottom = figBottoms[rowNum]
      IndexError: list index out of range
      
      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        The second error should be corrected in the latest version.

        The first is due to the out of sync bars between data0 and data1. In the default mode cerebro ruuning mode, runonce=True, the buffers are precreated when the datas are pre-loaded and the indicators are precalculated. If the original data sets are out of sync the length of the buffers will not match and matplotlib is very sensitive.

        Try switching to runonce=False.

        1 Reply Last reply Reply Quote 0
        • RandyT
          RandyT last edited by

          runonce=False does in fact solve the plot() problem.

          I'm running with version 1.9.19.105 and the error persists when setting plotinfo.plot = False for the dataset I want to suppress.

          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            The latest version was issued to exactly correct the problem shown there which shows up with plotvolume.

            The question here would be if the data in question has volume (some data feeds have the field, but it is empty or set to zero)

            1 Reply Last reply Reply Quote 0
            • RandyT
              RandyT last edited by

              The datafeeds I am working with both have volume. However there is some other data there which may be confusing the discovery? There is a snippet above of the ES data I am using and I will provide a snippet below of the SPY data I am looking at. Which brings up another issue.

              I've tried, using the following bit of code, to tell PandasData where to find the volume info in the feed and get a KeyError both using a numbered index or a name. Perhaps you can clear this up for me.

                  # Parse ES data file
                  data0path = os.path.join(modpath, args.data0)
                  data0frame = pd.read_csv(data0path, skiprows=1, parse_dates=True, index_col=0)
                  data0 = bt.feeds.PandasData(dataname=data0frame, volume=7, **dkwargs)
                  cerebro.adddata(data0, name='ES')
              
                  # Parse SPY data file
                  data1path = os.path.join(modpath, args.data1)
                  data1frame = pd.read_csv(data1path, skiprows=1, parse_dates=True, index_col=0)
                  data1 = bt.feeds.PandasData(dataname=data1frame, volume=5, openinterest=None, **dkwargs)
                  cerebro.adddata(data1, name='SPY')
              
              Traceback (most recent call last):
                File "systems/system-dv-swing.py", line 215, in <module>
                  runstrat()
                File "systems/system-dv-swing.py", line 104, in runstrat
                  results = cerebro.run(runonce=False)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 809, in run
                  runstrat = self.runstrategies(iterstrat)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/cerebro.py", line 870, in runstrategies
                  data.preload()
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 370, in preload
                  while self.load():
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feed.py", line 411, in load
                  _loadret = self._load()
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/backtrader/feeds/pandafeed.py", line 221, in _load
                  line[0] = self.p.dataname[colindex][self._idx]
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2059, in __getitem__
                  return self._getitem_column(key)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/frame.py", line 2066, in _getitem_column
                  return self._get_item_cache(key)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/generic.py", line 1386, in _get_item_cache
                  values = self._data.get(item)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/core/internals.py", line 3541, in get
                  loc = self.items.get_loc(item)
                File "/home/inmate/.virtualenvs/backtrader/lib/python2.7/site-packages/pandas/indexes/base.py", line 2136, in get_loc
                  return self._engine.get_loc(self._maybe_cast_indexer(key))
                File "pandas/index.pyx", line 139, in pandas.index.IndexEngine.get_loc (pandas/index.c:4164)
                File "pandas/index.pyx", line 161, in pandas.index.IndexEngine.get_loc (pandas/index.c:4028)
                File "pandas/src/hashtable_class_helper.pxi", line 732, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)
                File "pandas/src/hashtable_class_helper.pxi", line 740, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)
              KeyError: 7
              

              SPY data:

              Date, Open, High, Low, Close, Volume, UnAdjusted Close^M
              20000403,150.13000000,151.25000000,148.69000000,151.25000000,   85082,151.25000000^M
              20000404,151.75000000,153.00000000,141.39000000,150.13000000,  195855,150.12500000^M
              20000405,147.88000000,150.81000000,147.63000000,149.19000000,   83872,149.18750000^M
              20000406,150.25000000,151.69000000,149.00000000,150.48000000,   63785,150.48437500^M
              20000407,151.56000000,152.13000000,150.50000000,151.44000000,   60236,151.43750000^M
              20000410,151.75000000,153.11000000,150.31000000,150.84000000,   96242,150.84375000^M
              

              I ultimately want to suppress the plot of the SPY data which I have tried by setting plotinfo which is where I am running into this other error.

              1 Reply Last reply Reply Quote 0
              • B
                backtrader administrators last edited by

                1. If you use the integer the right one would be: volume=4, because datetime is the index and doesn't count as a column in pandas. From the class:

                  • Values possible for datetime

                    • None: the index contains the datetime
                    • -1: no index, autodetect column
                    • >= 0 or string: specific colum identifier

                  The same applies to volume=7 further above, which should be volume=6

                  In any case you may use: volume=' Volume' in both cases. Notice the leading whitespace in the name. This is because pandas doesn't trim/strip the whitespace when assigning names to the columns.

                  Note: stripping the names when comparing for autodetection could be considered.

                2. There is in any case an error in PandasData, which should apply ix, when an integer is used like above. A patch has been pushed into the development to address this.

                3. The tests for the new plotting functionality (which is actually no-plotting of datas) concentrated too much on plotting on another data. A patch has also been pushed into the development brach to address the no-plotting

                1 Reply Last reply Reply Quote 0
                • RandyT
                  RandyT last edited by

                  Great. I can confirm that the changes work with exception that using the string ' Volume' does not accomplish our goal. It still errors with KeyError: u' Volume'

                  Will work with the integer indexes for now.

                  Thanks again

                  1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators last edited by

                    It didn't work for you because you are using skip_rows=1, not giving pandas a chance to process the headers.

                    Try the following arguments:

                    skiprows=0, parse_dates=True, index_col=0, header=0
                    

                    And volume=' Volume' will work

                    1 Reply Last reply Reply Quote 0
                    • RandyT
                      RandyT last edited by

                      You are absolutely correct. Thanks for your help.

                      1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      • First post
                        Last post
                      Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors