Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Bug loading multidata?

    General Discussion
    5
    8
    1860
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Alessandro Di Piazza last edited by

      Dear all,
      when I call .adddata using a loop data are not loaded correctly, but when I load one by one it works correctly.

      for example

      this doesn't work:

      df = pd.read_csv(SECURITYLISTFILE)
      
      for index, row in df.iterrows():
          securityfile = DATAFOLDER + "/" + row[0] + ".csv"
          if os.path.exists(securityfile):    
              # Create a Data Feed
              data = bt.feeds.GenericCSVData(
                  dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                  nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
              print ("added " + securityfile)
              cerebro.adddata(data)  # Add the Data Feed to Cerebro
      

      This works:

      data0 = bt.feeds.GenericCSVData(
                  dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                  nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
      cerebro.adddata(data0)
      
      data1 = bt.feeds.GenericCSVData(
                  dataname='/jupyter/data/ENI.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                  nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
      cerebro.adddata(data1)
      
      1 Reply Last reply Reply Quote 0
      • A
        ab_trader last edited by

        I had no problems to load data using the loop. Based on the 2 pieces of code it is hard to understand what was going wrong. Maybe your dataframe contains incorrect data, maybe path to each security is created incorrectly.

        • If my answer helped, hit reputation up arrow at lower right corner of the post.
        • Python Debugging With Pdb
        • New to python and bt - check this out
        1 Reply Last reply Reply Quote 0
        • L
          lssilva last edited by

          is the dataname=securityfile always unique? maybe it is the same name?

          1 Reply Last reply Reply Quote 0
          • M
            mpskowron last edited by

            Can you add more details please? Standard output - any exceptions thrown? What do you exactly mean by 'this doesn't work'

            1 Reply Last reply Reply Quote 0
            • A
              Alessandro Di Piazza last edited by

              In the first case, the strategy is not executed and I obtain this error, on plot.

              IndexErrorTraceback (most recent call last)
              <ipython-input-4-384fd2e61059> in <module>()
              215 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
              216
              --> 217 cerebro.plot(style ='candle') #style ='candle' line
              218 # an = result[0]
              219 # for each in an.analyzers:

              /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
              997 rfig = plotter.plot(strat, figid=si * 100,
              998 numfigs=numfigs, iplot=iplot,
              --> 999 start=start, end=end, use=use)
              1000 # pfillers=pfillers2)
              1001

              /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
              209 xtemp.append(dt)
              210
              --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
              212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
              213

              IndexError: list index out of range

              If I comment the .plot method, nothing happens, the strategy is not executed.

              Thank you

              1 Reply Last reply Reply Quote 0
              • P
                Paska Houso last edited by

                There is a very obvious reason.

                @Alessandro-Di-Piazza said in Bug loading multidata?:

                    securityfile = DATAFOLDER + "/" + row[0] + ".csv"
                    if os.path.exists(securityfile):   
                

                Those are names generated dynamically (names which are not being shown in your output)

                vs

                @Alessandro-Di-Piazza said in Bug loading multidata?:

                data0 = bt.feeds.GenericCSVData(
                            dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
                

                These names are hardcoded.

                Print out the value inside the variable securityfile and see if that matches your hardcoded names. The bet is clear: they won't and it will be clear that if os.path.exists(... is actually preventing you from entering the logic, hence: no data feeds are actually being loaded.

                Debugging 101.

                In any case

                DATAFOLDER + "/" + row[0] + ".csv"
                

                is bad practice. Use

                os.path.join(DATAFOLDER, row[0] + '.csv')
                

                to abstract the difference between operating systems.

                Note: os.path.exists will also be True for directories and not only for files.

                1 Reply Last reply Reply Quote 0
                • A
                  Alessandro Di Piazza last edited by

                  Thank you for your reply.
                  I changed the code using

                  securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')

                  This is the code now
                  if not os.path.exists(SECURITYLISTFILE):
                  raise Exception('No file security file list ' + SECURITYLISTFILE)

                  cerebro = bt.Cerebro()
                  
                  today = datetime.date.today()
                  df = pd.read_csv(SECURITYLISTFILE)
                  
                  for index, row in df.iterrows():
                      securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')
                    
                      # cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
                      if os.path.exists(securityfile):    
                          # Create a Data Feed
                          data = bt.feeds.GenericCSVData(
                              dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                              nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
                          print ("added " + securityfile)
                          print (data)
                          cerebro.adddata(data)  # Add the Data Feed to Cerebro
                  
                  cerebro.addstrategy(TestStrategy)    # Add a strategy
                  cerebro.broker.setcash(10000.0)
                  
                  cerebro.addsizer(FixedFractionalSizer, perc=50)
                  
                  
                  print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
                  result = cerebro.run(runonce=False,  stdstats=False)
                  print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
                  
                  cerebro.plot(style ='candle') #style ='candle' line
                  

                  and this the output

                  added /jupyter/data/ENEL.mi.csv
                  <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0cb33490>
                  added /jupyter/data/ENI.mi.csv
                  <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a2d0>
                  added /jupyter/data/STM.mi.csv
                  <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a950>
                  added /jupyter/data/G.mi.csv
                  <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79afd0>
                  Starting Portfolio Value: 10000.00
                  Final Portfolio Value: 10000.00

                  IndexErrorTraceback (most recent call last)
                  <ipython-input-9-cbbbf2d55bb7> in <module>()
                  218 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
                  219
                  --> 220 cerebro.plot(style ='candle') #style ='candle' line
                  221 # an = result[0]
                  222 # for each in an.analyzers:

                  /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
                  997 rfig = plotter.plot(strat, figid=si * 100,
                  998 numfigs=numfigs, iplot=iplot,
                  --> 999 start=start, end=end, use=use)
                  1000 # pfillers=pfillers2)
                  1001

                  /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
                  209 xtemp.append(dt)
                  210
                  --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
                  212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
                  213

                  IndexError: list index out of range

                  A 1 Reply Last reply Reply Quote 0
                  • A
                    Alessandro Di Piazza @Alessandro Di Piazza last edited by

                    Solved:
                    I modified the lines inside the loop

                            data.plotinfo.plot = False
                            cerebro.adddata(data, name=row[0])  # Add the Data Feed to Cerebro
                    

                    And i added in my strategy:

                    def prenext(self):
                        self.next()
                    
                    1 Reply Last reply Reply Quote 0
                    • 1 / 1
                    • First post
                      Last post
                    Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors