Bug loading multidata?



  • Dear all,
    when I call .adddata using a loop data are not loaded correctly, but when I load one by one it works correctly.

    for example

    this doesn't work:

    df = pd.read_csv(SECURITYLISTFILE)
    
    for index, row in df.iterrows():
        securityfile = DATAFOLDER + "/" + row[0] + ".csv"
        if os.path.exists(securityfile):    
            # Create a Data Feed
            data = bt.feeds.GenericCSVData(
                dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
            print ("added " + securityfile)
            cerebro.adddata(data)  # Add the Data Feed to Cerebro
    

    This works:

    data0 = bt.feeds.GenericCSVData(
                dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
    cerebro.adddata(data0)
    
    data1 = bt.feeds.GenericCSVData(
                dataname='/jupyter/data/ENI.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
    cerebro.adddata(data1)


  • I had no problems to load data using the loop. Based on the 2 pieces of code it is hard to understand what was going wrong. Maybe your dataframe contains incorrect data, maybe path to each security is created incorrectly.



  • is the dataname=securityfile always unique? maybe it is the same name?



  • Can you add more details please? Standard output - any exceptions thrown? What do you exactly mean by 'this doesn't work'



  • In the first case, the strategy is not executed and I obtain this error, on plot.

    IndexErrorTraceback (most recent call last)
    <ipython-input-4-384fd2e61059> in <module>()
    215 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    216
    --> 217 cerebro.plot(style ='candle') #style ='candle' line
    218 # an = result[0]
    219 # for each in an.analyzers:

    /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
    997 rfig = plotter.plot(strat, figid=si * 100,
    998 numfigs=numfigs, iplot=iplot,
    --> 999 start=start, end=end, use=use)
    1000 # pfillers=pfillers2)
    1001

    /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
    209 xtemp.append(dt)
    210
    --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
    212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
    213

    IndexError: list index out of range

    If I comment the .plot method, nothing happens, the strategy is not executed.

    Thank you



  • There is a very obvious reason.

    @Alessandro-Di-Piazza said in Bug loading multidata?:

        securityfile = DATAFOLDER + "/" + row[0] + ".csv"
        if os.path.exists(securityfile):   
    

    Those are names generated dynamically (names which are not being shown in your output)

    vs

    @Alessandro-Di-Piazza said in Bug loading multidata?:

    data0 = bt.feeds.GenericCSVData(
                dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
    

    These names are hardcoded.

    Print out the value inside the variable securityfile and see if that matches your hardcoded names. The bet is clear: they won't and it will be clear that if os.path.exists(... is actually preventing you from entering the logic, hence: no data feeds are actually being loaded.

    Debugging 101.

    In any case

    DATAFOLDER + "/" + row[0] + ".csv"
    

    is bad practice. Use

    os.path.join(DATAFOLDER, row[0] + '.csv')
    

    to abstract the difference between operating systems.

    Note: os.path.exists will also be True for directories and not only for files.



  • Thank you for your reply.
    I changed the code using

    securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')

    This is the code now
    if not os.path.exists(SECURITYLISTFILE):
    raise Exception('No file security file list ' + SECURITYLISTFILE)

    cerebro = bt.Cerebro()
    
    today = datetime.date.today()
    df = pd.read_csv(SECURITYLISTFILE)
    
    for index, row in df.iterrows():
        securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')
      
        # cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
        if os.path.exists(securityfile):    
            # Create a Data Feed
            data = bt.feeds.GenericCSVData(
                dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
                nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
            print ("added " + securityfile)
            print (data)
            cerebro.adddata(data)  # Add the Data Feed to Cerebro
    
    cerebro.addstrategy(TestStrategy)    # Add a strategy
    cerebro.broker.setcash(10000.0)
    
    cerebro.addsizer(FixedFractionalSizer, perc=50)
    
    
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
    result = cerebro.run(runonce=False,  stdstats=False)
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    
    cerebro.plot(style ='candle') #style ='candle' line
    

    and this the output

    added /jupyter/data/ENEL.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0cb33490>
    added /jupyter/data/ENI.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a2d0>
    added /jupyter/data/STM.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a950>
    added /jupyter/data/G.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79afd0>
    Starting Portfolio Value: 10000.00
    Final Portfolio Value: 10000.00

    IndexErrorTraceback (most recent call last)
    <ipython-input-9-cbbbf2d55bb7> in <module>()
    218 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    219
    --> 220 cerebro.plot(style ='candle') #style ='candle' line
    221 # an = result[0]
    222 # for each in an.analyzers:

    /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
    997 rfig = plotter.plot(strat, figid=si * 100,
    998 numfigs=numfigs, iplot=iplot,
    --> 999 start=start, end=end, use=use)
    1000 # pfillers=pfillers2)
    1001

    /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
    209 xtemp.append(dt)
    210
    --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
    212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
    213

    IndexError: list index out of range



  • Solved:
    I modified the lines inside the loop

            data.plotinfo.plot = False
            cerebro.adddata(data, name=row[0])  # Add the Data Feed to Cerebro
    

    And i added in my strategy:

    def prenext(self):
        self.next()

Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.