Bug loading multidata?

Alessandro Di Piazza

Dear all,
when I call .adddata using a loop data are not loaded correctly, but when I load one by one it works correctly.

for example

this doesn't work:

df = pd.read_csv(SECURITYLISTFILE)

for index, row in df.iterrows():
    securityfile = DATAFOLDER + "/" + row[0] + ".csv"
    if os.path.exists(securityfile):    
        # Create a Data Feed
        data = bt.feeds.GenericCSVData(
            dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
        print ("added " + securityfile)
        cerebro.adddata(data)  # Add the Data Feed to Cerebro

This works:

data0 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
cerebro.adddata(data0)

data1 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENI.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
cerebro.adddata(data1)

ab_trader

I had no problems to load data using the loop. Based on the 2 pieces of code it is hard to understand what was going wrong. Maybe your dataframe contains incorrect data, maybe path to each security is created incorrectly.

lssilva

is the dataname=securityfile always unique? maybe it is the same name?

mpskowron

Can you add more details please? Standard output - any exceptions thrown? What do you exactly mean by 'this doesn't work'

Alessandro Di Piazza

In the first case, the strategy is not executed and I obtain this error, on plot.

IndexErrorTraceback (most recent call last)
<ipython-input-4-384fd2e61059> in <module>()
215 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
216
--> 217 cerebro.plot(style ='candle') #style ='candle' line
218 # an = result[0]
219 # for each in an.analyzers:

/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
997 rfig = plotter.plot(strat, figid=si * 100,
998 numfigs=numfigs, iplot=iplot,
--> 999 start=start, end=end, use=use)
1000 # pfillers=pfillers2)
1001

/usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
209 xtemp.append(dt)
210
--> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
213

IndexError: list index out of range

If I comment the .plot method, nothing happens, the strategy is not executed.

Thank you

Paska Houso

There is a very obvious reason.

@Alessandro-Di-Piazza said in Bug loading multidata?:

    securityfile = DATAFOLDER + "/" + row[0] + ".csv"
    if os.path.exists(securityfile):

Those are names generated dynamically (names which are not being shown in your output)

vs

@Alessandro-Di-Piazza said in Bug loading multidata?:

data0 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)

These names are hardcoded.

Print out the value inside the variable securityfile and see if that matches your hardcoded names. The bet is clear: they won't and it will be clear that if os.path.exists(... is actually preventing you from entering the logic, hence: no data feeds are actually being loaded.

Debugging 101.

In any case

DATAFOLDER + "/" + row[0] + ".csv"

is bad practice. Use

os.path.join(DATAFOLDER, row[0] + '.csv')

to abstract the difference between operating systems.

Note: os.path.exists will also be True for directories and not only for files.

Alessandro Di Piazza

Thank you for your reply.
I changed the code using

securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')

This is the code now
if not os.path.exists(SECURITYLISTFILE):
raise Exception('No file security file list ' + SECURITYLISTFILE)

cerebro = bt.Cerebro()

today = datetime.date.today()
df = pd.read_csv(SECURITYLISTFILE)

for index, row in df.iterrows():
    securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')
  
    # cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
    if os.path.exists(securityfile):    
        # Create a Data Feed
        data = bt.feeds.GenericCSVData(
            dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
        print ("added " + securityfile)
        print (data)
        cerebro.adddata(data)  # Add the Data Feed to Cerebro

cerebro.addstrategy(TestStrategy)    # Add a strategy
cerebro.broker.setcash(10000.0)

cerebro.addsizer(FixedFractionalSizer, perc=50)


print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
result = cerebro.run(runonce=False,  stdstats=False)
print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.plot(style ='candle') #style ='candle' line

and this the output

added /jupyter/data/ENEL.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0cb33490>
added /jupyter/data/ENI.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a2d0>
added /jupyter/data/STM.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a950>
added /jupyter/data/G.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79afd0>
Starting Portfolio Value: 10000.00
Final Portfolio Value: 10000.00

IndexErrorTraceback (most recent call last)
<ipython-input-9-cbbbf2d55bb7> in <module>()
218 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
219
--> 220 cerebro.plot(style ='candle') #style ='candle' line
221 # an = result[0]
222 # for each in an.analyzers:

/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
997 rfig = plotter.plot(strat, figid=si * 100,
998 numfigs=numfigs, iplot=iplot,
--> 999 start=start, end=end, use=use)
1000 # pfillers=pfillers2)
1001

/usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
209 xtemp.append(dt)
210
--> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
213

IndexError: list index out of range

Alessandro Di Piazza

Solved:
I modified the lines inside the loop

        data.plotinfo.plot = False
        cerebro.adddata(data, name=row[0])  # Add the Data Feed to Cerebro

And i added in my strategy:

def prenext(self):
    self.next()