Bug loading multidata?
-
Dear all,
when I call .adddata using a loop data are not loaded correctly, but when I load one by one it works correctly.for example
this doesn't work:
df = pd.read_csv(SECURITYLISTFILE) for index, row in df.iterrows(): securityfile = DATAFOLDER + "/" + row[0] + ".csv" if os.path.exists(securityfile): # Create a Data Feed data = bt.feeds.GenericCSVData( dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY), nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False) print ("added " + securityfile) cerebro.adddata(data) # Add the Data Feed to Cerebro
This works:
data0 = bt.feeds.GenericCSVData( dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY), nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False) cerebro.adddata(data0) data1 = bt.feeds.GenericCSVData( dataname='/jupyter/data/ENI.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY), nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False) cerebro.adddata(data1)
-
I had no problems to load data using the loop. Based on the 2 pieces of code it is hard to understand what was going wrong. Maybe your dataframe contains incorrect data, maybe path to each security is created incorrectly.
-
is the dataname=securityfile always unique? maybe it is the same name?
-
Can you add more details please? Standard output - any exceptions thrown? What do you exactly mean by 'this doesn't work'
-
In the first case, the strategy is not executed and I obtain this error, on plot.
IndexErrorTraceback (most recent call last)
<ipython-input-4-384fd2e61059> in <module>()
215 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
216
--> 217 cerebro.plot(style ='candle') #style ='candle' line
218 # an = result[0]
219 # for each in an.analyzers:/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
997 rfig = plotter.plot(strat, figid=si * 100,
998 numfigs=numfigs, iplot=iplot,
--> 999 start=start, end=end, use=use)
1000 # pfillers=pfillers2)
1001/usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
209 xtemp.append(dt)
210
--> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
213IndexError: list index out of range
If I comment the .plot method, nothing happens, the strategy is not executed.
Thank you
-
There is a very obvious reason.
@Alessandro-Di-Piazza said in Bug loading multidata?:
securityfile = DATAFOLDER + "/" + row[0] + ".csv" if os.path.exists(securityfile):
Those are names generated dynamically (names which are not being shown in your output)
vs
@Alessandro-Di-Piazza said in Bug loading multidata?:
data0 = bt.feeds.GenericCSVData( dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY), nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
These names are hardcoded.
Print out the value inside the variable
securityfile
and see if that matches your hardcoded names. The bet is clear: they won't and it will be clear thatif os.path.exists(...
is actually preventing you from entering the logic, hence: no data feeds are actually being loaded.Debugging 101.
In any case
DATAFOLDER + "/" + row[0] + ".csv"
is bad practice. Use
os.path.join(DATAFOLDER, row[0] + '.csv')
to abstract the difference between operating systems.
Note:
os.path.exists
will also beTrue
for directories and not only for files. -
Thank you for your reply.
I changed the code usingsecurityfile = os.path.join(DATAFOLDER, row[0] + '.csv')
This is the code now
if not os.path.exists(SECURITYLISTFILE):
raise Exception('No file security file list ' + SECURITYLISTFILE)cerebro = bt.Cerebro() today = datetime.date.today() df = pd.read_csv(SECURITYLISTFILE) for index, row in df.iterrows(): securityfile = os.path.join(DATAFOLDER, row[0] + '.csv') # cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'] if os.path.exists(securityfile): # Create a Data Feed data = bt.feeds.GenericCSVData( dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY), nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False) print ("added " + securityfile) print (data) cerebro.adddata(data) # Add the Data Feed to Cerebro cerebro.addstrategy(TestStrategy) # Add a strategy cerebro.broker.setcash(10000.0) cerebro.addsizer(FixedFractionalSizer, perc=50) print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) result = cerebro.run(runonce=False, stdstats=False) print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) cerebro.plot(style ='candle') #style ='candle' line
and this the output
added /jupyter/data/ENEL.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0cb33490>
added /jupyter/data/ENI.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a2d0>
added /jupyter/data/STM.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a950>
added /jupyter/data/G.mi.csv
<backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79afd0>
Starting Portfolio Value: 10000.00
Final Portfolio Value: 10000.00IndexErrorTraceback (most recent call last)
<ipython-input-9-cbbbf2d55bb7> in <module>()
218 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
219
--> 220 cerebro.plot(style ='candle') #style ='candle' line
221 # an = result[0]
222 # for each in an.analyzers:/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
997 rfig = plotter.plot(strat, figid=si * 100,
998 numfigs=numfigs, iplot=iplot,
--> 999 start=start, end=end, use=use)
1000 # pfillers=pfillers2)
1001/usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
209 xtemp.append(dt)
210
--> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
213IndexError: list index out of range
-
Solved:
I modified the lines inside the loopdata.plotinfo.plot = False cerebro.adddata(data, name=row[0]) # Add the Data Feed to Cerebro
And i added in my strategy:
def prenext(self): self.next()