3. Bug loading multidata?
For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Bug loading multidata?

  • A

    Dear all,
    when I call .adddata using a loop data are not loaded correctly, but when I load one by one it works correctly.

    for example

    this doesn't work:

    df = pd.read_csv(SECURITYLISTFILE)

for index, row in df.iterrows():
    securityfile = DATAFOLDER + "/" + row[0] + ".csv"
    if os.path.exists(securityfile):    
        # Create a Data Feed
        data = bt.feeds.GenericCSVData(
            dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
        print ("added " + securityfile)
        cerebro.adddata(data)  # Add the Data Feed to Cerebro

    This works:

    data0 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
cerebro.adddata(data0)

data1 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENI.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
cerebro.adddata(data1)
    0

  • I had no problems to load data using the loop. Based on the 2 pieces of code it is hard to understand what was going wrong. Maybe your dataframe contains incorrect data, maybe path to each security is created incorrectly.

    0
  • L

    is the dataname=securityfile always unique? maybe it is the same name?

    0
  • M

    Can you add more details please? Standard output - any exceptions thrown? What do you exactly mean by 'this doesn't work'

    0
  • A

    In the first case, the strategy is not executed and I obtain this error, on plot.

    IndexErrorTraceback (most recent call last)
    <ipython-input-4-384fd2e61059> in <module>()
    215 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    216
    --> 217 cerebro.plot(style ='candle') #style ='candle' line
    218 # an = result[0]
    219 # for each in an.analyzers:

    /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
    997 rfig = plotter.plot(strat, figid=si * 100,
    998 numfigs=numfigs, iplot=iplot,
    --> 999 start=start, end=end, use=use)
    1000 # pfillers=pfillers2)
    1001

    /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
    209 xtemp.append(dt)
    210
    --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
    212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
    213

    IndexError: list index out of range

    If I comment the .plot method, nothing happens, the strategy is not executed.

    Thank you

    0
  • P

    There is a very obvious reason.

    @Alessandro-Di-Piazza said in Bug loading multidata?:

        securityfile = DATAFOLDER + "/" + row[0] + ".csv"
    if os.path.exists(securityfile):

    Those are names generated dynamically (names which are not being shown in your output)

    vs

    @Alessandro-Di-Piazza said in Bug loading multidata?:

    data0 = bt.feeds.GenericCSVData(
            dataname='/jupyter/data/ENEL.mi.csv', fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)

    These names are hardcoded.

    Print out the value inside the variable securityfile and see if that matches your hardcoded names. The bet is clear: they won't and it will be clear that if os.path.exists(... is actually preventing you from entering the logic, hence: no data feeds are actually being loaded.

    Debugging 101.

    In any case

    DATAFOLDER + "/" + row[0] + ".csv"

    is bad practice. Use

    os.path.join(DATAFOLDER, row[0] + '.csv')

    to abstract the difference between operating systems.

    Note: os.path.exists will also be True for directories and not only for files.

    0
  • A

    Thank you for your reply.
    I changed the code using

    securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')

    This is the code now
    if not os.path.exists(SECURITYLISTFILE):
    raise Exception('No file security file list ' + SECURITYLISTFILE)

    cerebro = bt.Cerebro()

today = datetime.date.today()
df = pd.read_csv(SECURITYLISTFILE)

for index, row in df.iterrows():
    securityfile = os.path.join(DATAFOLDER, row[0] + '.csv')
  
    # cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
    if os.path.exists(securityfile):    
        # Create a Data Feed
        data = bt.feeds.GenericCSVData(
            dataname=securityfile, fromdate=datetime.datetime(STARTYEAR, STARTMONTH, STARTDAY), todate=datetime.datetime(ENDYEAR, ENDMONTH, ENDDAY),
            nullvalue=0.0, dtformat=('%Y-%m-%d'), datetime=0, open=1, high=2, low=3, close=4, volume=6, reverse=False)
        print ("added " + securityfile)
        print (data)
        cerebro.adddata(data)  # Add the Data Feed to Cerebro

cerebro.addstrategy(TestStrategy)    # Add a strategy
cerebro.broker.setcash(10000.0)

cerebro.addsizer(FixedFractionalSizer, perc=50)


print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
result = cerebro.run(runonce=False,  stdstats=False)
print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.plot(style ='candle') #style ='candle' line

    and this the output

    added /jupyter/data/ENEL.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0cb33490>
    added /jupyter/data/ENI.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a2d0>
    added /jupyter/data/STM.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79a950>
    added /jupyter/data/G.mi.csv
    <backtrader.feeds.csvgeneric.GenericCSVData object at 0x7fda0b79afd0>
    Starting Portfolio Value: 10000.00
    Final Portfolio Value: 10000.00

    IndexErrorTraceback (most recent call last)
    <ipython-input-9-cbbbf2d55bb7> in <module>()
    218 print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    219
    --> 220 cerebro.plot(style ='candle') #style ='candle' line
    221 # an = result[0]
    222 # for each in an.analyzers:

    /usr/local/lib/python2.7/dist-packages/backtrader/cerebro.pyc in plot(self, plotter, numfigs, iplot, start, end, savefig, figfilename, width, height, dpi, tight, use, **kwargs)
    997 rfig = plotter.plot(strat, figid=si * 100,
    998 numfigs=numfigs, iplot=iplot,
    --> 999 start=start, end=end, use=use)
    1000 # pfillers=pfillers2)
    1001

    /usr/local/lib/python2.7/dist-packages/backtrader/plot/plot.pyc in plot(self, strategy, figid, numfigs, iplot, start, end, use, **kwargs)
    209 xtemp.append(dt)
    210
    --> 211 self.pinf.xstart = bisect.bisect_left(dts, xtemp[0])
    212 self.pinf.xend = bisect.bisect_right(dts, xtemp[-1])
    213

    IndexError: list index out of range

    0
    A
     1 Reply
  • A

    Solved:
    I modified the lines inside the loop

            data.plotinfo.plot = False
        cerebro.adddata(data, name=row[0])  # Add the Data Feed to Cerebro

    And i added in my strategy:

    def prenext(self):
    self.next()
    0