For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

How to feed a custom pandas dataframe in backtrader?



  • I am new to backtrader and I am trying to backtest a simple strategy using my custom pandas dataframe. This code fetches stock data and modifies the dataframe data by adding 3 additional columns.

    from datetime import datetime
    import backtrader as bt
    import pandas as pd
    from pandas_datareader import data as pdr
    import matplotlib.pyplot as plt
    
    
    data = pdr.get_data_yahoo('AAPL', start=datetime(2017, 8, 13), end=datetime(2018, 8, 14))
    
    data.columns=['high', 'low', 'open', 'close', 'volume', 'adj close']
    data['pct']=data.close.pct_change(1)
    data['pct2']=data.close.pct_change(2)
    data['pct3']=data.close.pct_change(3)
    
    
    ax = plt.gca()
    data.plot(kind='line',y='close',use_index=True,color='blue', ax=ax)
    data.plot(kind='line',y='pct',use_index=True, color='red', ax=ax)
    plt.show()
    
    class PandasData(bt.feed.DataBase):
        params = (
            ('datetime', None),
            ('open', -1),
            ('high', -1),
            ('low', -1),
            ('close', -1),
            ('volume', -1),
            ('openinterest', None),
            ('adj close', -1),
            ('pct', -1),
            ('pct2', -1),
            ('pct3', -1),
        )
    
    
    
    class MyStrategy(bt.Strategy):
        def __init__(self):
            self.sma= bt.ind.SMA(period=10)
        def next(self):
            if self.sma>self.data.close and self.data.pct>0 or self.data.pct2>0:
                print('BUY CREATE, %.2f' % self.data.close[0])
            elif self.sma<self.data.close and self.data.pct<0 or self.data.pct2<0:
                print('SELL CREATE, %.2f' % self.data.close[0])
                self.order = self.sell()
    
    cerebro = bt.Cerebro()
    cerebro.addstrategy(MyStrategy)
    
    #data = bt.feeds.YahooFinanceData(dataname='MSFT', fromdate=datetime(2011, 1, 1),todate=datetime(2012, 12, 31))
    cerebro.adddata(PandasData(data))
    cerebro.broker.setcash(100000.0)
    
    cerebro.addsizer(bt.sizers.FixedSize, stake=1)
    
    cerebro.broker.setcommission(commission=0.05)
    
    cerebro.addanalyzer(bt.analyzers.SharpeRatio, _name='mysharpe')
    cerebro.addanalyzer(bt.analyzers.DrawDown, _name='DrawDown')
    
    print('Starting Balance: %.2f' % cerebro.broker.getvalue())
    
    strats =  cerebro.run(stdstats=False)
    strat = strats[0]
    cerebro.addobserver(bt.observers.Value)
    
    
    print('Sharpe Ratio:', strat.analyzers.mysharpe.get_analysis())
    print('DrawDown:', strat.analyzers.DrawDown.get_analysis())
    print('Final Balance: %.2f' % cerebro.broker.getvalue())
    

    But when I execute strats = cerebro.run(stdstats=False) it shows this error

    IndexError: array assignment index out of range

    How do I feed a custom pandas dataframe in backtrader?


  • administrators

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    How do I feed a custom pandas dataframe in backtrader?

    By properly reading the documentation and not only what you think you have to read and just copying some random lines.

    See this thread: https://community.backtrader.com/topic/1811/getting-attributeerror-genericcsv-object-has-no-attribute-setenvironment-error



  • Thanks for the link It really helped but still doesn't answer my problem. How to feed our custom dataframe to backtrader.

    I am extremely new to backtrader and it will take me many trials and error sto understand its syntax and inner working.I will be frank I just copied the codes from pandas data feed example page. I really didn't see this there The above excerpt from the PandasData class shows the keys: it was below the code excerpts and it didn't catch my eye. I think its because of the text font size and the location of that sentence. I recommend changing its location from below to the top of the code.

    As you suggested in another post I removed all unnecessary codes and only kept the bare minimum.

    from datetime import datetime
    import backtrader as bt
    import pandas as pd
    from pandas_datareader import data as pdr
    
    
    
    data = pdr.get_data_yahoo('AAPL', start=datetime(2017, 8, 13), end=datetime(2018, 8, 14))
    
    data.columns=['high', 'low', 'open', 'close', 'volume', 'adj_close']
    
    
    data['pct']=data.close.pct_change(1)
    data['pct2']=data.close.pct_change(5)
    data['pct3']=data.close.pct_change(10)
    
    class TestStrategy(bt.Strategy):
        def next(self):
            print(','.join(str(x) for x in [self.datetime.datetime(0), self.data.pct[0]]))
    
    class PandasData(bt.feed.PandasData):
        linesoverride = True  # discard usual OHLC structure
        # datetime must be present and last
        lines = ('adj_close','pct','pct2','pct3')
        params = (
            ('datetime', None),
            ('open', -1),
            ('high', -1),
            ('low', -1),
            ('close', -1),
            ('volume', -1),
            ('openinterest', None),
            ('adj_close', -1),
            ('pct', -1),
            ('pct2', -1),
            ('pct3', -1),
        )
    
    cerebro = bt.Cerebro()
    cerebro.addstrategy(TestStrategy)
    
    df = bt.feeds.PandasData(dataname=data)
    #df=PandasData(dataname=data)
    cerebro.adddata(df)
    cerebro.run()
    

    I went to data feed reference page to understand more about bt.feeds.PandasData but its a little confusing.

    Lines:
        close
        low
        high
        open
        volume
        openinterest
        datetime
    Params:
        dataname (None)
        name ()
        compression (1)
        timeframe (5)
        fromdate (None)
    

    I understand params is method's parameters but lines I have never come across this concept in programming. When running the code I get this error

    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'pct'
    

    I assume its releated to lines somehow. what is happening here?


  • administrators

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    class PandasData(bt.feed.PandasData):
        linesoverride = True  # discard usual OHLC structure
        # datetime must be present and last
    

    In this case it is quite unclear why you would want to discard the OHLC structure when you apparently have one. This is for sure not code copied from anywhere. And it won't probably work. You have also copied the comment that says that "datetime must present and last", but don't define a datetime line.

    I am rather sure that you are not looking to override the preexisting datetime-OHLC-volume structure (i.e.: the predefined lines)

    See: https://community.backtrader.com/topic/1832/creating-bid-ask-pandas-datafeed-error

    And see: Docs - Extending a Data Feed

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    I understand params is method's parameters but lines I have never come across this concept in programming

    It's not a programming concept. It's a trading concept. You get all the close prices, plot them along the date axis and join them with a line ... and you have a line. The same you have with any other component of a time series. Call it array if that makes your life easier.

    See:

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    df = bt.feeds.PandasData(dataname=data)
    #df=PandasData(dataname=data)
    cerebro.adddata(df)
    

    Furthermore if you define a custom class and you don't use it ... because you use the internal data feed.

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'pct'
    

    It will for sure never find your pct field.

    @backtrader14 said in How to feed a custom pandas dataframe in backtrader?:

    class PandasData(bt.feed.PandasData):
        linesoverride = True  # discard usual OHLC structure
        # datetime must be present and last
        lines = ('adj_close','pct','pct2','pct3')
        params = (
            ('datetime', None),
            ('open', -1),
            ('high', -1),
            ('low', -1),
            ('close', -1),
            ('volume', -1),
            ('openinterest', None),
            ('adj_close', -1),
            ('pct', -1),
            ('pct2', -1),
            ('pct3', -1),
        )
    

    Taking into account that you know the order in which your columns are in the dataframe, it seems rather strange to redefine the parameters and set all of them to -1. The only reason to redefine the parameters would be to assign fixed values to them, to make sure you are locating the columns.



  • I misunderstood -1, i thought -1 is for auto detecting the columns itself. I don't know what I am doing it wrong here but it still not working.

    class PandasData(bt.feed.DataBase):
        lines = ('adj_close','pct','pct2','pct3')
        params = (
            ('datetime', None),
            ('open',3),
            ('high',1),
            ('low',2),
            ('close',4),
            ('volume',5),
            ('openinterest',None),
            ('adj_close',6),
            ('pct',7),
            ('pct2',8),
            ('pct3',9),
        )
    
    
    #df = bt.feeds.PandasData(dataname=data)
    df=PandasData(dataname=data)
    cerebro.adddata(df)
    cerebro.run()
    

    Whe I run this there is no error but instead of printing pct values it shows this [<__main__.TestStrategy object at 0x7f69e452cb70>]

    I know my bug is somewhere in my custom pandas data class but don't know how to solve it,



  • I have solved the problem thanks to @backtrader and this previous thread. The problem was in my inheritance of PandasData class. I have been using bt.feed.DataBase and bt.feed.PandasData objects for inheriting to my custom class but both were wrong it was bt.feeds.PandasData. This is my working code.

    from datetime import datetime
    import backtrader as bt
    import pandas as pd
    from pandas_datareader import data as pdr
    
    
    
    data = pdr.get_data_yahoo('AAPL', start=datetime(2017, 8, 13), end=datetime(2018, 8, 14))
    
    data.columns=['high', 'low', 'open', 'close', 'volume', 'adj_close']
    
    
    data['pct']=data.close.pct_change(1)
    data['pct2']=data.close.pct_change(5)
    data['pct3']=data.close.pct_change(10)
    
    class TestStrategy(bt.Strategy):
        def next(self):
            #print("hello")
            print(','.join(str(x) for x in [self.datetime.datetime(0), self.data.pct2[0]]))
    
    cerebro = bt.Cerebro()
    cerebro.addstrategy(TestStrategy)
    
    class PandasData(bt.feeds.PandasData):
        lines = ('adj_close','pct','pct2','pct3')
        params = (
            ('datetime', None),
            ('open','open'),
            ('high','high'),
            ('low','low'),
            ('close','close'),
            ('volume','volume'),
            ('openinterest',None),
            ('adj_close','adj_close'),
            ('pct','pct'),
            ('pct2','pct2'),
            ('pct3','pct3'),
        )
    
    
    #df = bt.feeds.PandasData(dataname=data)
    df=PandasData(dataname=data)
    cerebro.adddata(df)
    cerebro.run()
    

    And thank you backtrader for the help.