For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Stuck in extending Pandas Dataframe



  • My Dataframe look like this :

    Date          Open      High       Low     Close           Vol      Chgp        Chg        Signal
    2010-07-16  2.258520  2.300215  2.258520  2.272418  4.593081e+09       NaN       NaN            False
    2010-07-19  2.244621  2.251570  2.223773  2.230722  1.295949e+09 -0.018349 -0.041696            False
    2010-07-20  2.223773  2.244621  2.223773  2.223773  4.307979e+08 -0.003115 -0.006949            False
    2010-07-21  2.230722  2.244621  2.230722  2.230722  2.977816e+08  0.003125  0.006949            False
    2010-07-22  2.244621  2.300215  2.237672  2.293266  1.009744e+09  0.028037  0.062544            False
    2010-07-23  2.321063  2.432252  2.314114  2.425303  1.522445e+09  0.057576  0.132037             True
    2010-07-26  2.425303  2.460049  2.404455  2.432252  1.186923e+09  0.002865  0.006949            False
    2010-07-27  2.425303  2.501745  2.418353  2.432252  1.040815e+09  0.000000  0.000000            False
    2010-07-28  2.439201  2.494796  2.432252  2.487846  5.923614e+08  0.022857  0.055594            False
    2010-07-29  2.473948  2.494796  2.446151  2.487846  1.104862e+09  0.000000  0.000000            False
    2010-07-30  2.460049  2.466998  2.418353  2.439201  5.110851e+08 -0.019553 -0.048645            False
    2010-08-02  2.453100  2.466998  2.432252  2.446151  3.612042e+08  0.002849  0.006949            False
    2010-08-03  2.453100  2.466998  2.425303  2.439201  2.884245e+08 -0.002841 -0.006949            False
    2010-08-04  2.432252  2.439201  2.404455  2.411404  2.782356e+08 -0.011396 -0.027797            False
    2010-08-05  2.425303  2.432252  2.390556  2.390556  1.825187e+08 -0.008646 -0.020848            False
    

    The problems is I can't locate the columns correctly when using the following and tried to print "Open" column in next():

        class PandasData_Extend(btfeeds.PandasData):
        
            lines = ('Vol','Chgp','Chg','Signal_3_in_row')
            
            params = (
                ('nocase', True),
                ('datetime', None),
                ('openinterest',None),
                ('volume', None),
                ('Vol', -1),
                ('Chgp', -1),
                ('Chg', -1),
                ('Signal', -1),
            )            
            
            btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + ['Vol','Chgp','Chg','Signal']
    
        
        # --- self defined function to calculate algorithm -> Generate the Signal column
        daybar_df = trader.tc.get_bar_ex_eod_direct('1288')
        trader.algo._set_data(daybar_df)
        trader.algo.Signal_3_in_row()
        df        = trader.algo.data
        # ------ End of my function ------
        
        data = PandasData_Extend(dataname= df)
        cerebro.adddata(data)
        
        class Test(bt.Strategy):
    
            def __init__(self):
                self.Open   = self.datas[0].open
                self.Vol    = self.datas[0].Vol
    
                
            def next(self):
                print(self.data.open[0])
                print(self.data.Vol[0])
    
                
        cerebro.addstrategy(Test)
        cerebro.run()
    

    Errors:
    0_1516851449421_mycode2.JPG

    1. My function generate Dataframe with capitalized first letter in columns name ( Which defined globally in my algorithm )
    2. The 'Vol' print correctly, but not the 'Open'

    When I tried to define all columns
    class PandasData_Extend(btfeeds.PandasData):

            lines = ('Open','High','Low','Close','Vol','Chgp','Chg','Signal')
            
            params = (
                ('nocase', True),
                ('datetime', None),
                ('open', None),
                ('high', None),
                ('low', None),
                ('close', None),
                ('openinterest',None),
                ('volume', None),
                ('Open', -1),
                ('High', -1),
                ('Low', -1),
                ('Close', -1),
                ('Vol', -1),
                ('Chgp', -1),
                ('Chg', -1),
                ('Signal_3_in_row', -1),
            )            
            
            btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'datetime', u'Open', u'High', u'Low', 
                                                                             u'Close', u'Vol', u'Chgp', u'Chg', u'Signal']
    
        
    
        # --- self defined function to calculate algorithm -> Generate the Signal column
        daybar_df = trader.tc.get_bar_ex_eod_direct('1288')
        trader.algo._set_data(daybar_df)
        trader.algo.Signal_3_in_row()
        df        = trader.algo.data
        # ------ End of my function ------
        
        data = PandasData_Extend(dataname= df)
        cerebro.adddata(data)
        
        class Test(bt.Strategy):
            
            params = dict(period=20)
    
            def __init__(self):
                self.Open   = self.datas[0].Open
                self.High   = self.datas[0].High
                self.Low    = self.datas[0].Low
                self.Close  = self.datas[0].Close
                self.Vol    = self.datas[0].Vol
                self.Date   = self.datas[0].datetime
                self.Chgp   = self.datas[0].Chgp
                self.order = None
                
            def next(self):
                print(self.data.Open[0])
                print(self.data.Vol[0])
    
                
        cerebro.addstrategy(Test)
        cerebro.run()
    

    And the result like this :
    0_1516852205370_mycode4.JPG

    I would like to use my own DataFrame(my own algorithm which i maintained for over four years) to work with backtrader, but i failed to extend from PandasData and put everything in to a lines object. I have tried everything I could, any help will be appreciated!



  • Dear all,

    I have solved my problem:

        class PandasData_Extend(btfeeds.PandasData):
        
            lines = ('vol', 'chgp','chg','signal_3_in_row')
            
            params = (
                ('nocase', True),
                ('datetime', None),
                ('vol', -1),
                ('chgp', -1),
                ('chg', -1),
                ('signal_3_in_row', -1),
            )            
            
            btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'vol', u'chgp', u'chg', u'signal_3_in_row']
    
        
    
        # --- self defined function to calculate algorithm -> Generate the Signal column
        daybar_df = trader.tc.get_bar_ex_eod_direct('1288')
        trader.algo._set_data(daybar_df)
        trader.algo.Signal_3_in_row()
        df        = trader.algo.data
        #df.columns = [u'open', u'high', u'low',  u'close', u'vol', u'chgp', u'chg', u'signal']
        print(df)
        # ------ End of my function ------
        
        data = PandasData_Extend(dataname= df)
        cerebro.adddata(data)
        
        class Test(bt.Strategy):
            
            params = dict(period=20)
    
            def __init__(self):
                self.Open   = self.datas[0].open
                self.High   = self.datas[0].high
                self.Low    = self.datas[0].low
                self.Close  = self.datas[0].close
                self.Vol    = self.datas[0].vol
                self.Date   = self.datas[0].datetime
                self.Chgp   = self.datas[0].chgp
                self.Signal = self.datas[0].signal_3_in_row
                self.order = None
                
            def next(self):
                print(self.Date[0], self.Open[0], self.High[0], self.Low[0], self.Close[0], self.Vol[0], self.Chgp[0], self.Signal[0])
    
                
        cerebro.addstrategy(Test)
        cerebro.run()
    

    output:

    736709.0 4.04 4.11 4.02 4.04 321136467.0 0.01 1.0
    736710.0 4.06 4.19 4.05 4.18 292812368.0 0.0346534653465 1.0
    736711.0 4.18 4.3 4.14 4.3 512041198.0 0.0287081339713 1.0
    736712.0 4.4 4.46 4.37 4.45 678387827.0 0.0348837209302 1.0
    736713.0 4.47 4.5 4.45 4.5 424092684.0 0.0112359550562 1.0
    736716.0 4.47 4.54 4.4 4.42 298852642.0 -0.0177777777778 0.0
    736717.0 4.46 4.65 4.44 4.61 511448990.0 0.0429864253394 0.0
    736718.0 4.57 4.8 4.55 4.8 593277910.0 0.0412147505423 0.0
    

    Thanks for the platform anyway!



  • Too late to the party. There was a thread about that topic ... which would have given you the clue you already found.



  • @paska-houso Thanks for that !

    I am using this now to make everything more "dynamic" (my Dataframe have different columns according my algo):

            col = set(df.columns.tolist()) - set(trader.property_eod_normal_header)
            
            class PandasData_Signal(btfeeds.PandasData):
                lines  = tuple(col)
                params = tuple([tuple((x, -1)) for x in lines])
                datafields = btfeeds.PandasData.datafields + (list(col))
    


  • Good. In any case you may want to have a look at this thread which has been discussing the same topic: