For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

How to filter large stock universe then run trading system on small group



  • Hello,

    I am new to backtrader and have been looking through the forums and docs to try to find a solution to my problem but i couldn't find anything that solves it completely.

    I have two different datafeeds avaiable to me. The first being daily OHLC data for 100 stocks and then I have tick data for those same 100 stocks.

    My idea is to run each stock through a filter daily, and then use that smaller universe of stocks to run my algo.

    So far I have been able to filter the large universe of stocks each day into a dictionary that will contain only a handful of names. The trouble is that I have no idea how to use this new dictionary to run a second strategy as backtrader doesn't seem to be able to support that. I also am not sure how to get the name of ticker that is being triggered by my filters. Without the ticker name, im not sure how I can identify the datas.

    Does anyone have a solution?


  • administrators

    You want to run two different algos and you need to run backtrader twice. As you would need to run other platforms twice.

    1. You run your first algo on a cerebro and end up with a dictionary
    2. You use the dictionary to make a 2nd cerebro run with that restricted set of data feeds

    @sagittarius19 said in How to filter large stock universe then run trading system on small group:

    I also am not sure how to get the name of ticker that is being triggered by my filters

    You have created the 1st set of data feeds, so it seems you already have access to the ticker names. We probably won't understand your worries without seeing some code.



  • list item@backtrader thank you for the quick reply.

    @sagittarius19 said in How to filter large stock universe then run trading system on small group:

    I also am not sure how to get the name of ticker that is being triggered by my filters

    You have created the 1st set of data feeds, so it seems you already have access to the ticker names. We probably won't understand your worries without seeing some code.

    What i mean to say is, that there is no column in datas that says which ticker I am using. Here is a sample of code.

    def __init__(self): 
           self.Universe = []
       def next(self):    
           for i, d in enumerate(self.datas):  
               self.day_open = self.datas[i].open
               self.day_close = self.datas[i].close
               self.date = self.datas[i].datetime.date(0)
               if self.day_open[0]> ((self.day_close[-1])*1.05):
                   self.Universe.append(self.date)
       def stop(self):
           print ("Stock list: %s" % self.Universe)
    

    This is only able to append the dates of times where the stocks are over 5% up on the day at the open. I send those date values to the list 'self.Universe' but I dont know how to record in the same list or a separate list the ticker that the filter was running since there is no "symbol" or "ticker" column in the dataframe.


  • administrators

    @sagittarius19 said in How to filter large stock universe then run trading system on small group:

           self.date = self.datas[i].datetime.date(0)
           if self.day_open[0]> ((self.day_close[-1])*1.05):
               self.Universe.append(self.date)
    

    You are appending a datetime.date instance to your universe. It would really be a miracle if you could actually find out to which data feed it belongs.

    You can add each data feed with a name and use it with self.dnames or self.data._name.

    See:



  • @backtrader I have updated my code below:

    import datetime 
    from datetime import timedelta
    import os.path  
    import sys 
    import pandas as pd
    import backtrader.feeds as btfeeds
    import pandas_datareader.data as web
    import backtrader as bt 
    import backtrader.indicators as btind
    import backtrader.analyzers as btanalyzers
    
    class Pandasclass(btfeeds.PandasData):
        lines = ('symbol')
        params = (('symbol',-1)) 
        datafields = btfeeds.PandasData.datafields + (['symbol'])
    class Universe_filter(bt.Strategy):
        Universe_date = []
        Universe_data = []
        def __init__(self):
            pass
        def next(self):    
            for i, d in enumerate(self.datas): 
                self.day_open = self.datas[i].open
                self.day_close = self.datas[i].close
                self.date = self.datas[i].datetime.date(0)
                self.symbol = self.datas[i].symbol
                if self.day_open[0]> ((self.day_close[-1])*1.05):
                    self.Universe_date.append(self.date[0])
                    self.Universe_data.append(self.symbol[0])
        def stop(self):
            print ("Stock dates: %s" % self.Universe_date)
            print ("Stock datas: %s" % self.Universe_data)
            print (help(btfeeds.PandasData)) 
    if __name__ == '__main__':  
        cerebro = bt.Cerebro()
        cerebro.addstrategy(Universe_filter)
        start = datetime.date(2017,4,1)
        end = datetime.date(2018,3,19) 
        #stocks = ["AAPL","MSFT",'TSLA','FB','AMZN','GOOGL','BABA','PYPL','OSTK']
        stocks = ['FB','AAPL'] 
        for i in stocks: 
            DF_1 = web.DataReader([i], "morningstar", start, end)
            DF_1= pd.DataFrame(DF_1)
            DF_1.reset_index(level = 'Symbol', inplace=True,drop=False)
            data1 = Pandasclass(dataname = DF_1)
            cerebro.adddata(data1)       
        cerebro.run()        
    

    I am still getting this error though. Any idea what to do?

    Traceback (most recent call last):
      File "c:\Users\Robert\Desktop\Python docs\Extended pandas import attempt.py", line 12, in <module>
        class Pandasclass(btfeeds.PandasData):
      File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\lineseries.py", line 350, in __new__
        cls = super(MetaLineSeries, meta).__new__(meta, name, bases, dct)
      File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\metabase.py", line 239, in __new__
        cls.params = params._derive(name, newparams, morebasesparams)
      File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\metabase.py", line 116, in _derive
        clsinfo.update(info)
      File "C:\pyth27 fold\pyth27path\lib\_abcoll.py", line 571, in update
        for key, value in other:
    ValueError: too many values to unpack
    

  • administrators

    Your params definition is wrong. You are defining a tuple of 2 elements and not a tuple in a tuple.



  • @backtrader thanks for the help regarding this. I know its prolly a very simple fix. Here is what I updated to, and im still getting an error.

    class Pandasclass(btfeeds.PandasData):
           lines = ('symbol')
           params = (('symbol', -1),)            
           btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol']
    
    File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 13, in <module>
       class Pandasclass(btfeeds.PandasData):
     File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 364, in __new__
       linesoverride, lalias=la)
     File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 130, in _derive
       clslines = baselines + lines
    TypeError: can only concatenate tuple (not "str") to tuple
    

    When i put another dummy value next to it, I got this error.

    class Pandasclass(btfeeds.PandasData):
       
           lines = ('symbol','blank')
           params = (('symbol', -1),('blank', -1),)            
           btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol', u'blank']
    
    File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 49, in <module>
       cerebro.run()        
     File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run
       runstrat = self.runstrategies(iterstrat)
     File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1209, in runstrategies
       data.preload()
     File "c:\Python36\lib\site-packages\backtrader\feed.py", line 435, in preload
       while self.load():
     File "c:\Python36\lib\site-packages\backtrader\feed.py", line 476, in load
       _loadret = self._load()
     File "c:\Python36\lib\site-packages\backtrader\feeds\pandafeed.py", line 255, in _load
       line[0] = self.p.dataname.iloc[self._idx, colindex]
     File "c:\Python36\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__
       self.array[self.idx + ago] = value
    TypeError: must be real number, not str
    

    Does datas not allow for non numerical values to be in the columns?



  • Update: I ended up just using the _names function as you suggested earlier. It works just as well. I would like to know thought, if i can call on none numerical values in the columns of the datas.

    class Universe_filter(bt.Strategy):
       Universe_date = []
       Universe_data = []
       def __init__(self):
           pass
       def next(self):    
           for i, d in enumerate(self.datas): 
               self.day_open = self.datas[i].open
               self.day_close = self.datas[i].close
               self.date = self.datas[i].datetime.date
               self.symbol = self.datas[i]._name
               if self.day_open[0]> ((self.day_close[-1])*1.05):
                   self.Universe_date.append(self.date(0))
                   self.Universe_data.append(self.symbol)
       def stop(self):
           print ("Stock dates: %s" % self.Universe_date)
           print ("Stock datas: %s" % self.Universe_data)
            
    if __name__ == '__main__':  
       cerebro = bt.Cerebro()
       cerebro.addstrategy(Universe_filter)
       start = datetime.date(2017,4,1)
       end = datetime.date(2018,3,19) 
       #stocks = ["AAPL","MSFT",'TSLA','FB','AMZN','GOOGL','BABA','PYPL','OSTK']
       stocks = ['FB','AAPL'] 
       for i in stocks: 
           DF_1 = web.DataReader([i], "morningstar", start, end)
           DF_1= pd.DataFrame(DF_1)
           DF_1.reset_index(level = 'Symbol', inplace=True,drop=False)
           data1 = btfeeds.PandasData(dataname = DF_1)
           cerebro.adddata(data1, name = i)       
       cerebro.run()        
    
    
    
    [Running] python "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py"
    Stock dates: [datetime.date(2017, 7, 27), datetime.date(2017, 8, 2)]
    Stock datas: ['FB', 'AAPL']
    

  • administrators

    @sagittarius19 said in How to filter large stock universe then run trading system on small group:

    @backtrader thanks for the help regarding this. I know its prolly a very simple fix. Here is what I updated to, and im still getting an error.

    class Pandasclass(btfeeds.PandasData):
           lines = ('symbol')
           params = (('symbol', -1),)            
           btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol']
    
    File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 13, in <module>
       class Pandasclass(btfeeds.PandasData):
     File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 364, in __new__
       linesoverride, lalias=la)
     File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 130, in _derive
       clslines = baselines + lines
    TypeError: can only concatenate tuple (not "str") to tuple
    

    When i put another dummy value next to it, I got this error.

    class Pandasclass(btfeeds.PandasData):
       
           lines = ('symbol','blank')
           params = (('symbol', -1),('blank', -1),)            
           btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol', u'blank']
    
    File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 49, in <module>
       cerebro.run()        
     File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run
       runstrat = self.runstrategies(iterstrat)
     File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1209, in runstrategies
       data.preload()
     File "c:\Python36\lib\site-packages\backtrader\feed.py", line 435, in preload
       while self.load():
     File "c:\Python36\lib\site-packages\backtrader\feed.py", line 476, in load
       _loadret = self._load()
     File "c:\Python36\lib\site-packages\backtrader\feeds\pandafeed.py", line 255, in _load
       line[0] = self.p.dataname.iloc[self._idx, colindex]
     File "c:\Python36\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__
       self.array[self.idx + ago] = value
    TypeError: must be real number, not str
    

    Does datas not allow for non numerical values to be in the columns?

    All values are floats. This is not a database.