How to filter large stock universe then run trading system on small group

Sagittarius19

Hello,

I am new to backtrader and have been looking through the forums and docs to try to find a solution to my problem but i couldn't find anything that solves it completely.

I have two different datafeeds avaiable to me. The first being daily OHLC data for 100 stocks and then I have tick data for those same 100 stocks.

My idea is to run each stock through a filter daily, and then use that smaller universe of stocks to run my algo.

So far I have been able to filter the large universe of stocks each day into a dictionary that will contain only a handful of names. The trouble is that I have no idea how to use this new dictionary to run a second strategy as backtrader doesn't seem to be able to support that. I also am not sure how to get the name of ticker that is being triggered by my filters. Without the ticker name, im not sure how I can identify the datas.

Does anyone have a solution?

backtrader

You want to run two different algos and you need to run backtrader twice. As you would need to run other platforms twice.

You run your first algo on a cerebro and end up with a dictionary
You use the dictionary to make a 2nd cerebro run with that restricted set of data feeds

@sagittarius19 said in How to filter large stock universe then run trading system on small group:

I also am not sure how to get the name of ticker that is being triggered by my filters

You have created the 1st set of data feeds, so it seems you already have access to the ticker names. We probably won't understand your worries without seeing some code.

Sagittarius19

list item@backtrader thank you for the quick reply.

@sagittarius19 said in How to filter large stock universe then run trading system on small group:

I also am not sure how to get the name of ticker that is being triggered by my filters

You have created the 1st set of data feeds, so it seems you already have access to the ticker names. We probably won't understand your worries without seeing some code.

What i mean to say is, that there is no column in datas that says which ticker I am using. Here is a sample of code.

def __init__(self): 
       self.Universe = []
   def next(self):    
       for i, d in enumerate(self.datas):  
           self.day_open = self.datas[i].open
           self.day_close = self.datas[i].close
           self.date = self.datas[i].datetime.date(0)
           if self.day_open[0]> ((self.day_close[-1])*1.05):
               self.Universe.append(self.date)
   def stop(self):
       print ("Stock list: %s" % self.Universe)

This is only able to append the dates of times where the stocks are over 5% up on the day at the open. I send those date values to the list 'self.Universe' but I dont know how to record in the same list or a separate list the ticker that the filter was running since there is no "symbol" or "ticker" column in the dataframe.

backtrader

@sagittarius19 said in How to filter large stock universe then run trading system on small group:

       self.date = self.datas[i].datetime.date(0)
       if self.day_open[0]> ((self.day_close[-1])*1.05):
           self.Universe.append(self.date)

You are appending a datetime.date instance to your universe. It would really be a miracle if you could actually find out to which data feed it belongs.

You can add each data feed with a name and use it with self.dnames or self.data._name.

See:

Sagittarius19

@backtrader I have updated my code below:

import datetime 
from datetime import timedelta
import os.path  
import sys 
import pandas as pd
import backtrader.feeds as btfeeds
import pandas_datareader.data as web
import backtrader as bt 
import backtrader.indicators as btind
import backtrader.analyzers as btanalyzers

class Pandasclass(btfeeds.PandasData):
    lines = ('symbol')
    params = (('symbol',-1)) 
    datafields = btfeeds.PandasData.datafields + (['symbol'])
class Universe_filter(bt.Strategy):
    Universe_date = []
    Universe_data = []
    def __init__(self):
        pass
    def next(self):    
        for i, d in enumerate(self.datas): 
            self.day_open = self.datas[i].open
            self.day_close = self.datas[i].close
            self.date = self.datas[i].datetime.date(0)
            self.symbol = self.datas[i].symbol
            if self.day_open[0]> ((self.day_close[-1])*1.05):
                self.Universe_date.append(self.date[0])
                self.Universe_data.append(self.symbol[0])
    def stop(self):
        print ("Stock dates: %s" % self.Universe_date)
        print ("Stock datas: %s" % self.Universe_data)
        print (help(btfeeds.PandasData)) 
if __name__ == '__main__':  
    cerebro = bt.Cerebro()
    cerebro.addstrategy(Universe_filter)
    start = datetime.date(2017,4,1)
    end = datetime.date(2018,3,19) 
    #stocks = ["AAPL","MSFT",'TSLA','FB','AMZN','GOOGL','BABA','PYPL','OSTK']
    stocks = ['FB','AAPL'] 
    for i in stocks: 
        DF_1 = web.DataReader([i], "morningstar", start, end)
        DF_1= pd.DataFrame(DF_1)
        DF_1.reset_index(level = 'Symbol', inplace=True,drop=False)
        data1 = Pandasclass(dataname = DF_1)
        cerebro.adddata(data1)       
    cerebro.run()

I am still getting this error though. Any idea what to do?

Traceback (most recent call last):
  File "c:\Users\Robert\Desktop\Python docs\Extended pandas import attempt.py", line 12, in <module>
    class Pandasclass(btfeeds.PandasData):
  File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\lineseries.py", line 350, in __new__
    cls = super(MetaLineSeries, meta).__new__(meta, name, bases, dct)
  File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\metabase.py", line 239, in __new__
    cls.params = params._derive(name, newparams, morebasesparams)
  File "C:\pyth27 fold\pyth27path\lib\site-packages\backtrader\metabase.py", line 116, in _derive
    clsinfo.update(info)
  File "C:\pyth27 fold\pyth27path\lib\_abcoll.py", line 571, in update
    for key, value in other:
ValueError: too many values to unpack

backtrader

Your params definition is wrong. You are defining a tuple of 2 elements and not a tuple in a tuple.

Sagittarius19

@backtrader thanks for the help regarding this. I know its prolly a very simple fix. Here is what I updated to, and im still getting an error.

class Pandasclass(btfeeds.PandasData):
       lines = ('symbol')
       params = (('symbol', -1),)            
       btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol']

File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 13, in <module>
   class Pandasclass(btfeeds.PandasData):
 File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 364, in __new__
   linesoverride, lalias=la)
 File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 130, in _derive
   clslines = baselines + lines
TypeError: can only concatenate tuple (not "str") to tuple

When i put another dummy value next to it, I got this error.

class Pandasclass(btfeeds.PandasData):
   
       lines = ('symbol','blank')
       params = (('symbol', -1),('blank', -1),)            
       btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol', u'blank']

File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 49, in <module>
   cerebro.run()        
 File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run
   runstrat = self.runstrategies(iterstrat)
 File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1209, in runstrategies
   data.preload()
 File "c:\Python36\lib\site-packages\backtrader\feed.py", line 435, in preload
   while self.load():
 File "c:\Python36\lib\site-packages\backtrader\feed.py", line 476, in load
   _loadret = self._load()
 File "c:\Python36\lib\site-packages\backtrader\feeds\pandafeed.py", line 255, in _load
   line[0] = self.p.dataname.iloc[self._idx, colindex]
 File "c:\Python36\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__
   self.array[self.idx + ago] = value
TypeError: must be real number, not str

Does datas not allow for non numerical values to be in the columns?

Sagittarius19

Update: I ended up just using the _names function as you suggested earlier. It works just as well. I would like to know thought, if i can call on none numerical values in the columns of the datas.

class Universe_filter(bt.Strategy):
   Universe_date = []
   Universe_data = []
   def __init__(self):
       pass
   def next(self):    
       for i, d in enumerate(self.datas): 
           self.day_open = self.datas[i].open
           self.day_close = self.datas[i].close
           self.date = self.datas[i].datetime.date
           self.symbol = self.datas[i]._name
           if self.day_open[0]> ((self.day_close[-1])*1.05):
               self.Universe_date.append(self.date(0))
               self.Universe_data.append(self.symbol)
   def stop(self):
       print ("Stock dates: %s" % self.Universe_date)
       print ("Stock datas: %s" % self.Universe_data)
        
if __name__ == '__main__':  
   cerebro = bt.Cerebro()
   cerebro.addstrategy(Universe_filter)
   start = datetime.date(2017,4,1)
   end = datetime.date(2018,3,19) 
   #stocks = ["AAPL","MSFT",'TSLA','FB','AMZN','GOOGL','BABA','PYPL','OSTK']
   stocks = ['FB','AAPL'] 
   for i in stocks: 
       DF_1 = web.DataReader([i], "morningstar", start, end)
       DF_1= pd.DataFrame(DF_1)
       DF_1.reset_index(level = 'Symbol', inplace=True,drop=False)
       data1 = btfeeds.PandasData(dataname = DF_1)
       cerebro.adddata(data1, name = i)       
   cerebro.run()


[Running] python "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py"
Stock dates: [datetime.date(2017, 7, 27), datetime.date(2017, 8, 2)]
Stock datas: ['FB', 'AAPL']

backtrader

@sagittarius19 said in How to filter large stock universe then run trading system on small group:

@backtrader thanks for the help regarding this. I know its prolly a very simple fix. Here is what I updated to, and im still getting an error.

class Pandasclass(btfeeds.PandasData):
       lines = ('symbol')
       params = (('symbol', -1),)            
       btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol']

File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 13, in <module>
   class Pandasclass(btfeeds.PandasData):
 File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 364, in __new__
   linesoverride, lalias=la)
 File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 130, in _derive
   clslines = baselines + lines
TypeError: can only concatenate tuple (not "str") to tuple

When i put another dummy value next to it, I got this error.

class Pandasclass(btfeeds.PandasData):
   
       lines = ('symbol','blank')
       params = (('symbol', -1),('blank', -1),)            
       btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol', u'blank']

File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 49, in <module>
   cerebro.run()        
 File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run
   runstrat = self.runstrategies(iterstrat)
 File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1209, in runstrategies
   data.preload()
 File "c:\Python36\lib\site-packages\backtrader\feed.py", line 435, in preload
   while self.load():
 File "c:\Python36\lib\site-packages\backtrader\feed.py", line 476, in load
   _loadret = self._load()
 File "c:\Python36\lib\site-packages\backtrader\feeds\pandafeed.py", line 255, in _load
   line[0] = self.p.dataname.iloc[self._idx, colindex]
 File "c:\Python36\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__
   self.array[self.idx + ago] = value
TypeError: must be real number, not str

Does datas not allow for non numerical values to be in the columns?

All values are floats. This is not a database.