@run-out Yes, now that you mention it I remember I saw the pull request, but I don't understand why it's not merged yet (it was commited on june).

@run-out Anyway, thanks for your support, it works!

@vladisld OK, thanks for the explanation. I'll revert to my first solution with pandas.

@vladisld My goal is to find stocks by technical indicators and other conditions (integrated in a React web app). My first implementation was to use a single cached pandas dataframe with one row for each stock. Each row had all the information needed for each stock (ATR, ATR(-1), ...). Each search lasted between 1 and 2 seconds.

As I discovered Backtrader I liked a lot his architecture so I migrated everything to it. Now the query lasts 7-8 seconds, but if I increase one day the information for each stock dataframe, 1 second is increased (more or less).

Are you telling then that all dataframes are iterated? I would like to create all information before run, so I can get information directly at stop.

Thanks a lot for your support.

Hi, again. Sorry, but I am stuck. Anyone has any suggestion?

Hi. I've developed an analyzer that uses a data list with 1700 stocks. Each stock has a pandas dataframe with information of his latest ticks OHLC and some indicators. All this information is created outside of cerebro.

At stop method I look for different conditions, as for example if today's close is above the close of three days ago.

As the length of each stock dataframe increases, the process time of the cerebro.run is worse (by seconds). I don't understand why does it happen if at stop I am iterating only the stocks, not each row of their pandas.

How can improve the process time? (I've tried to define 'next' with pass, ...).

The process is:

1. For all symbols I've in a database, I load their ticks and compute his indicators

...
df['rsi'] = btalib.rsi(df, period=14).df
df['atr'] = btalib.atr(df).df

df = df.tail(5)                
data_list.append((df, symbol))

2. This datafeed is added at cerebro:

def execute_analysis(data_list, queries):        
        start_time = timeit.default_timer()
        cerebro = bt.Cerebro()
        
        for i in range(len(self.data_list)):            
            data = PandasData(
                dataname=data_list[i][0], # Pandas DataFrame
                name=data_list[i][1] # The symbol
                )   
                             
            cerebro.adddata(data)     
        cerebro.addanalyzer(ScreenerAnalyzer, _name="screener", queries=queries)

3. The run analyzer has only defined the method 'stop':

class ScreenerAnalyzer(bt.Analyzer):    
 
    def stop(self):
        print('{}: Results'.format(self.datas[0].datetime.date()))
        print('-'*80)
        self.rets = list()
            
        for i, d in enumerate(self.datas):
           ...

@run-out Anyway, thanks for your support, it works!

@run-out Yes, now that you mention it I remember I saw the pull request, but I don't understand why it's not merged yet (it was commited on june).

@run-out Here you got it (last columns are sma20 and sma50):

o h l c v d s symbol max52w min52w sma20 sma50
ts
2020-02-18 85.110069 85.527370 83.987318 84.245651 2872500 0.0 0 A NaN NaN NaN NaN
2020-02-19 84.156212 85.288892 82.914237 84.802040 4741300 0.0 0 A NaN NaN NaN NaN
2020-02-20 84.354941 84.444359 82.824826 83.798531 2543600 0.0 0 A NaN NaN NaN NaN
2020-02-21 83.361355 84.543710 82.864565 84.523842 1762500 0.0 0 A NaN NaN NaN NaN
2020-02-24 81.692144 82.059763 79.506263 79.983185 2919200 0.0 0 A NaN NaN NaN NaN
... ... ... ... ... ... ... .. ... ... ... ... ...
2020-11-24 118.959999 118.959999 111.430000 114.680000 5559900 0.0 0 A NaN NaN 108.683499 104.917219
2020-11-25 116.089996 116.959999 113.410004 114.349998 2862500 0.0 0 A NaN NaN 109.317999 105.209415
2020-11-27 113.769997 114.980003 112.940002 114.089996 983700 0.0 0 A NaN NaN 109.921499 105.502801
2020-11-30 113.889999 117.070000 113.669998 116.900002 3778400 0.0 0 A NaN NaN 110.661999 105.856778
2020-12-01 117.580002 118.379997 115.040001 115.360001 1753700 0.0 0 A NaN NaN 111.167999 106.204910

Using btalib crossover an error is shown:

Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/computation/expressions.py:200: UserWarning: evaluating in Python space because the '-' operator is not supported by numexpr/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/computation/expressions.py:200: UserWarning: evaluating in Python space because the '-' operator is not supported by numexpr for the bool dtype, use '^' instead

My code uses as input parameter a pandas dataframe:

if len(df)>50:
     sma50 = btalib.sma(df, period=50)
     df['sma50'] = btalib.sma(df, period=50).df

     if len(df)>200:
           sma200 = btalib.sma(df, period=200)
           df['sma200'] = sma200.df      
           btalib.crossover(sma50, sma200)

@SPECULARI-FUND I've the same problem. Have you any solution?

@Xavier-Escudero said in Cerebro run spends 20 seconds without any analyzer:

@run-out Thanks. I've tried but the next error is shown:

clslines = baselines + lines
TypeError: can only concatenate tuple (not "str") to tuple

It seems the error appears when I declare the lines field:

class PandasData(btfeed.PandasData):
    '''
    The ``dataname`` parameter inherited from ``feed.DataBase`` is the pandas
    DataFrame
    '''    
    lines = ('sma')
    
    params = (
        ('nullvalue', 0.0),
        # Possible values for datetime (must always be present)
        #  None : datetime is the "index" in the Pandas Dataframe
        #  -1 : autodetect position or case-wise equal name
        #  >= 0 : numeric index to the colum in the pandas dataframe
        #  string : column name (as index) in the pandas dataframe
        ('datetime', None),

        # Possible values below:
        #  None : column not present
        #  -1 : autodetect position or case-wise equal name
        #  >= 0 : numeric index to the colum in the pandas dataframe
        #  string : column name (as index) in the pandas dataframe
        ('open', 'o'),
        ('high', 'h'),
        ('low', 'l'),
        ('close', 'c'),
        ('volume', 'v'),
        ('openinterest', None),
        ('sma', -1)        
    )
    
    datafields = btfeed.PandasData.datafields + (
        [
            'sma'
        ]
    )    

And I am creating the data as:

  df = pd.DataFrame.from_records(ticks) 
  df = df.join(btalib.sma(df, period=20).df)

The data shows that everything is ok, all are floats or NaN (for example first values of SMA), so I don't understand the meaning of error.

Thanks again.

New: It works adding ',' at the end of lines:

lines = ('sma',)