For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Maximum number of data lines?



  • Is there a maximum number of data lines (different equities to trade simultaneously) that Backtrader can reasonably handle?

    Right now I am using hourly data for 215 stocks for 4 years. The completion of the backtest takes several hours. This is all right but I wonder how much data could be added. Would it still work with 500 stocks? 1000 stocks? 2000?

    Is there a formula to predict the performance basing on the amount of data?

    I reckon that the answer also depends on the number of indicators & observers and their complexity, but still, if the backtest takes, say, 5 hours with 215 stocks, should it be near 50 for 2150 stocks or not?

    I have 16 GB of RAM so that should not be an obstacle.

    Thanks in advance.



  • Number of data feeds which can be added to backtrader is unlimited.

    As you correctly noticed run time depends on the amount of features added. So best way to get answers is to run several tests with different number of data feeds and make some extrapolation.



  • I would probably measure separately run time for data feed adding and strategy processing. Might give you more accurate results.


  • administrators

    Let's see how many bar you have

    • Weeks: 52
    • Max Trading Days per week: 5
    • Max Trading Days Year: 260
    • Trading Holidays (approx): 8
    • Trading Days per Year: 252
    • Hours per Day: 8.5
    • Total Hours per Year: 2142
    • Total Years: 4
    • Total Hourly Bars per Stock: 8568
    • Number of Stocks: 215
    • Total Number of Bars: 1,842,120 (under 2 Million)

    It's unclear what you do but see please this:

    A complete trading run using pypy (the fastest interpreter and JIT) can be done in roughly 2.5 minutes for 2 Million bars (actual execution time is 2 minutes 36 seconds and 94 centiseconds in a test equipment which is by no means idle)



  • In that case, I must be doing something very wrong indeed.

    I suspect that it is my usage of the Indicators & Observers. Right now I am creating them separately for each data feed similarly to following:

    class MyStrategy(bt.Strategy):
    
      my_indicators = {}
    
      def __init__(self):
    
        for data in self.datas:
          symbol = data.p.dataname.split("\\")[1].split(".")[0]
          self.my_indicators[symbol] = MyIndicator(data)
    

    I create an Observer for each feed principally in the same way (in the main, ofc).

    It did not look quite right when I wrote it but with a small number of feeds, it did not matter much.

    Is this to blame and if yes, what is the right way to use the Indicators with multiple data feeds?


  • administrators

    @kriku said in Maximum number of data lines?:

    I create an Observer for each feed

    With 215 data feeds it seems really pointless to create Observers, which are meant for plotting.

    @kriku said in Maximum number of data lines?:

    self.my_indicators[symbol] = MyIndicator(data)

    There is no other way to use indicators But if your indicators calculate the trajectory of rockets from the Earth to Pluto on each iteration ... it's going to take time.

    The article shows that indicators are also created for each data feed (100 in the sample)

    Without code ... there is no diagnostic which is possible.


Log in to reply
 

});