Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Mutidata + Multicore Optimization AttributeError

    General Code/Help
    2
    3
    1089
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      ThatBlokeDave last edited by ThatBlokeDave

      Hello All,
      I have run into some peculiar results when trying to optimise a strategy that uses multiple data feeds. The results I am getting also seem to be effected by maxcpus. The key word here is "seems". I am not really sure how they are interrelated (if at all).

      I have put together a little demonstration code to see if others can reproduce it. However, first let me describe what I am seeing.

      1. If I run a simple script with a single data feed, I can optimise without issue.
      2. If I add a second data feed, my strategy exits with an AttributeError
      3. If I reduce the maxcpus to 1, essentially disabling multicore support, then the optimisation runs but all of the final results are the same.

      The test script is below. To drive it, we just need to change the following 4 lines

      # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = True
          multicore = 1
          testStrategy = multTestStrategy
          second_data = True
      # -------------------------------------------------------------------------/
      

      And the script:

      import backtrader as bt
      from datetime import datetime
      
      
      class singleTestStrategy(bt.Strategy):
          params = (
              ('period',21),
              )
      
          def __init__(self):
              self.startcash = self.broker.getvalue()
              self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period)
      
          def next(self):
              if not self.position:
                  if self.rsi < 30:
                      self.buy(size=100)
              else:
                  if self.rsi > 70:
                      self.sell(size=100)
      
      
      
      class multTestStrategy(bt.Strategy):
          params = (
              ('period',21),
              )
      
          def __init__(self):
              self.startcash = self.broker.getvalue()
              self.data_dict = dict()
      
              for i, d in enumerate(self.datas):
                  self.data_dict[d] = dict()
                  self.data_dict[d]['rsi'] = bt.indicators.RSI_SMA(d, period=self.params.period)
      
          def next(self):
      
              for i, d in enumerate(self.datas):
                  pos = self.broker.getposition(d).size
                  if not pos:
                      if self.data_dict[d]['rsi'] < 30:
                          self.buy(d,size=10)
                  else:
                      if self.data_dict[d]['rsi'] > 70:
                          self.sell(d,size=10)
      
      
      if __name__ == '__main__':
          #Variable for our starting cash
          startcash = 10000
      
          # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = True
          multicore = 0
          testStrategy = singleTestStrategy
          second_data = False
          # -------------------------------------------------------------------------/
      
          #Create an instance of cerebro
          cerebro = bt.Cerebro(optreturn=False, maxcpus=multicore)
      
          #Add our strategy
          if optimize:
              cerebro.optstrategy(testStrategy, period=[14,21,28])
          else:
              cerebro.addstrategy(testStrategy, period=21)
      
      
          data = bt.feeds.Quandl(
              dataname='AAPL',
              fromdate = datetime(2016,1,1),
              todate = datetime(2017,1,1),
              buffered= True
              )
      
          data2 = bt.feeds.Quandl(
              dataname='AMZN',
              fromdate = datetime(2016,1,1),
              todate = datetime(2017,1,1),
              buffered= True
              )
      
          #Add the data to Cerebro
          cerebro.adddata(data)
      
          #Add the data to Cerebro
          if second_data:
              cerebro.adddata(data2)
      
          # Set our desired cash start
          cerebro.broker.setcash(startcash)
      
          # Run over everything
          if optimize:
              opt_runs = cerebro.run()
      
              # Generate results list
              final_results_list = []
              for run in opt_runs:
                  for strategy in run:
                      value = round(strategy.broker.get_value(),2)
                      PnL = round(value - startcash,2)
                      period = strategy.params.period
                      final_results_list.append([period,PnL])
      
              #Sort Results List
              by_period = sorted(final_results_list, key=lambda x: x[0])
              by_PnL = sorted(final_results_list, key=lambda x: x[1], reverse=True)
      
              #Print results
              print('Results: Ordered by period:')
              for result in by_period:
                  print('Period: {}, PnL: {}'.format(result[0], result[1]))
              print('Results: Ordered by Profit:')
              for result in by_PnL:
                  print('Period: {}, PnL: {}'.format(result[0], result[1]))
          else:
              # Run over everything
              cerebro.run()
      
              #Get final portfolio Value
              portvalue = cerebro.broker.getvalue()
              pnl = portvalue - startcash
      
              #Print out the final result
              print('Final Portfolio Value: ${}'.format(portvalue))
              print('P/L: ${}'.format(pnl))
      
              #Finally plot the end results
              cerebro.plot(style='candlestick')
      
      

      Test Case 1 - Single Data with Optimization

      This is the baseline - Working example

      Settings:

      # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = True
          multicore = 0
          testStrategy = singleTestStrategy
          second_data = False
      # -------------------------------------------------------------------------/
      

      Output:

      Results: Ordered by period:
      Period: 14, PnL: 2153.0
      Period: 21, PnL: 1710.0
      Period: 28, PnL: 1327.0
      Results: Ordered by Profit:
      Period: 14, PnL: 2153.0
      Period: 21, PnL: 1710.0
      Period: 28, PnL: 1327.0
      

      Test Case 2 - Multi-Data / Multicore with Optimization

      Settings:

      # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = True
          multicore = 0
          testStrategy = multTestStrategy
          second_data = True
      # -------------------------------------------------------------------------/
      

      Output:

      Exception in thread Thread-3:
      Traceback (most recent call last):
        File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
          self.run()
        File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run
          self._target(*self._args, **self._kwargs)
        File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 463, in _handle_results
          task = get()
        File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 251, in recv
          return _ForkingPickler.loads(buf.getbuffer())
      AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_aa890439e0cc4922a6091c60d2c6e113' on <module 'backtrader.lineseries' from 'C:\\Users\\Dave\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\backtrader\\lineseries.py'>
      

      Test Case 3 - Multi-Data / Multicore WITHOUT Optimization

      Again, just a positive working test case showing that things are working on the multTestStrategy

      Settings:

      # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = False
          multicore = 0
          testStrategy = multTestStrategy
          second_data = True
      # -------------------------------------------------------------------------/
      

      Output:

      Final Portfolio Value: $10046.32851701423
      P/L: $46.328517014229874
      

      Test Case 4 - Multi-Data / Single Core with Optimization

      The final test case is also quite perculiar. Limiting maxcpus to 1. I get the correct period from the strategyreturned by cerebrobut the get_value() broker calls all seem to return the same figure.

      Settings:

      # ------------------------ PLAY WITH THESE PARAMS -------------------------/
          optimize  = True
          multicore = 1
          testStrategy = multTestStrategy
          second_data = True
      # -------------------------------------------------------------------------/
      

      Output:

      Results: Ordered by period:
      Period: 14, PnL: 333.98
      Period: 21, PnL: 333.98
      Period: 28, PnL: 333.98
      Results: Ordered by Profit:
      Period: 14, PnL: 333.98
      Period: 21, PnL: 333.98
      Period: 28, PnL: 333.98
      

      As we can see the period is changing but the PnL remains the same.

      So overall, I am not sure what is causing it but I am also not sure how where to begin with debugging!

      B 1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators @ThatBlokeDave last edited by

        @thatblokedave said in Mutidata + Multicore Optimization AttributeError:

        If I reduce the maxcpus to 1, essentially disabling multicore support, then the optimisation runs but all of the final results are the same.

        The problem is that you have to store the PnL value in the stop method of the Strategy. Because unless you do that you talk to the last instance of the PnL which cerebro keeps record of. (With maxcpus > 1 it would be the same)

        See here: Community - Optimization: Can you reproduce this bug?

        @thatblokedave said in Mutidata + Multicore Optimization AttributeError:

        AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_aa890439e0cc4922a6091c60d2c6e113' on <module 'backtrader.lineseries' from 'C:\\Users\\Dave\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\backtrader\\lineseries.py'>
        

        Run with cerebro.run(stdstats=False) and the observers won't run and won't generate the error.

        T 1 Reply Last reply Reply Quote 1
        • T
          ThatBlokeDave @backtrader last edited by

          @backtrader Awesome! Thanks, I will take a look

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
          $(document).ready(function () { app.coldLoad(); }); }