Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Running Optiomisation fails on Google Cloud and AWS

    General Discussion
    2
    7
    1454
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      j45p41 last edited by

      Hi Guys,

      I am trying to run a basic optimisation on a virtual cloud server and it just hangs - tried google and amazon.

      Here are my parameters:

      cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(40, 90))
      

      Works okay on my windows machine natively under windows and under wsl compatibility layer for ubuntu. Could this be due to the google and aws machines being virtual machines?

      The following processes are created on aws:

      1519 jaspal_+  20   0  569808 119788   5628 R  69.8  1.2   0:23.75 python3                                        
       1517 jaspal_+  20   0  571092 121288   5628 R  69.1  1.2   0:23.92 python3                                        
       1514 jaspal_+  20   0  573568 123668   5616 S  67.4  1.2   0:24.58 python3                                        
       1504 jaspal_+  20   0 1958668 1.311g  20696 S  67.1 13.4   0:17.77 python3                                        
       1518 jaspal_+  20   0  578120 127868   5628 R  65.8  1.2   0:24.01 python3                                        
       1515 jaspal_+  20   0  573256 123348   5628 R  65.4  1.2   0:24.21 python3                                        
       1513 jaspal_+  20   0  582864 132484   5628 R  63.5  1.3   0:23.01 python3                                        
       1516 jaspal_+  20   0  572428 122100   5616 S  60.8  1.2   0:23.92 python3                                        
       1520 jaspal_+  20   0  579880 129984   5628 R  59.1  1.3   0:23.88 python3           
      

      Any help would be very much appreciated!

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        You could try optimizing a simple strategy, for example a moving average crossover, for a very limited amount of values (10-12), (20-22) and with a limited amount of data, for example 50 bars.

        If it works it means you are hitting the limits of your virtual machines in terms of either CPU, RAM or both.

        J 2 Replies Last reply Reply Quote 1
        • J
          j45p41 @backtrader last edited by

          @backtrader First of all thanks for responding. I actually printed out the output for each iteration and tried it on my local machine (which I thought was working okay but as it was a long back test and my machine is slow i must have not let it go this far). Anyway here is the error I got:

          multiprocessing.pool.RemoteTraceback: 
          """
          Traceback (most recent call last):
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 119, in worker
              result = (True, func(*args, **kwds))
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__
              return self.runstrategies(iterstrat, predata=predata)
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1290, in runstrategies
              self._runonce(runstrats)
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1648, in _runonce
              strat._once()
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once
              indicator._once()
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once
              indicator._once()
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 632, in _once
              self.once(self._minperiod, self.buflen())
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 756, in once
              self._once_op(start, end)
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 773, in _once_op
              dst[i] = op(srca[i], srcb[i])
          ZeroDivisionError: float division by zero
          """
          
          The above exception was the direct cause of the following exception:
          
          Traceback (most recent call last):
            File "C:/Users/Jaspal/OneDrive/Financial Stuff/btrader_git/first_strategy.py", line 266, in <module>
              opt_runs = cerebro.run(tradehistory=False)
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1140, in run
              for r in pool.imap(self, iterstrats):
            File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 735, in next
              raise value
          ZeroDivisionError: float division by zero
          
          

          is there any way that i can handle this exception?

          1 Reply Last reply Reply Quote 0
          • J
            j45p41 last edited by

            I guess I could use:

            try:
            XXXX
            except ZeroDivisionError:
            print '0' #or whatever
            

            just not sure how to implement. Ideally I would like the optomisation to continue with next usable values.

            1 Reply Last reply Reply Quote 0
            • J
              j45p41 last edited by

              okay got it!

              used:

                  self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)
              

              sourced from:

              https://community.backtrader.com/topic/98/float-division-by-zero

              amazing what you can find when you search around :)

              1 Reply Last reply Reply Quote 0
              • J
                j45p41 @backtrader last edited by

                @backtrader Hello again. So now I have been able to resolve this issue I running my optimisation in a for loop so that it repeats itself using a for loop:

                if __name__ == '__main__':
                
                    sys.stdout = Logger("firststrategy.log")
                
                
                    periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12])
                    periods.loc[1] = ('2017-01-01','2017-02-01')
                    periods.loc[2] = ('2017-02-01','2017-03-01')
                    periods.loc[3] = ('2017-03-01','2017-04-01')
                    periods.loc[4] = ('2017-04-01','2017-05-01')
                    periods.loc[5] = ('2017-05-01','2017-06-01')
                    periods.loc[6] = ('2017-06-01','2017-07-01')
                    periods.loc[7] = ('2017-07-01','2017-08-01')
                    periods.loc[8] = ('2017-08-01','2017-09-01')
                    periods.loc[9] = ('2017-09-01','2017-10-01')
                    periods.loc[10] = ('2017-10-01','2017-11-01')
                    periods.loc[11] = ('2017-11-01','2017-12-01')
                    periods.loc[12] = ('2017-12-01','2017-12-31')
                
                    for index, row in periods.iterrows():
                
                
                        # Variable for our starting cash
                        startcash = 10000
                        # Create an instance of cerebro
                        cerebro = bt.Cerebro(optreturn=False)
                
                        # ADD STRATEGY OPTIMISATION
                        cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))
                
                

                It runs great for the first date range but then the system runs out of memory on the second. The issue is that it is not flushing out the memory from the previous run. Is there a way to get cerbero to reset itself for second (and subsequent) iterations?

                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators last edited by

                  It's not cerebro what you want to flush because you are creating a new instance each time, it is python itself and depends on the rest of things you do in your code (which have obviously been redacted)

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                  $(document).ready(function () { app.coldLoad(); }); }