For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Running Optiomisation fails on Google Cloud and AWS



  • Hi Guys,

    I am trying to run a basic optimisation on a virtual cloud server and it just hangs - tried google and amazon.

    Here are my parameters:

    cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(40, 90))
    

    Works okay on my windows machine natively under windows and under wsl compatibility layer for ubuntu. Could this be due to the google and aws machines being virtual machines?

    The following processes are created on aws:

    1519 jaspal_+  20   0  569808 119788   5628 R  69.8  1.2   0:23.75 python3                                        
     1517 jaspal_+  20   0  571092 121288   5628 R  69.1  1.2   0:23.92 python3                                        
     1514 jaspal_+  20   0  573568 123668   5616 S  67.4  1.2   0:24.58 python3                                        
     1504 jaspal_+  20   0 1958668 1.311g  20696 S  67.1 13.4   0:17.77 python3                                        
     1518 jaspal_+  20   0  578120 127868   5628 R  65.8  1.2   0:24.01 python3                                        
     1515 jaspal_+  20   0  573256 123348   5628 R  65.4  1.2   0:24.21 python3                                        
     1513 jaspal_+  20   0  582864 132484   5628 R  63.5  1.3   0:23.01 python3                                        
     1516 jaspal_+  20   0  572428 122100   5616 S  60.8  1.2   0:23.92 python3                                        
     1520 jaspal_+  20   0  579880 129984   5628 R  59.1  1.3   0:23.88 python3           
    

    Any help would be very much appreciated!


  • administrators

    You could try optimizing a simple strategy, for example a moving average crossover, for a very limited amount of values (10-12), (20-22) and with a limited amount of data, for example 50 bars.

    If it works it means you are hitting the limits of your virtual machines in terms of either CPU, RAM or both.



  • @backtrader First of all thanks for responding. I actually printed out the output for each iteration and tried it on my local machine (which I thought was working okay but as it was a long back test and my machine is slow i must have not let it go this far). Anyway here is the error I got:

    multiprocessing.pool.RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 119, in worker
        result = (True, func(*args, **kwds))
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__
        return self.runstrategies(iterstrat, predata=predata)
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1290, in runstrategies
        self._runonce(runstrats)
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1648, in _runonce
        strat._once()
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once
        indicator._once()
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once
        indicator._once()
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 632, in _once
        self.once(self._minperiod, self.buflen())
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 756, in once
        self._once_op(start, end)
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 773, in _once_op
        dst[i] = op(srca[i], srcb[i])
    ZeroDivisionError: float division by zero
    """
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "C:/Users/Jaspal/OneDrive/Financial Stuff/btrader_git/first_strategy.py", line 266, in <module>
        opt_runs = cerebro.run(tradehistory=False)
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1140, in run
        for r in pool.imap(self, iterstrats):
      File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 735, in next
        raise value
    ZeroDivisionError: float division by zero
    
    

    is there any way that i can handle this exception?



  • I guess I could use:

    try:
    XXXX
    except ZeroDivisionError:
    print '0' #or whatever
    

    just not sure how to implement. Ideally I would like the optomisation to continue with next usable values.



  • okay got it!

    used:

        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)
    

    sourced from:

    https://community.backtrader.com/topic/98/float-division-by-zero

    amazing what you can find when you search around :)



  • @backtrader Hello again. So now I have been able to resolve this issue I running my optimisation in a for loop so that it repeats itself using a for loop:

    if __name__ == '__main__':
    
        sys.stdout = Logger("firststrategy.log")
    
    
        periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12])
        periods.loc[1] = ('2017-01-01','2017-02-01')
        periods.loc[2] = ('2017-02-01','2017-03-01')
        periods.loc[3] = ('2017-03-01','2017-04-01')
        periods.loc[4] = ('2017-04-01','2017-05-01')
        periods.loc[5] = ('2017-05-01','2017-06-01')
        periods.loc[6] = ('2017-06-01','2017-07-01')
        periods.loc[7] = ('2017-07-01','2017-08-01')
        periods.loc[8] = ('2017-08-01','2017-09-01')
        periods.loc[9] = ('2017-09-01','2017-10-01')
        periods.loc[10] = ('2017-10-01','2017-11-01')
        periods.loc[11] = ('2017-11-01','2017-12-01')
        periods.loc[12] = ('2017-12-01','2017-12-31')
    
        for index, row in periods.iterrows():
    
    
            # Variable for our starting cash
            startcash = 10000
            # Create an instance of cerebro
            cerebro = bt.Cerebro(optreturn=False)
    
            # ADD STRATEGY OPTIMISATION
            cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))
    
    

    It runs great for the first date range but then the system runs out of memory on the second. The issue is that it is not flushing out the memory from the previous run. Is there a way to get cerbero to reset itself for second (and subsequent) iterations?


  • administrators

    It's not cerebro what you want to flush because you are creating a new instance each time, it is python itself and depends on the rest of things you do in your code (which have obviously been redacted)


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.