Running Optiomisation fails on Google Cloud and AWS
-
Hi Guys,
I am trying to run a basic optimisation on a virtual cloud server and it just hangs - tried google and amazon.
Here are my parameters:
cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(40, 90))Works okay on my windows machine natively under windows and under wsl compatibility layer for ubuntu. Could this be due to the google and aws machines being virtual machines?
The following processes are created on aws:
1519 jaspal_+ 20 0 569808 119788 5628 R 69.8 1.2 0:23.75 python3 1517 jaspal_+ 20 0 571092 121288 5628 R 69.1 1.2 0:23.92 python3 1514 jaspal_+ 20 0 573568 123668 5616 S 67.4 1.2 0:24.58 python3 1504 jaspal_+ 20 0 1958668 1.311g 20696 S 67.1 13.4 0:17.77 python3 1518 jaspal_+ 20 0 578120 127868 5628 R 65.8 1.2 0:24.01 python3 1515 jaspal_+ 20 0 573256 123348 5628 R 65.4 1.2 0:24.21 python3 1513 jaspal_+ 20 0 582864 132484 5628 R 63.5 1.3 0:23.01 python3 1516 jaspal_+ 20 0 572428 122100 5616 S 60.8 1.2 0:23.92 python3 1520 jaspal_+ 20 0 579880 129984 5628 R 59.1 1.3 0:23.88 python3Any help would be very much appreciated!
-
You could try optimizing a simple strategy, for example a moving average crossover, for a very limited amount of values (10-12), (20-22) and with a limited amount of data, for example 50 bars.
If it works it means you are hitting the limits of your virtual machines in terms of either CPU, RAM or both.
-
@backtrader First of all thanks for responding. I actually printed out the output for each iteration and tried it on my local machine (which I thought was working okay but as it was a long back test and my machine is slow i must have not let it go this far). Anyway here is the error I got:
multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1007, in __call__ return self.runstrategies(iterstrat, predata=predata) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1290, in runstrategies self._runonce(runstrats) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1648, in _runonce strat._once() File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once indicator._once() File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 292, in _once indicator._once() File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 632, in _once self.once(self._minperiod, self.buflen()) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 756, in once self._once_op(start, end) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\linebuffer.py", line 773, in _once_op dst[i] = op(srca[i], srcb[i]) ZeroDivisionError: float division by zero """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:/Users/Jaspal/OneDrive/Financial Stuff/btrader_git/first_strategy.py", line 266, in <module> opt_runs = cerebro.run(tradehistory=False) File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1140, in run for r in pool.imap(self, iterstrats): File "C:\Users\Jaspal\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 735, in next raise value ZeroDivisionError: float division by zerois there any way that i can handle this exception?
-
I guess I could use:
try: XXXX except ZeroDivisionError: print '0' #or whateverjust not sure how to implement. Ideally I would like the optomisation to continue with next usable values.
-
okay got it!
used:
self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)sourced from:
https://community.backtrader.com/topic/98/float-division-by-zero
amazing what you can find when you search around :)
-
@backtrader Hello again. So now I have been able to resolve this issue I running my optimisation in a for loop so that it repeats itself using a for loop:
if __name__ == '__main__': sys.stdout = Logger("firststrategy.log") periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12]) periods.loc[1] = ('2017-01-01','2017-02-01') periods.loc[2] = ('2017-02-01','2017-03-01') periods.loc[3] = ('2017-03-01','2017-04-01') periods.loc[4] = ('2017-04-01','2017-05-01') periods.loc[5] = ('2017-05-01','2017-06-01') periods.loc[6] = ('2017-06-01','2017-07-01') periods.loc[7] = ('2017-07-01','2017-08-01') periods.loc[8] = ('2017-08-01','2017-09-01') periods.loc[9] = ('2017-09-01','2017-10-01') periods.loc[10] = ('2017-10-01','2017-11-01') periods.loc[11] = ('2017-11-01','2017-12-01') periods.loc[12] = ('2017-12-01','2017-12-31') for index, row in periods.iterrows(): # Variable for our starting cash startcash = 10000 # Create an instance of cerebro cerebro = bt.Cerebro(optreturn=False) # ADD STRATEGY OPTIMISATION cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))It runs great for the first date range but then the system runs out of memory on the second. The issue is that it is not flushing out the memory from the previous run. Is there a way to get cerbero to reset itself for second (and subsequent) iterations?
-
It's not cerebro what you want to flush because you are creating a new instance each time, it is python itself and depends on the rest of things you do in your code (which have obviously been redacted)