IndexError: array assignment index out of range
-
I'm getting a similar error as in this post and this post, however the solutions provided there do not work on my script. Unless I'm misunderstanding of course.
The dataset I'm using can be downloaded from my google drive here for reproducibility:
https://drive.google.com/file/d/1fSyEPE9FszABg7ej_7mbB9ts4oHjFmMg/view?usp=sharing
It's a self-made dataset that re-creates "ticks" from a 1 minute dataset of the SPY. So the dataset is in 15 second intervals (re: 4 x 15 seconds in 1 minute). I am using this dataset with
replaydata
to upsample it to 5 minutes.I've created a minimal working example below to reproduce the error, just ensure the dataset provided is in the same directory as the script.
In my real implementation, I'm using a genetic algorithm to help with optimization of parameters, NOT BT's optimize function, so you will see terms pertaining to GA's in there, like
gene_space
andchromosome
etc. It's well commented and easy to follow.Basically, the code runs for the first combination of parameters, but then gives this error before starting the second one:
Traceback (most recent call last): File "test5.py", line 169, in <module> results = cerebro.run(tradehistory=True) File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\feed.py", line 523, in load retff = ff(self, *fargs, **fkwargs) File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\resamplerfilter.py", line 697, in __call__ data._updatebar(self.bar.lvalues(), forward=False, ago=0) File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\feed.py", line 570, in _updatebar line[0 + ago] = val File "C:\Users\chalu\AppData\Roaming\Python\Python37\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__ self.array[self.idx + ago] = value IndexError: array assignment index out of range
...which doesn't really help ME much, other than I think it has something to do with
replaydata
.I've read through the two posts and ensured that I'm not using
[]
brackets when I should be using()
brackets, particularly in the__init__
function, but I do use[0]
in a few spots in thenext
, but this isn't the problem as I've tested this script with another dataset WITHOUT using replay and it works fine.Hopefully someone who has come across this can spot the issue? Here is the code:
from __future__ import (absolute_import, division, print_function, unicode_literals) from numpy import where import backtrader as bt import json # Create a Stratey Class class StrategyClass(bt.Strategy): # Define a self.gene_space. I'll be manually assigning values # in the main program start below, but this emulates my real # code in my project. In other words, don't mind this for now gene_space={'lenSMA':{'low':10, 'high':500, 'step': 10}, 'lenEMA':{'low':10, 'high':500, 'step': 10}, 'keltnerLenK':{'low':20, 'high':1000, 'step': 20}, 'keltnerMult':{'low':0.1, 'high':20, 'step': 0.1}, 'keltnerATRLen':{'low':5, 'high':200, 'step': 5}, 'stochPerK':{'low':5, 'high':200, 'step': 5}, 'stochSmoothk':{'low':5, 'high':200, 'step': 5}, 'stochPerD':{'low':5, 'high':200, 'step': 5}, 'macdFastLen':{'low':10, 'high':500, 'step': 10}, 'macdSlowLen':{'low':20, 'high':1000, 'step': 20}, 'macdSignalLen':{'low':10, 'high':500, 'step': 10}, } # Define the __init__ function, whereby we pass in a list of # current "gene" values def __init__(self, current_chromosome): self.current_chromosome = current_chromosome # Assign the current_chromosome values to the strategy class before backtesting them for i, key in enumerate(self.gene_space): self.gene_space[key] = self.current_chromosome[i] self.out = bt.indicators.MovingAverageSimple(self.data1.close, period=self.gene_space['lenSMA']) self.out2 = bt.indicators.ExponentialMovingAverage(self.data1.close, period=self.gene_space['lenEMA']) self.ma = bt.indicators.MovingAverageSimple(self.data1.close, period=self.gene_space['keltnerLenK']) self.rangema = bt.indicators.AverageTrueRange(self.data1, period=self.gene_space['keltnerATRLen']) # self.stoch = bt.talib.STOCH(self.data1.high, self.data1.low, self.data1.close, self.gene_space['stochPerK']) self.highest_high = bt.indicators.Highest(self.data1.high, period=self.gene_space['stochPerK']) self.lowest_low = bt.indicators.Lowest(self.data1.low, period=self.gene_space['stochPerK']) self.fast_ma = bt.indicators.MovingAverageSimple(self.data1.close, period=self.gene_space['macdFastLen']) self.slow_ma = bt.indicators.MovingAverageSimple(self.data1.close, period=self.gene_space['macdSlowLen']) self.upper = self.ma + self.rangema * self.gene_space['keltnerMult'] self.lower = self.ma - self.rangema * self.gene_space['keltnerMult'] self.kden = self.highest_high - self.lowest_low self.knum = self.data1.close - self.lowest_low self.stoch = 100.0 * bt.DivByZero(self.knum, self.kden, zero=0.0) self.k = bt.indicators.MovingAverageSimple(self.stoch, period=self.gene_space['stochSmoothk']) self.subtracted = self.fast_ma - self.slow_ma self.signal = bt.indicators.MovingAverageSimple(self.subtracted, period=self.gene_space['macdSignalLen']) self.hist = (self.fast_ma - self.slow_ma) - self.signal # Next Function def next(self): # Order conditions self.enter_long = where((self.data1.high[0] > self.out[0]) & (self.data1.high[0] < self.upper[0]) & (self.data1.high[0] > self.lower[0]) & (self.hist[0] < 0) & (self.k[0] < 50) & (self.data1.high[0] > self.out2[0]), True, False) self.enter_short = where((self.data1.low[0] < self.out[0]) & (self.data1.low[0] < self.upper[0]) & (self.data1.low[0] > self.lower[0]) & (self.hist[0] > 0) & (self.k[0] > 50) & (self.data1.low[0] < self.out2[0]), True, False) self.exit_long = self.enter_short self.exit_short = self.enter_long # If we are NOT currently holding a position if not self.position or self.position.size == 0: if self.enter_long: self.buy(data=self.data1, price=self.data1.close[0] + self.limit_order_threshold, exectype=bt.Order.Limit) elif self.enter_short: self.sell(data=self.data1, price=self.data1.close[0] - self.limit_order_threshold, exectype=bt.Order.Limit) # Check if we are LONG if self.position.size > 0: if self.enter_short or self.exit_long: self.close() self.sell(data=self.data1, price=self.data1.close[0] - self.limit_order_threshold, exectype=bt.Order.Limit) # Check if we are SHORT elif self.position.size < 0: if self.enter_long or self.exit_short: self.close() self.buy(data=self.data1, price=self.data1.close[0] + self.limit_order_threshold, exectype=bt.Order.Limit) # MAIN PROGRAM START # Just assign 2 sets of random chromosome values, the code # will error out after the first one is done. chromosomes_to_test = [[90, 390, 860, 5.9, 175, 55, 95, 150, 320, 720, 120], [390, 90, 740, 18.7, 190, 195, 70, 150, 450, 380, 80]] # Generate the cerebro dataset from the df's data = bt.feeds.GenericCSVData( dataname="SPY_5_1_252_days_backtest_dataset.csv", reverse=False, dtformat=('%Y-%m-%d %H:%M:%S'), timeframe=bt.TimeFrame.Seconds, compression=15, # It's a 15 second "tick" dataset datetime=0, open=1, high=2, low=3, close=4, volume=5, openinterest=-1 ) # Now backtest both for current_chromosome in chromosomes_to_test: print("Current Chromosome:", current_chromosome) # Create a cerebro entity cerebro = bt.Cerebro(stdstats=False) # Add some price slippage to the broker cerebro.broker = bt.brokers.BackBroker(slip_fixed=0.03) # Create orders on Close of current bar or not cerebro.broker.set_coc(False) # Add a strategy class, along with the current_chromosome to test cerebro.addstrategy(StrategyClass, current_chromosome=current_chromosome) # Add the Replay Data Feed to Cerebro cerebro.replaydata(data) data.plotinfo.plotmaster = data # ignore: this just ensures that it plots both intervals on the same plot cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=5) # Upsampled to 5 minutes # Set our desired cash start cerebro.broker.setcash(1000) # Add a FixedSize/AllInSizer/PercentSizer sizer according to the stake # 100% will get rejects due to comission, so set to less than 100 cerebro.addsizer(bt.sizers.PercentSizer, percents=90) #, stake=10 # Set a commission % cerebro.broker.setcommission(commission=0.001) # Add an analyzer or two to get some stats after the run cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name='trade_analyzer') # Run results = cerebro.run(tradehistory=True) # Kowalski, analysis! analysis = results[0].analyzers.getbyname("trade_analyzer").get_analysis() # If you want to inspect the analysis dict, use this, but will clutter # up genetipy's output print(json.dumps(analysis, indent=4)) # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
-
I should re-iterate that the code runs for the first combo of parameters, and starts the 2nd one, but then gives the error, so when debugging, make sure it'll run through both without error. Trades being placed or not is irrelevant as this is just an example code, but running all the way through while getting around this error is the goal. Thanks!
-
No takers?
-
@matt-wilson Trying to replicate something on my side, but struggling.
Have you tried it with only 2 chromosomes before attempting the whole lot?
-
Hey @pierre-cilliers-0 !
Haha another issue on my side only? Lovely.
I don't suppose BackTrader stores a cache or anything does it? I've restarted my computer and and running the latest version of BackTrader on Python 3.7.9 on Windows 10 in case that's relevant.
I've provided a bigger list of chromosomes to try below, so just replace my 2 list with this bigger one and see what you see. If you run into the same error with this new list, that'll tell me it's something to do with the chromosomes, but it's odd because I still get that error with just these two.
Another thing I've tried is to replace this line:
results = cerebro.run(tradehistory=True)
with
try: results = cerebro.run(tradehistory=True) except: print("DOH!") continue
...and see what happens, but the same thing, after the first run, it just prints the
"DOH!"
for every chromosome after the first one. Strange...Anyway, let me know how you make out when you get a chance!
chromosomes_to_test = [[260, 200, 400, 11.6, 150, 135, 100, 55, 320, 120, 35], [30, 380, 530, 0.7, 40, 80, 195, 145, 40, 660, 16], [380, 260, 350, 3.2, 175, 75, 160, 50, 210, 960, 37], [330, 430, 40, 4.1, 115, 35, 55, 35, 350, 400, 27], [410, 130, 210, 17.9, 105, 40, 160, 105, 460, 200, 31], [460, 460, 240, 13.8, 40, 130, 90, 165, 470, 440, 32], [340, 430, 50, 18.0, 60, 75, 10, 30, 470, 400, 22], [70, 430, 500, 12.2, 110, 50, 150, 95, 270, 260, 49], [420, 250, 860, 16.9, 75, 110, 15, 20, 380, 180, 8], [420, 350, 490, 13.6, 60, 175, 160, 130, 410, 920, 39], [240, 240, 90, 6.5, 170, 65, 155, 135, 360, 60, 18], [170, 80, 280, 2.0, 80, 100, 35, 155, 420, 900, 5], [190, 280, 290, 9.2, 110, 195, 135, 170, 350, 800, 46], [370, 370, 740, 1.2, 85, 160, 50, 145, 360, 180, 11], [310, 60, 450, 2.2, 15, 45, 175, 30, 490, 200, 18], [410, 400, 860, 4.4, 85, 30, 80, 100, 460, 140, 36], [160, 130, 980, 2.0, 25, 125, 165, 40, 60, 780, 29], [120, 120, 520, 7.5, 75, 5, 95, 195, 440, 840, 48], [60, 100, 30, 5.5, 130, 155, 80, 150, 150, 260, 5], [110, 200, 260, 8.7, 75, 5, 175, 45, 10, 700, 32], [120, 160, 890, 3.2, 65, 120, 125, 185, 200, 660, 36], [330, 190, 80, 10.7, 170, 140, 105, 110, 400, 300, 16], [440, 50, 280, 16.5, 45, 170, 5, 30, 240, 740, 30], [140, 60, 730, 18.6, 110, 15, 195, 115, 110, 100, 18], [120, 30, 650, 9.0, 55, 30, 85, 65, 60, 940, 43], [80, 100, 300, 18.2, 45, 100, 30, 75, 80, 60, 4], [280, 320, 620, 3.6, 115, 90, 80, 195, 110, 800, 38], [40, 130, 110, 15.3, 90, 105, 110, 150, 160, 300, 29], [390, 40, 80, 11.9, 125, 70, 90, 25, 260, 420, 18], [100, 410, 410, 14.8, 190, 75, 180, 165, 360, 240, 5], [80, 360, 490, 15.5, 15, 60, 85, 95, 280, 840, 36], [20, 340, 150, 7.8, 20, 130, 15, 10, 200, 120, 37], [490, 40, 700, 1.9, 15, 105, 25, 45, 430, 680, 37], [310, 30, 430, 17.4, 95, 125, 15, 115, 50, 440, 18], [160, 430, 720, 9.5, 100, 140, 110, 60, 400, 740, 35], [370, 50, 360, 4.8, 90, 165, 150, 195, 490, 840, 11], [340, 450, 680, 19.1, 160, 130, 115, 5, 320, 540, 11], [20, 270, 660, 19.6, 110, 165, 120, 190, 250, 280, 5], [400, 460, 700, 18.5, 105, 165, 185, 135, 430, 880, 1], [240, 200, 30, 18.1, 10, 125, 15, 185, 290, 940, 31], [260, 390, 780, 2.7, 125, 10, 15, 170, 450, 340, 44], [70, 290, 150, 10.6, 130, 170, 110, 150, 220, 960, 22], [270, 260, 190, 8.9, 40, 135, 95, 155, 150, 820, 36], [300, 30, 30, 8.8, 115, 105, 35, 100, 30, 680, 31], [280, 60, 920, 4.6, 55, 140, 70, 15, 370, 580, 32], [240, 220, 440, 9.6, 60, 65, 165, 175, 130, 480, 39], [400, 20, 970, 8.2, 105, 55, 90, 95, 70, 720, 23], [40, 170, 980, 4.4, 170, 5, 80, 100, 40, 700, 44], [450, 100, 420, 13.7, 65, 130, 75, 145, 220, 740, 18], [330, 190, 130, 3.1, 10, 70, 5, 5, 230, 600, 19], [320, 30, 560, 7.4, 60, 145, 160, 20, 220, 140, 47], [290, 40, 830, 19.4, 75, 60, 120, 55, 170, 540, 21], [420, 160, 940, 16.7, 45, 110, 70, 160, 200, 680, 17], [20, 460, 880, 18.7, 90, 95, 5, 70, 380, 460, 6], [390, 100, 50, 8.5, 60, 100, 90, 120, 50, 400, 43], [440, 360, 840, 13.8, 100, 60, 185, 100, 40, 320, 23], [440, 330, 740, 5.5, 80, 170, 90, 130, 270, 540, 47], [380, 100, 290, 15.7, 90, 65, 175, 190, 200, 840, 16], # Original two here [90, 390, 860, 5.9, 175, 55, 95, 150, 320, 720, 120], [390, 90, 740, 18.7, 190, 195, 70, 150, 450, 380, 80]]
-
I am facing the same error, which is strange.
-
@backtrader Any ideas?
-
@matt-wilson recently solved this problem, but I think it could be caused by many reasons. In my situation, I used open[1] which is a future data when buying and I changed to close[0] and solved. I think may be you can review your strategy. It might be an idea mistake rather than code.