Hi Kevin --
Another option for you is to use the IB data cache code that brettelliot wrote at https://community.backtrader.com/topic/1007/how-to-cached-data-from-ib-and-use-it-later/2
It exposes the IB connection so that the ib api parameter for RTH is available (use_rth).
Also gets you a nice cache and a bit of a speedup as well.
B.
Your code is extremely useful when dealing with IB's down times. And it's a nice speedup as well.
Just one quick suggestion/bug fix -- I've found that sometimes there's a numpy int64 error generated by line 268 in pandafeed (AttributeError: 'numpy.int64' object has no attribute 'to_pydatetime') due to the fact that some datetimes are stored in int64 in pandas and need to be converted manually. The line I've added to alleviate this is:
try:
# check if we have this cached
self._df = pd.read_csv(filename,
parse_dates=True,
index_col=0)
self._df.index = pd.to_datetime(self._df.index, format='%Y-%m-%dT%H:%M:%S')
More information on this datetime conversion can be found here: https://community.backtrader.com/topic/676/bug-using-pandas-hdf
Again, great work brettelliot.
@backtrader My pleasure Daniel - I'm probably being too cagey with the above so here's something that might (or might not) help you and what I am working towards. The below is my use of a zigzag indicator with very minimal filtering to identify wavelet inflection points at various increasing frequencies (decomposition level edge detection):
So far so good.
As well you can specify a minimum threshold for the current bar minus the last bar to qualify as being recognized as a minimum, as well as a threshold for the minimum distance to the last peak/valley. Naturally, as in the case below, if you set these thresholds to non-zero values you can have peaks without valleys and valleys without peaks in some instances.
If you find a bug please let me know and any feedback is appreciated.
And here's a screenshot of this on a randomly (and poorly) selected SMA and the thresholds were just set arbitrarily just to show an example:
Re: Zigzag indicator
The previously posted zigzag study above was quite useful and my compliments to the authors. I needed something slightly different, however, more suitable for real-time identification and so here's a slightly tighter version that's about 80% smaller (from 100 lines of code down to about 18).
This zigzag indicator is more of an indicator than a study and therefore it identifies peaks one bar "late" which is the only way to behave for real time peak identification. Most importantly, it should (in theory) not set peaks/valleys retroactively. It does not offer the various lines of the other zigzag -- I had no need to know the bars since the last peak or valley but these can be added after the fact in a single line by counting a list comprehension or by using a ternary iteration.
'''
Author: B. Bradford
MIT License
Copyright (c) B. Bradford
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''
class bbzigzag(bt.Indicator):
plotinfo = dict(subplot=True,
zigzag=dict(_name='zigzag', color='lightblue', ls='--', _skipnan=True), )
plotlines = dict(
zigzag_peak=dict(marker='v', markersize=4.0, color='red', fillstyle='full', ls=''),
zigzag_valley=dict(marker='^', markersize=4.0, color='red', fillstyle='full', ls=''),
zigzag=dict(_name='zigzag', color='red', ls='--', _skipnan=True),
)
params = (
('up_retrace', 0.015),
('dn_retrace', 0.015),
('bardist', 0.015), # distance to last max/min (perc/100)
('plotdistance', 0.03), #distance to plot arrows (alters high/low indicator lines but not zigzag line)
)
lines = ('zigzag',
'zigzag_peak',
'zigzag_valley')
def __init__(self):
self.setminperiod(2)
self.pks = [0]
def once(self, start, end):
lstinputdata = self.datas[0].array[:]
self.trnd = [0 if lstinputdata[i] == lstinputdata[i-1] else 1 if lstinputdata[i-1] > lstinputdata[i] else -1 for i in range(1, len(lstinputdata))]
self.nogaps = [self.trnd[i-1] if self.trnd[i] == 0 else self.trnd[i] for i in range(len(self.trnd))]
nogapslast=self.nogaps[0]
for i in range(len(self.nogaps)):
if nogapslast != self.nogaps[i]: # peak found
if abs(lstinputdata[i] - self.pks[-1]) > self.params.bardist * self.pks[-1]:
self.pks.append(lstinputdata[i])
idx = -(len(self.nogaps)-i-1)
absplotdist = lstinputdata[i] * self.params.plotdistance
if self.nogaps[i] > nogapslast: self.lines.zigzag_peak[idx] = lstinputdata[i] + absplotdist
if self.nogaps[i] < nogapslast: self.lines.zigzag_valley[idx] = lstinputdata[i] - absplotdist
self.lines.zigzag[idx] = (self.zigzag_peak[idx] - absplotdist) if self.zigzag_peak[idx] == self.zigzag_peak[idx] else (self.zigzag_valley[idx] + absplotdist) if self.zigzag_valley[idx] == self.zigzag_valley[idx] else float('NaN')
nogapslast = self.nogaps[i]
@backtrader Thanks -- in the same vein I greatly admire your work on Backtrader. It's excellent.
Fixed a bug when an odd combination of parameters is supplied. This is the same chart with haar and no incremental option. Blocky, but exactly the same concept:
And a screenshot:
This screenshot shows the wavelet indicator as "testindicator2" plotted over the main data series. You can see that the lower frequency band makes a good long term indicator in a trending market. As the markets contract into choppiness, you may switch to a higher resolution wavelet or a smaller coefficient level which results in a shorter length of input series data used.
I'm considering making a daubechies wavelet indicator with lifting which certain people are likely highly interested in but because of the enormous amount of time involved I might charge. I'm sure someone will find a bug or two and any feedback is appreciated.
Here's an indicator (Haar wavelet with lifting) which, similar to the other wavelet implementations I posted, can essentially break a time series down to constituent frequency components. However because this is implemented via lifting it reduces boundary effects To be more compliant with Daniel R.'s vision, this does not use numpy and all calculations are done internally with no libraries.
This can be used as a crossover indicator for someone who wants an off the shelf solution. For the more advanced user I've also implemented basic thresholding of the wavelet coefficients (threshlist), or you can pick out a wavelet level in isolation (coeffpick).
Further, it also has an incremental flag which although very computationally expensive (an inefficiency that I may refine later on) shows the current bar value without computing from future bars. I don't know that this really makes as much of a difference in practice as an indicator, however.
This is implemented in Python 3.
To call:
examplelwt = bbhaarlift(self.data, coefflev=9, coeffpick = 3, blnincremental = True, threshlist=[1, 1, 1, 1, 0, 0, 0, 0, 0])
Where coeffpick and threshlist are mutually exclusive use one or the other. If both are provided, coeffpick takes precedence. The above picks out the third wavelet level out of nine total.
Because this uses lifting, 2**n bars are needed for calculation. If coefflev isn't provided, the routine uses the next lowest power of 2 bars. Ie. all the bars it can given your input data length.
And here's the code:
'''
Author: B. Bradford adapted from information provided by
bearcave.com and Ian Kaplan.
MIT License
Copyright (c) B. Bradford
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''
import backtrader as bt
# input lengths must be a power of two for haar with lifting:
class bbhaarlift(bt.Indicator):
plotinfo = dict(subplot=False)
lines = ('haarlift', )
params = (('coefflev', None),
('coeffpick', None),
('blnincremental', False),
('threshlist', [1])) #2**coefflev data points required or 2**coefflev
@staticmethod
def next_power_of_2(x):
return 1 if x == 0 else 2 ** (x.bit_length() -1)
def once(self, start, end):
self.blnforward = 1
self.blninverse = 2
#create dummy line to manipulate and desired 2** data length (max or selected)
lstinputdata = self.data.array[:] #make a copy of the list/array
if self.params.coefflev is None: #max available data length
pow2padding = self.next_power_of_2(len(lstinputdata)) #input needs to be power of two data points
else: #selected 2** from coefflev
pow2padding = 2 ** self.params.coefflev
if self.next_power_of_2(len(lstinputdata)) < pow2padding: raise ValueError("COEFF LEVEL %s GREATER THAN MAX DATA LENGTH %s AVAILABLE" % (self.params.coefflev, self.next_power_of_2(len(lstinputdata))))
if self.params.coefflev is None:
self.params.coefflev = pow2padding.bit_length() - 1
# If specific coeffpick selected set it to one, and others to zero:
if self.params.coeffpick is not None:
self.params.threshlist = [1] * self.params.coeffpick + [0] * (
self.params.coefflev - self.params.coeffpick)
dpcnt = len(lstinputdata) - pow2padding #only do loop once at end of array if blnincremental not true
#Count backwards through line to leave most recent value in place if incremental:
while dpcnt > 0:
dpcnt -= 1 #decrement immediately as list slicing is zero-based
inlist = self.data.array[dpcnt:dpcnt + pow2padding]
ftresult = bbhaarlift.forwardtrans(self, vec=inlist) #forward transform
threshedresult = bbhaarlift.wvltthreshold(coeffary=ftresult, threshlist=self.params.threshlist)
itresult = bbhaarlift.inversetrans(self, vec=threshedresult) #inverse transform
dummyaryout = lstinputdata[:dpcnt] + itresult + lstinputdata[dpcnt + pow2padding:]
lstinputdata = dummyaryout # self.lines[0].array[pow2padding:] + itresult
#Only run once if not doing incremental:
if not self.params.blnincremental: break
self.lines[0].array = lstinputdata
def wvltthreshold(coeffary, threshlist):
print ("bbwaveletlifting: len(coeffary).bit_length()", len(coeffary).bit_length())
if len(coeffary).bit_length() < len(threshlist):
raise ValueError('WARNING: THRESHOLD LIST LENGTH IS SMALLER THAN NUMBER OF COEFF LEVELS.')
threshedresult = coeffary[:]
for i in range(1, len(threshlist)):
for x in range(2**i, 2**(i+1)):
print ("bbwaveletlifting: i, x, len(threshlist) ", i, x, len(threshlist))
threshedresult[x] = coeffary[x] * threshlist[i]
return threshedresult
def split(vec, N):
start = 1
end = N - 1
while (start < end):
#Could do this with list comprehension, but want to stick to the original java structure:
for i in range(start, end, 2):
tmp = vec[i]
vec[i] = vec [i + 1]
vec[i+1] = tmp
start += 1
end -= 1
return vec
def merge(self, vec, N):
half = N >> 1
start = half-1
end = half
while (start > 0):
for i in range(start, end, 2):
tmp = vec[i]
vec[i] = vec[i+1]
vec[i+1] = tmp
start -= 1
end += 1
def forwardtrans(self, vec):
'''
The result of forwardTrans is a set of wavelet coefficients
ordered by increasing frequency and an approximate average
of the input data set in vec[0]. The coefficient bands
follow this element in powers of two (e.g., 1, 2, 4, 8...).
'''
N = len(vec)
print ("forwardtrans N: ", N)
#set for counter:
n = N
print ("forwardtrans n: ", n)
while n > 1:
print ("forwardtrans split...")
vec = bbhaarlift.split(vec, n)
print("forwardtrans predict...")
vec = bbhaarlift.predict (self, vec, n, self.blnforward)
print("update split...")
vec = bbhaarlift.update (self, vec, n, self.blnforward)
n = n >> 1
return vec
def inversetrans(self, vec):
N = len(vec)
n = 2
while n <= N:
bbhaarlift.update(self, vec, n, self.blninverse)
bbhaarlift.predict (self, vec, n, self.blninverse)
bbhaarlift.merge(self, vec, n)
n = n << 1
return vec
def predict(self, vec, N, direction):
half = N >> 1
cnt = 0
for i in range (0, half, 1):
predictval = float(vec[i])
j = int(i+half)
if (direction == self.blnforward):
vec[j] = vec[j] - predictval
elif (direction == self.blninverse):
vec[j] = vec[j] + predictval
else:
debug.print("haar: predict: bad direction value")
return vec
def update(self, vec, N, direction):
half = N >> 1
for i in range (0, half, 1):
j = int(i + half)
updateVal = float(vec[j] / 2.0)
if (direction == self.blnforward):
vec[i] = vec[i] + updateVal
elif (direction == self.blninverse):
vec[i] = vec[i] - updateVal
else:
debug.print("haar update: bad direction value")
return vec
@backtrader said in Optimization: Can you reproduce this bug?:
Fascinating difference and gracias. I altered the code and it now works, as below (the original code is from https://backtest-rookies.com/2017/06/26/optimize-strategies-backtrader/ )
import backtrader as bt
import datetime
class firstStrategy(bt.Strategy):
params = (
('period', 21),
)
def __init__(self):
self.startcash = self.broker.getvalue()
self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period)
def next(self):
if not self.position:
if self.rsi < 30:
self.buy(size=100)
else:
if self.rsi > 70:
self.sell(size=100)
def stop(self):
self.value = round(self.broker.get_value(), 2)
print(bt.__version__)
sunderlying = ["YHOO"]
print("UNDERLYING:", sunderlying)
#Variable for our starting cash
startcash = 10000
#Create an instance of cerebro
cerebro = bt.Cerebro(optreturn=False, maxcpus=1)
#Add our strategy
cerebro.optstrategy(firstStrategy, period=range(14,21))
#Get Apple data from Yahoo Finance.
data = bt.feeds.YahooFinanceData(
dataname='AAPL',
fromdate = datetime.datetime(2016,1,1),
todate = datetime.datetime(2017,1,1),
buffered= True
)
cerebro.adddata(data)
# Set our desired cash start
cerebro.broker.setcash(startcash)
# Run over everything
opt_runs = cerebro.run()
# Generate results list
final_results_list = []
for run in opt_runs:
for strategy in run:
value = strategy.value
PnL = round(value - startcash,2)
period = strategy.params.period
final_results_list.append([period,PnL])
#Sort Results List
by_period = sorted(final_results_list, key=lambda x: x[0])
by_PnL = sorted(final_results_list, key=lambda x: x[1], reverse=True)
#Print results
print('Results: Ordered by period:')
for result in by_period:
print('Period: {}, PnL: {}'.format(result[0], result[1]))
print('Results: Ordered by Profit:')
for result in by_PnL:
print('Period: {}, PnL: {}'.format(result[0], result[1]))
The output is now correct:
/usr/bin/python3.6 /docker_stocks/DELETETEMP.py
1.9.64.122
UNDERLYING: ['YHOO']
Results: Ordered by period:
Period: 14, PnL: 2062.0
Period: 15, PnL: 2079.0
Period: 16, PnL: 1718.0
Period: 17, PnL: 2141.0
Period: 18, PnL: 2002.0
Period: 19, PnL: 1451.0
Period: 20, PnL: 1883.0
Results: Ordered by Profit:
Period: 17, PnL: 2141.0
Period: 15, PnL: 2079.0
Period: 14, PnL: 2062.0
Period: 18, PnL: 2002.0
Period: 20, PnL: 1883.0
Period: 16, PnL: 1718.0
Period: 19, PnL: 1451.0
Process finished with exit code 0