Wavelet bandpass indicator for backtrader

Hi,
Here's a wavelet bandpass filter indicator using the pywt (python wavelets) library. I haven't found the backtrader_contrib github so I'll just leave this here in case anyone wants to use it. GPL3. No support, warranties, either expressed or implied. No guarantee that this will have efficacy for your particular application.
With minimal (or no) changes this should support:
Haar (haar)
Daubechies (db)
Symlets (sym)
Coiflets (coif)
Biorthogonal (bior)
Reverse biorthogonal (rbio)
“Discrete” FIR approximation of Meyer wavelet (dmey)
Gaussian wavelets (gaus)
Mexican hat wavelet (mexh)
Morlet wavelet (morl)
Complex Gaussian wavelets (cgau)
Shannon wavelets (shan)
Frequency BSpline wavelets (fbsp)
Complex Morlet wavelets (cmor)Good luck,
B. Bradford#!/usr/bin/env python # * coding: utf8; pyindentoffset:4 * ############################################################################## # # Copyright (C) 2017 Benjamin Bradford (bigdavediode2 (at ) yahoo.com) # Wavelet deconstruction to specified subband/level # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. # ############################################################################### from __future__ import (absolute_import, division, print_function, unicode_literals) import backtrader.indicators as btind import backtrader as bt import pywt class bbDWT(bt.Indicator): # Approximation, detail coefficient lines if wanted: lines = ('cA2', 'cD2', 'cD1', ) params = (('wavelettype', 'db2'), ('coefflevels', 5), ('coeffpick', 0), ) def once(self, start, end): super(bbDWT, self).__init__() lstinputdata = self.data.array coeffs = pywt.wavedecn(lstinputdata, self.params.wavelettype, level=self.params.coefflevels) # Use if you wish to graph spectrum bands/coefficients: # coeffcA2, coeffcD2, coeffcD1 = coeffs # print(coeffs) print(len(coeffs)) arr, coeff_slices = pywt.coeffs_to_array(coeffs) print(coeff_slices) # Zero out all but selected wavelet coeff level (avoids having to pad arrays): for i in range(0, len(coeff_slices)): if i != self.params.coeffpick: #Remove the "slice" portion of the returned slices up to the bracket to run as a slice command: s1 = str(coeff_slices[i]) s1 = s1[s1.find('slice'):s1.find(')')+1] execstring = 's1 = ' + s1 print(execstring) exec(execstring) arr[s1] = 0 print(arr) coeffs_from_array = pywt.array_to_coeffs(arr, coeff_slices) dummy_array = pywt.waverecn(coeffs_from_array, self.params.wavelettype) self.lines.cA2.array = dummy_array

@bigdavediode said in Wavelet bandpass indicator for backtrader:
wavelet bandpass filter
This looks really interesting. I'll be honest I never even heard of this before. I've done a bit of googling around the subject just now. Could you explain how this could be used in trading? Cheers mate

Hi,
You can, with minimal changes to the code, use it in two ways: smoothing of the function by zeroing out certain frequency bands (a bandpass filter and thus noise reduction), or like I've implemented it here to extract any particular frequency band and chart it. If you're a shortterm trader you would, perhaps, extract highfrequency bands. If you're more long term then you select those longerterm frequencies.
Good luck,
B. 
@bigdavediode hmm..When you saying smoothing of function. Could I use to eg with price as input to look for cycles? Or would I apply this to e.g. an indicator to again look for key frequencies within that. Any chance you could upload a simple example demonstrating? Perhaps with a visulisation/plot before and after? Sounds highly interesting.

@RichardO'Regan Yes, you could smooth the price by zeroing out the highfrequency bands (noise) by changing the "for" loop if from != to >. This would remove the highest frequencies. This might reveal (as an example) low frequency seasonal cycles.
Choose your wavelet type carefully (perhaps start with Haar wavelets) and also please keep in mind that low frequencies have lower "temporal resolution"  ie. they are less timespecific, so if you are looking for tomorrow's signals with EOD data inputted, low frequency bands will have "leaks" of the next day's completed data into the previous day's signal. That wouldn't happen (as significantly) with higher frequencies more appropriate to daily prediction. Translation: use the right tool for the job.
Here's a simple example strategy:
import backtrader.indicators as btind from bbwavelet import bbDWT class firstStrategy(bt.Strategy): def __init__(self): self.dwt = bbDWT(self.data, wavelettype='haar', coefflevels = 3, coeffpick = 1)

@bigdavediode Could you kindly suggest some introductory material about wawelets? That would help me in the effort of understanding how to use your code. Many thanks in advance, and thank you for sharing your work.

@lvaghi Mathlab has some good introductory videos: https://www.youtube.com/watch?v=QX1xGVFqmw
I found this site interesting, but I haven't read it entirely: http://www.bearcave.com/misl/misl_tech/wavelets/
One thing that is a benefit with wavelets is their orthogonality. Information in one signal level is, for the most part, independent of the other signal bands.
B.

heya Dave, just invested an hour reading your links. Interesting stuff. I think there is probably a number of people in this forum also interested in this subject. I have a bit of an overview from reading up on this, though I feel it would take me weeks to get up to speed with this subject! I'm wondering how money could be made using these techniques, i.e. where is the edge?
 Perhaps I could look for reversions of actual price with the filtered price.. e.g. when filtered price was less than market price sell market etc..
 Or perhaps I could use filtered line to do trend calculations in the hope that I could predict trend a bit quicker before others in market did ..catch a move earlier etc.
You have given some code examples on how to use, if you have time, would you be able to provide a basic visual example e.g. a line of close of a stock and the wavelet filtered line, and what the line means . etc? Would be really helpful to see something visual  aids understanding).
Cheers mate
Rich 
@RichardO'Regan Any type of indicator that you use a moving average for, crossover, for example, results in a loss of information from the original signal, autocorrelation, and lag. https://www.researchgate.net/post/What_are_the_disadvantages_of_moving_average_filter_when_using_it_with_time_series_data Wavelets address some of these issues if used properly.
I recommend deciding what your average hold time for a trade is (seconds, minutes, days) and use wavelet levels that correspond to the frequency that you are interested in to examine if there is a cyclical nature at the frequency that you trade within.
 depends on whether you are a meanreversion trader and 2) depends on whether you are trend following. Whatever your predilections are wavelets can remove the noise definitely better than MA's and, imo, more usefully than FFT's.
Hope that helps,
B. 
@RichardO'Regan You requested sample code to test the indicator with  enjoy:
Author: www.backtestrookies.com with small changes to chart wavelet by B. Bradford MIT License Copyright (c) 2017 backtestrookies.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' import backtrader as bt from datetime import datetime import backtrader.indicators as btind from bbwavelet import bbDWT class firstStrategy(bt.Strategy): def __init__(self): # self.rsi = bt.indicators.RSI_SMA(self.data.close, period=21) # self.dwt = bbDWT(self.data, wavelettype='haar', coefflevels = 5, coeffpick = 4) # Variable for our starting cash startcash = 10000 # Create an instance of cerebro cerebro = bt.Cerebro() # Add our strategy cerebro.addstrategy(firstStrategy) # Get Apple data from Yahoo Finance. data = bt.feeds.YahooFinanceData( dataname='AAPL', fromdate=datetime(2017, 4, 1), todate=datetime(2017, 6, 10), buffered=True ) # Add the data to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(startcash) # Run over everything cerebro.run() # Get final portfolio Value portvalue = cerebro.broker.getvalue() pnl = portvalue  startcash # Print out the final result print('Final Portfolio Value: ${}'.format(portvalue)) print('P/L: ${}'.format(pnl)) # Finally plot the end results cerebro.plot(style='candlestick')

Hey thanks very much for that Dave.
I had a runtime crash. Rather than duplicate code, I sent screenshots. Run in a Jupyter Notebook. instead of importing bbwavelet.py module, i just copied and pasted your forum code into the notebook, so that bbDWT was already defined. 
Hi Richard  couple of ideas just off the cuff: one is that I'm running numpy1.13.1, another is that whatever dataset you're running pywt on is returning an insufficiently sized data set or nulled out slice command (it looks fine as I glance at it though). I expect you are using python 2.7, of course.
Rather than reinventing the wheel you're welcome to use my public docker containers  the third includes pywt, pycharm, etc.: https://hub.docker.com/r/bigdavediode/pycharm_backtrader3/
Perhaps I should make these containers a separate post as they may be useful to others.

@RichardO'Regan  I have no idea if this would work as I have not tested it, but slice(x) is equivalent to the slice :x, and slice(x, y, z) is x:y:z . Perhaps if you pull out s1 to the equivalent slice(x) that would work for you. All this line does is to rather brutally threshold the particular wavelet component to zero. I do not use Jupyter and have no (convenient) way of testing under the older version of numpy that I presume you are using but it does work fine under the current version.

Here's a working version of wavelet highpass, lowpass and bandpass filters using either the discrete wavelet transform or MODWT (requires https://github.com/pistonly/modwtpy/blob/master/modwt.py).
Each indicator includes a Incremental version, which incrementally builds up the values instead of running the transform on the full data to avoid leaking back data since in live trading you would not have more data available.
import operator import backtrader as bt import pywt import numpy as np from analysis.modwt import modwt, imodwt class _DwtBase(bt.Indicator): plotinfo = dict(subplot=False) lines = ('pass',) params = ( ('wavelet', 'db2'), ('levels', 8), ('filter', 6), ('extension', 'smooth'), ) def __init__(self): self.wavelet = pywt.Wavelet(self.p.wavelet) level_max = pywt.dwt_max_level(len(self.data.array), self.wavelet) self.levels = self.p.levels if self.levels > level_max: self.levels = level_max def _dwt(self, data): coeffs = pywt.wavedecn(data, wavelet=self.wavelet, level=self.levels, mode=self.p.extension) arr, slices = pywt.coeffs_to_array(coeffs) # Zero out all but selected wavelet coeff level for i in range(0, len(slices)): if self.op(i, self.params.filter): if 'd' in slices[i]: s1 = slices[i]['d'][0] else: s1 = slices[i][0] arr[s1] = 0 coeffs = pywt.array_to_coeffs(arr, slices) return pywt.waverecn(coeffs, wavelet=self.wavelet, mode=self.p.extension) class _ModWtBase(bt.Indicator): plotinfo = dict(subplot=False) lines = ('pass',) params = ( ('wavelet', 'db2'), ('levels', 8), ('filter', 6), ) def __init__(self): self.wavelet = pywt.Wavelet(self.p.wavelet) self.levels = self.p.levels def _extend(self, data): data = np.asarray(data) ext = data[::1] return np.concatenate((ext, data, ext)) def _unextend(self, data): data = np.asarray(data) slen = len(data) // 3 return data[slen:2*slen] def _modwt(self, data): coeffs = modwt(self._extend(data), wavelet=self.wavelet, level=self.levels) for i in range(0, len(coeffs)): if self.op(i, self.params.filter): coeffs[i1][:] = 0 return self._unextend(imodwt(coeffs, wavelet=self.wavelet)) class _DwtPassFilter(_DwtBase): def oncestart(self, start, end): pass def once(self, start, end): self.lines[0].array = self._dwt(self.data.array) class _ModWtPassFilter(_ModWtBase): def oncestart(self, start, end): pass def once(self, start, end): self.lines[0].array = self._modwt(self.data.array) class _DwtIncrementalPassFilter(_DwtBase): def __init__(self): super().__init__() self.data_ = [] def next(self): self.data_.append(self.data[0]) try: rdata = self._dwt(self.data_) except ValueError: return self.lines[0][0] = rdata[1] class _ModWtIncrementalPassFilter(_ModWtBase): def __init__(self): super().__init__() self.data_ = [] def next(self): self.data_.append(self.data[0]) try: rdata = self._modwt(self.data_) except ValueError: return self.lines[0][0] = rdata[1] class DwtLowPassFilter(_DwtPassFilter): ''' Discrete wavelet transform lowpass filter. ''' def __init__(self): self.op = operator.ge super().__init__() class DwtIncrementalLowPassFilter(_DwtIncrementalPassFilter): ''' Discrete wavelet transform lowpass filter. ''' def __init__(self): self.op = operator.ge super().__init__() class DwtHighPassFilter(_DwtPassFilter): ''' Discrete wavelet transform highpass filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.le super().__init__() class DwtIncrementalHighPassFilter(_DwtIncrementalPassFilter): ''' Discrete wavelet transform highpass filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.le super().__init__() class DwtBandFilter(_DwtPassFilter): ''' Discrete wavelet transform band filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.ne super().__init__() class DwtIncrementalBandFilter(_DwtIncrementalPassFilter): ''' Discrete wavelet transform band filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.ne super().__init__() class ModWtLowPassFilter(_ModWtPassFilter): ''' Maximum overlap discrete wavelet transform lowpass filter. ''' def __init__(self): self.op = operator.ge super().__init__() class ModWtIncrementalLowPassFilter(_ModWtIncrementalPassFilter): ''' Maximum overlap discrete wavelet transform lowpass filter. ''' def __init__(self): self.op = operator.ge super().__init__() class ModWtHighPassFilter(_ModWtPassFilter): ''' Maximum overlap discrete wavelet transform highpass filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.le super().__init__() class ModWtIncrementalHighPassFilter(_ModWtIncrementalPassFilter): ''' Maximum overlap discrete wavelet transform highpass filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.le super().__init__() class ModWtBandFilter(_ModWtPassFilter): ''' Maximum overlap discrete wavelet transform band filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.ne super().__init__() class ModWtIncrementalBandFilter(_ModWtIncrementalPassFilter): ''' Maximum overlap discrete wavelet transform band filter. ''' plotinfo = dict(subplot=True) def __init__(self): self.op = operator.ne super().__init__()

Cool  thanks!

@bigdavediode  Would you mind sharing the dockerfile you used for those docker images? I am trying to get this running on docker and would like to see what you did.
Many thanks in advance. Feel free to point me to a github repo if there's one.

@mcr1974 https://hub.docker.com/r/bigdavediode/pycharm_backtrader3/
No dockerfile  just built it and pushed. Adapted from mbabik/backtrader and just layered on everything from aptget to git to MatPlotlib, tcl/tk, Tkinter, pycharm, and a bunch of other necessary includes.
Just fyi, pycharm is not a workable IDE for multicore support in backtrader. Use it for development, then IDLE or something else for optimization.

Momentarily confused when trying out Kaber's code  for those who care my code is python 2, and his code appears to be python 3. Changing super() to super(classname, self) successfully executes under python 2.