Problem with Trade @ 0 and multi time series
-
Hi,
As I mainly trade spreads on Commodities, I had to find a way to load several time series per commodity. So after some work, I ended up with a setup where for each future I have a chain that is formed of several time series (all of them have different start/end times, one time serie for each spread). I also load first a dummy reference as the first serie to help with the synchronization of all the time series (the dummy reference is a time serie from the minimum start to the maximum end of all the time series).
This seems to work pretty well, except that a couple of days ago, while doing some testing I ran into a weird issue. Spreads are financial instruments that can be negative, positive or equal to zero, so I need to be able to trade at any price. Which seems to be working fine when loading only one time serie, but a trade a zero on the second serie doesn't seem to work as expected (or I am missing something).
See below my code (can be ran as is with the data - see link in the code):
import datetime as dt import os import pandas as pd import backtrader as bt # OneDrive Link: https://1drv.ms/f/s!Ai-XKOnv2eoFg9Fyyo_asLi9_EapqQ # Change below to where the data is: root = '.\\' def business_date_range(start, end): """From a start and end date, returns a list of business days dates :param start: str/datetime - Start Date :param end: str/datetime - End Date :return: list - Business Dates """ # Convert to datetime if not isinstance(start, dt.datetime): start = dt.datetime.strptime(start, '%Y-%m-%d') if not isinstance(end, dt.datetime): end = dt.datetime.strptime(end, '%Y-%m-%d') return [d for d in [start + dt.timedelta(days=i) for i in range((end - start).days + 1)] if d.weekday() < 5] def create_reference_data(start, end): """Create a dummy file to contains dates for the whole backtest - this is used as a reference for the backtest when not all time series are aligned properly. :param start: str/datetime - Start Date :param end: str/datetime - End Date """ path = get_file_path('Reference') # Go through all the dates dates = business_date_range(start, end) ld = [] fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI'] for d in dates: ld.append({'Date': d, 'Open': 1, 'High': 2, 'Low': 3, 'Close': 4, 'Volume': 5, 'OI': 6}) # Create DataFrame df = pd.DataFrame(ld) df = df.set_index('Date', drop=True) df.index.names = [None] df = df[fields] # Save DataFrame df.to_csv(path, header=False) return df def get_market_df(ticker, start_date=None, end_date=None): """From a ticker get the data from the data repository and store into a dataframe :param ticker: Customized ticker :param start_date: Optional parameter start date :param end_date: Optional parameter start date :return: Dataframe containing the data or None if there's nothing in the database """ print('Get dataframe for: {}'.format(ticker)) file = get_file_path(ticker) if os.path.exists(file): fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI'] # Read Data df = pd.read_csv(file, sep=',', parse_dates=True, header=None, names=fields) # Remove week end data df = df[df.index.dayofweek < 5] # Check if data needs to be truncated if start_date is not None: df = df[df.index >= start_date] if end_date is not None: df = df[df.index <= end_date] # Check if missing data if df.isnull().values.any(): print('Missing data in: {} - Count: {}, please check!'.format(ticker, df.isnull().sum().sum())) return df else: return None def get_file_path(ticker): """Get data path for the ticker (Also returns the path for the dummy reference). :param ticker: str - Customized ticker :return: str - Path """ if ticker == 'Reference': path = os.path.join(root, 'Reference.txt') else: path = os.path.join(root, '{}.txt'.format(ticker)) return path class TradeZero(bt.Strategy): """Test TradeZero to identify if we can trade at 0!""" def __init__(self): # Internal Strategy Variables self.date = self.datas[0].datetime.date # Reference self.order = None # To keep track of pending orders self.names = self.getdatanames() # Exit self.exit = dt.date(2009, 6, 11) def log(self, level, message): print('{} - {} - {}'.format(level, self.date(0), message)) def notify_order(self, order): name = order.data._name if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed if order.status in [order.Completed]: side = 'Buy' if order.isbuy() else 'Sell' message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}' self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm)) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name)) self.order = None def notify_trade(self, trade): name = trade.data._name if not trade.isclosed: return self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm)) def next(self): if self.order: return # Initialize Data dta = self.datas[0] # Debugging # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0])) name, pos = self.names[0], self.getpositionbyname(self.names[0]) # Entry if not pos and self.date(0) == dt.date(2009, 2, 3): self.log('INFO', '{} - Entry on first bar!'.format(name)) if self.date(0).day % 2 == 0: # Sell on even days self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0])) self.order = self.order_target_size(data=dta, target=-1) else: self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0])) self.order = self.order_target_size(data=dta, target=1) # Final Exit if pos and self.date(0) == self.exit: self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0])) self.order = self.close(data=dta, exectype=bt.Order.Close) class TradeZero2(bt.Strategy): """Test TradeZero to identify if we can trade at 0!""" def __init__(self): # Internal Strategy Variables self.date = self.datas[0].datetime.date # Reference self.order = None # To keep track of pending orders self.names = self.getdatanames() del self.names[0] # Remove Reference self.log('INFO', 'Symbols: {}'.format(self.names)) # Exit self.exit = dt.date(2009, 6, 11) def log(self, level, message): print('{} - {} - {}'.format(level, self.date(0), message)) def notify_order(self, order): name = order.data._name if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed if order.status in [order.Completed]: side = 'Buy' if order.isbuy() else 'Sell' message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}' self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm)) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name)) self.order = None def notify_trade(self, trade): name = trade.data._name if not trade.isclosed: return self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm)) def prenext(self): self.next() def next(self): if self.order: return if self.date(0) == dt.date(2009, 2, 3): self.log('DEBUG', 'This is where the problem is!') self.log('DEBUG', '{} {}'.format(self.datas[1].close[0], self.names[0])) # Initialize Data dta = self.datas[1] # Debugging # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0])) name, pos = self.names[0], self.getpositionbyname(self.names[0]) # Entry if not pos and self.date(0) == dt.date(2009, 2, 3): self.log('INFO', '{} - Entry on first bar!'.format(name)) if self.date(0).day % 2 == 0: # Sell on even days self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0])) self.order = self.order_target_size(data=dta, target=-1) else: self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0])) self.order = self.order_target_size(data=dta, target=1) # Final Exit if pos and self.date(0) == self.exit: self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0])) self.order = self.close(data=dta, exectype=bt.Order.Close) def test_trade_zero(ticker, multi): cash = 100000.0 # Setup Cerebro cerebro = bt.Cerebro() cerebro.broker.setcash(cash) # Data if multi: data = bt.feeds.PandasData(dataname=create_reference_data('2008-06-18', '2009-06-15')) cerebro.adddata(data, name='Reference') df = get_market_df(ticker) data = bt.feeds.PandasData(dataname=df) cerebro.adddata(data, name=ticker) comms = bt.CommissionInfo(commission=2.75, margin=950, mult=400) cerebro.broker.addcommissioninfo(comms, name=ticker) # Strategy cerebro.addstrategy(TradeZero2 if multi else TradeZero) # Backtest cerebro.run() if __name__ == '__main__': test_trade_zero('LHS1M09', False) test_trade_zero('LHS1M09', True) test_trade_zero('LHS2M09', True)Now, there are 3 cases in my test code:
1. Loading only one time serie of LHS1M09: we can see that the trade at 0 works
2. Loading 2 time series, the Reference and LHS1M09: the trade at 0 doesn't work, the order is submitted on LHS1M09 but executed on the Reference somehow?
3. Loading 2 time series, the Reference and LHS2M09 (which is the same as LHS1M09 except that I change the close price of the day where the trade will take place to a non-zero value): the trade now worksNow below is the output of the 3 cases:
Case 1:
Get dataframe for: LHS1M09 INFO - 2009-02-03 - LHS1M09 - Entry on first bar! INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0 INFO - 2009-02-04 - LHS1M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75 INFO - 2009-06-11 - LHS1M09 - Exit on last bar @ -3.425! INFO - 2009-06-12 - LHS1M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75 INFO - 2009-06-12 - LHS1M09 - Trade PnL - Gross: -1220.00, Net: -1225.50Case 2:
Get dataframe for: LHS1M09 INFO - 2009-06-15 - Symbols: ['LHS1M09'] DEBUG - 2009-02-03 - This is where the problem is! DEBUG - 2009-02-03 - 0.0 LHS1M09 # Close price is 0.00 INFO - 2009-02-03 - LHS1M09 - Entry on first bar! INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0 INFO - 2009-02-04 - Reference - Execution (Buy) - Price: 1.00, Cost: 1.00, Comms: 0.00 # This is where the problem is!Case 3:
Get dataframe for: LHS2M09 INFO - 2009-06-15 - Symbols: ['LHS2M09'] DEBUG - 2009-02-03 - This is where the problem is! DEBUG - 2009-02-03 - 0.05 LHS2M09 # Close price is 0.05 INFO - 2009-02-03 - LHS2M09 - Entry on first bar! INFO - 2009-02-03 - LHS2M09 - Buy Order (Front) @ 0.05 INFO - 2009-02-04 - LHS2M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75 # Now works fine INFO - 2009-06-11 - LHS2M09 - Exit on last bar @ -3.425! INFO - 2009-06-12 - LHS2M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75 INFO - 2009-06-12 - LHS2M09 - Trade PnL - Gross: -1220.00, Net: -1225.50I am not sure if I am missing something obvious or if there is a bug here. Any help would be greatly appreciated.
Thanks!
-
So I have done a bit of debugging and found where the data gets changed to Reference from LHS1M09.
It happens in the following test:
data = data or self.datas[0]Hence why it is not a problem when dealing with only one serie but when dealing with multi-series, we revert back to self.datas[0] which is "Reference" in my case.
Somehow, even though data is a valid time serie, the test seems to return False when Close is equal to 0.0.
See below a test:
self.log('DEBUG', 'Date: {} - Close {} - Test: {}!'.format(self.date(0), dta.close[0], bool(dta)))With the following results:
DEBUG - 2009-01-29 - Date: 2009-01-29 - Close 0.075 - Test: True! DEBUG - 2009-01-30 - Date: 2009-01-30 - Close 0.05 - Test: True! DEBUG - 2009-02-02 - Date: 2009-02-02 - Close -0.175 - Test: True! DEBUG - 2009-02-03 - Date: 2009-02-03 - Close 0.0 - Test: False! DEBUG - 2009-02-04 - Date: 2009-02-04 - Close -0.05 - Test: True! DEBUG - 2009-02-05 - Date: 2009-02-05 - Close -0.25 - Test: True! DEBUG - 2009-02-06 - Date: 2009-02-06 - Close -0.6 - Test: True!I guess this is due to 0.0 evaluated as False... I'll try to change the code to:
data = data if data is not None else self.datas[0]@backtrader: Any ideas on this?
Thanks!
-
@laurent-michelizza said in Problem with Trade @ 0 and multi time series:
data = data or self.datas[0]This is because when you run this code in
next, you are in theStage 2mode and a boolean evaluation of a lines object (data feeds, indicators, observers, ...) uses the current[0]point. This is done by overloading the operators and intended to support code patterns likeclass MyStrategy(bt.Strategy): def __init__(self): self.signal = bt.ind.SMA(self.data) > self.data def next(self): if self.signal: self.buy()The different overloading during Stage 1 and Stage 2 allows for example the lazy evaluation of the logical comparison seen above during
__init__This is documented here: Docs - Platform Concepts (look for Stage 1 and Stage 2)
Using the
isoperator is the way to go.
-
The problem though is that the line of code that I am referencing (see below) is in the backtrader codebase (4 instances in strategy.py) and not in my code as I always reference data explicitly.
data = data or self.datas[0]As a hack, I have amended the line in strategy.py to the below:
data = data if data is not None else self.datas[0]This is not ideal though. Is there a way around it or would you consider amending this piece of code?
PS: Thanks for the link, I will look at it in more details.
-
Indeed. At some point in time there was an attempt to remove those checks (one operator overloading was implemented) Obviously, some of them survived the attempt.
-
Situation which can (probably) only triggered when working with, not so usual, things like spreads (or else the stock/future you have in your portfolio is really worth nothing)