Problem with Trade @ 0 and multi time series

[Laurent Michelizza]

Hi,

As I mainly trade spreads on Commodities, I had to find a way to load several time series per commodity. So after some work, I ended up with a setup where for each future I have a chain that is formed of several time series (all of them have different start/end times, one time serie for each spread). I also load first a dummy reference as the first serie to help with the synchronization of all the time series (the dummy reference is a time serie from the minimum start to the maximum end of all the time series).

This seems to work pretty well, except that a couple of days ago, while doing some testing I ran into a weird issue. Spreads are financial instruments that can be negative, positive or equal to zero, so I need to be able to trade at any price. Which seems to be working fine when loading only one time serie, but a trade a zero on the second serie doesn't seem to work as expected (or I am missing something).

See below my code (can be ran as is with the data - see link in the code):

import datetime as dt
import os
import pandas as pd

import backtrader as bt

# OneDrive Link: https://1drv.ms/f/s!Ai-XKOnv2eoFg9Fyyo_asLi9_EapqQ
# Change below to where the data is:
root = '.\\'


def business_date_range(start, end):
   """From a start and end date, returns a list of business days dates

   :param start: str/datetime - Start Date
   :param end: str/datetime - End Date
   :return: list - Business Dates
   """
   # Convert to datetime
   if not isinstance(start, dt.datetime):
       start = dt.datetime.strptime(start, '%Y-%m-%d')
   if not isinstance(end, dt.datetime):
       end = dt.datetime.strptime(end, '%Y-%m-%d')

   return [d for d in [start + dt.timedelta(days=i) for i in range((end - start).days + 1)] if d.weekday() < 5]


def create_reference_data(start, end):
   """Create a dummy file to contains dates for the whole backtest - this is used as a reference for the backtest
   when not all time series are aligned properly.

   :param start: str/datetime - Start Date
   :param end: str/datetime - End Date
   """
   path = get_file_path('Reference')
   # Go through all the dates
   dates = business_date_range(start, end)
   ld = []
   fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
   for d in dates:
       ld.append({'Date': d, 'Open': 1, 'High': 2, 'Low': 3, 'Close': 4, 'Volume': 5, 'OI': 6})
   # Create DataFrame
   df = pd.DataFrame(ld)
   df = df.set_index('Date', drop=True)
   df.index.names = [None]
   df = df[fields]
   # Save DataFrame
   df.to_csv(path, header=False)

   return df


def get_market_df(ticker, start_date=None, end_date=None):
   """From a ticker get the data from the data repository and store into a dataframe

   :param ticker: Customized ticker
   :param start_date: Optional parameter start date
   :param end_date: Optional parameter start date
   :return: Dataframe containing the data or None if there's nothing in the database
   """
   print('Get dataframe for: {}'.format(ticker))
   file = get_file_path(ticker)
   if os.path.exists(file):
       fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
       # Read Data
       df = pd.read_csv(file, sep=',', parse_dates=True, header=None, names=fields)
       # Remove week end data
       df = df[df.index.dayofweek < 5]
       # Check if data needs to be truncated
       if start_date is not None:
           df = df[df.index >= start_date]
       if end_date is not None:
           df = df[df.index <= end_date]
       # Check if missing data
       if df.isnull().values.any():
           print('Missing data in: {} - Count: {}, please check!'.format(ticker, df.isnull().sum().sum()))
       return df
   else:
       return None


def get_file_path(ticker):
   """Get data path for the ticker (Also returns the path for the dummy reference).

   :param ticker: str - Customized ticker
   :return: str - Path
   """
   if ticker == 'Reference':
       path = os.path.join(root, 'Reference.txt')
   else:
       path = os.path.join(root, '{}.txt'.format(ticker))

   return path


class TradeZero(bt.Strategy):
   """Test TradeZero to identify if we can trade at 0!"""

   def __init__(self):
       # Internal Strategy Variables
       self.date = self.datas[0].datetime.date  # Reference
       self.order = None  # To keep track of pending orders
       self.names = self.getdatanames()
       # Exit
       self.exit = dt.date(2009, 6, 11)

   def log(self, level, message):
       print('{} - {} - {}'.format(level, self.date(0), message))

   def notify_order(self, order):
       name = order.data._name
       if order.status in [order.Submitted, order.Accepted]:
           # Buy/Sell order submitted/accepted to/by broker - Nothing to do
           return
       # Check if an order has been completed
       if order.status in [order.Completed]:
           side = 'Buy' if order.isbuy() else 'Sell'
           message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
           self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
       elif order.status in [order.Canceled, order.Margin, order.Rejected]:
           self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))

       self.order = None

   def notify_trade(self, trade):
       name = trade.data._name
       if not trade.isclosed:
           return
       self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))

   def next(self):
       if self.order:
           return
       # Initialize Data
       dta = self.datas[0]
       # Debugging
       # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
       name, pos = self.names[0], self.getpositionbyname(self.names[0])
       # Entry
       if not pos and self.date(0) == dt.date(2009, 2, 3):
           self.log('INFO', '{} - Entry on first bar!'.format(name))
           if self.date(0).day % 2 == 0:  # Sell on even days
               self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=-1)
           else:
               self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=1)

       # Final Exit
       if pos and self.date(0) == self.exit:
           self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
           self.order = self.close(data=dta, exectype=bt.Order.Close)


class TradeZero2(bt.Strategy):
   """Test TradeZero to identify if we can trade at 0!"""

   def __init__(self):
       # Internal Strategy Variables
       self.date = self.datas[0].datetime.date  # Reference
       self.order = None  # To keep track of pending orders
       self.names = self.getdatanames()
       del self.names[0]  # Remove Reference
       self.log('INFO', 'Symbols: {}'.format(self.names))
       # Exit
       self.exit = dt.date(2009, 6, 11)

   def log(self, level, message):
       print('{} - {} - {}'.format(level, self.date(0), message))

   def notify_order(self, order):
       name = order.data._name
       if order.status in [order.Submitted, order.Accepted]:
           # Buy/Sell order submitted/accepted to/by broker - Nothing to do
           return
       # Check if an order has been completed
       if order.status in [order.Completed]:
           side = 'Buy' if order.isbuy() else 'Sell'
           message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
           self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
       elif order.status in [order.Canceled, order.Margin, order.Rejected]:
           self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))

       self.order = None

   def notify_trade(self, trade):
       name = trade.data._name
       if not trade.isclosed:
           return
       self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))

   def prenext(self):
       self.next()

   def next(self):
       if self.order:
           return
       if self.date(0) == dt.date(2009, 2, 3):
           self.log('DEBUG', 'This is where the problem is!')
           self.log('DEBUG', '{} {}'.format(self.datas[1].close[0], self.names[0]))
       # Initialize Data
       dta = self.datas[1]
       # Debugging
       # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
       name, pos = self.names[0], self.getpositionbyname(self.names[0])
       # Entry
       if not pos and self.date(0) == dt.date(2009, 2, 3):
           self.log('INFO', '{} - Entry on first bar!'.format(name))
           if self.date(0).day % 2 == 0:  # Sell on even days
               self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=-1)
           else:
               self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=1)
       # Final Exit
       if pos and self.date(0) == self.exit:
           self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
           self.order = self.close(data=dta, exectype=bt.Order.Close)


def test_trade_zero(ticker, multi):
   cash = 100000.0
   # Setup Cerebro
   cerebro = bt.Cerebro()
   cerebro.broker.setcash(cash)
   # Data
   if multi:
       data = bt.feeds.PandasData(dataname=create_reference_data('2008-06-18', '2009-06-15'))
       cerebro.adddata(data, name='Reference')
   df = get_market_df(ticker)
   data = bt.feeds.PandasData(dataname=df)
   cerebro.adddata(data, name=ticker)
   comms = bt.CommissionInfo(commission=2.75, margin=950, mult=400)
   cerebro.broker.addcommissioninfo(comms, name=ticker)
   # Strategy
   cerebro.addstrategy(TradeZero2 if multi else TradeZero)
   # Backtest
   cerebro.run()


if __name__ == '__main__':
   test_trade_zero('LHS1M09', False)
   test_trade_zero('LHS1M09', True)
   test_trade_zero('LHS2M09', True)

Now, there are 3 cases in my test code:
1. Loading only one time serie of LHS1M09: we can see that the trade at 0 works
2. Loading 2 time series, the Reference and LHS1M09: the trade at 0 doesn't work, the order is submitted on LHS1M09 but executed on the Reference somehow?
3. Loading 2 time series, the Reference and LHS2M09 (which is the same as LHS1M09 except that I change the close price of the day where the trade will take place to a non-zero value): the trade now works

Now below is the output of the 3 cases:

Case 1:

Get dataframe for: LHS1M09
INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
INFO - 2009-02-04 - LHS1M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75
INFO - 2009-06-11 - LHS1M09 - Exit on last bar @ -3.425!
INFO - 2009-06-12 - LHS1M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
INFO - 2009-06-12 - LHS1M09 - Trade PnL - Gross: -1220.00, Net: -1225.50

Case 2:

Get dataframe for: LHS1M09
INFO - 2009-06-15 - Symbols: ['LHS1M09']
DEBUG - 2009-02-03 - This is where the problem is!
DEBUG - 2009-02-03 - 0.0 LHS1M09  # Close price is 0.00
INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
INFO - 2009-02-04 - Reference - Execution (Buy) - Price: 1.00, Cost: 1.00, Comms: 0.00  # This is where the problem is!

Case 3:

Get dataframe for: LHS2M09
INFO - 2009-06-15 - Symbols: ['LHS2M09']
DEBUG - 2009-02-03 - This is where the problem is!
DEBUG - 2009-02-03 - 0.05 LHS2M09  # Close price is 0.05
INFO - 2009-02-03 - LHS2M09 - Entry on first bar!
INFO - 2009-02-03 - LHS2M09 - Buy Order (Front) @ 0.05
INFO - 2009-02-04 - LHS2M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75  # Now works fine
INFO - 2009-06-11 - LHS2M09 - Exit on last bar @ -3.425!
INFO - 2009-06-12 - LHS2M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
INFO - 2009-06-12 - LHS2M09 - Trade PnL - Gross: -1220.00, Net: -1225.50

I am not sure if I am missing something obvious or if there is a bug here. Any help would be greatly appreciated.

Thanks!

[Laurent Michelizza]

So I have done a bit of debugging and found where the data gets changed to Reference from LHS1M09.

It happens in the following test:

data = data or self.datas[0]

Hence why it is not a problem when dealing with only one serie but when dealing with multi-series, we revert back to self.datas[0] which is "Reference" in my case.

Somehow, even though data is a valid time serie, the test seems to return False when Close is equal to 0.0.

See below a test:

self.log('DEBUG', 'Date: {} - Close {} - Test: {}!'.format(self.date(0), dta.close[0], bool(dta)))

With the following results:

DEBUG - 2009-01-29 - Date: 2009-01-29 - Close 0.075 - Test: True!
DEBUG - 2009-01-30 - Date: 2009-01-30 - Close 0.05 - Test: True!
DEBUG - 2009-02-02 - Date: 2009-02-02 - Close -0.175 - Test: True!
DEBUG - 2009-02-03 - Date: 2009-02-03 - Close 0.0 - Test: False!
DEBUG - 2009-02-04 - Date: 2009-02-04 - Close -0.05 - Test: True!
DEBUG - 2009-02-05 - Date: 2009-02-05 - Close -0.25 - Test: True!
DEBUG - 2009-02-06 - Date: 2009-02-06 - Close -0.6 - Test: True!

I guess this is due to 0.0 evaluated as False... I'll try to change the code to:

data = data if data is not None else self.datas[0]

@backtrader: Any ideas on this?

Thanks!

[backtrader]

@laurent-michelizza said in Problem with Trade @ 0 and multi time series:

data = data or self.datas[0]

This is because when you run this code in next, you are in the Stage 2 mode and a boolean evaluation of a lines object (data feeds, indicators, observers, ...) uses the current [0] point. This is done by overloading the operators and intended to support code patterns like

class MyStrategy(bt.Strategy):
    def __init__(self):
        self.signal = bt.ind.SMA(self.data) > self.data

    def next(self):
        if self.signal:
            self.buy()

The different overloading during Stage 1 and Stage 2 allows for example the lazy evaluation of the logical comparison seen above during __init__

This is documented here: Docs - Platform Concepts (look for Stage 1 and Stage 2)

Using the is operator is the way to go.

[Laurent Michelizza]

The problem though is that the line of code that I am referencing (see below) is in the backtrader codebase (4 instances in strategy.py) and not in my code as I always reference data explicitly.

data = data or self.datas[0]

As a hack, I have amended the line in strategy.py to the below:

data = data if data is not None else self.datas[0]

This is not ideal though. Is there a way around it or would you consider amending this piece of code?

PS: Thanks for the link, I will look at it in more details.

[backtrader]

Indeed. At some point in time there was an attempt to remove those checks (one operator overloading was implemented) Obviously, some of them survived the attempt.

[backtrader]

Situation which can (probably) only triggered when working with, not so usual, things like spreads (or else the stock/future you have in your portfolio is really worth nothing)