For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Problem with Trade @ 0 and multi time series



  • Hi,

    As I mainly trade spreads on Commodities, I had to find a way to load several time series per commodity. So after some work, I ended up with a setup where for each future I have a chain that is formed of several time series (all of them have different start/end times, one time serie for each spread). I also load first a dummy reference as the first serie to help with the synchronization of all the time series (the dummy reference is a time serie from the minimum start to the maximum end of all the time series).

    This seems to work pretty well, except that a couple of days ago, while doing some testing I ran into a weird issue. Spreads are financial instruments that can be negative, positive or equal to zero, so I need to be able to trade at any price. Which seems to be working fine when loading only one time serie, but a trade a zero on the second serie doesn't seem to work as expected (or I am missing something).

    See below my code (can be ran as is with the data - see link in the code):

    import datetime as dt
    import os
    import pandas as pd
    
    import backtrader as bt
    
    # OneDrive Link: https://1drv.ms/f/s!Ai-XKOnv2eoFg9Fyyo_asLi9_EapqQ
    # Change below to where the data is:
    root = '.\\'
    
    
    def business_date_range(start, end):
       """From a start and end date, returns a list of business days dates
    
       :param start: str/datetime - Start Date
       :param end: str/datetime - End Date
       :return: list - Business Dates
       """
       # Convert to datetime
       if not isinstance(start, dt.datetime):
           start = dt.datetime.strptime(start, '%Y-%m-%d')
       if not isinstance(end, dt.datetime):
           end = dt.datetime.strptime(end, '%Y-%m-%d')
    
       return [d for d in [start + dt.timedelta(days=i) for i in range((end - start).days + 1)] if d.weekday() < 5]
    
    
    def create_reference_data(start, end):
       """Create a dummy file to contains dates for the whole backtest - this is used as a reference for the backtest
       when not all time series are aligned properly.
    
       :param start: str/datetime - Start Date
       :param end: str/datetime - End Date
       """
       path = get_file_path('Reference')
       # Go through all the dates
       dates = business_date_range(start, end)
       ld = []
       fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
       for d in dates:
           ld.append({'Date': d, 'Open': 1, 'High': 2, 'Low': 3, 'Close': 4, 'Volume': 5, 'OI': 6})
       # Create DataFrame
       df = pd.DataFrame(ld)
       df = df.set_index('Date', drop=True)
       df.index.names = [None]
       df = df[fields]
       # Save DataFrame
       df.to_csv(path, header=False)
    
       return df
    
    
    def get_market_df(ticker, start_date=None, end_date=None):
       """From a ticker get the data from the data repository and store into a dataframe
    
       :param ticker: Customized ticker
       :param start_date: Optional parameter start date
       :param end_date: Optional parameter start date
       :return: Dataframe containing the data or None if there's nothing in the database
       """
       print('Get dataframe for: {}'.format(ticker))
       file = get_file_path(ticker)
       if os.path.exists(file):
           fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
           # Read Data
           df = pd.read_csv(file, sep=',', parse_dates=True, header=None, names=fields)
           # Remove week end data
           df = df[df.index.dayofweek < 5]
           # Check if data needs to be truncated
           if start_date is not None:
               df = df[df.index >= start_date]
           if end_date is not None:
               df = df[df.index <= end_date]
           # Check if missing data
           if df.isnull().values.any():
               print('Missing data in: {} - Count: {}, please check!'.format(ticker, df.isnull().sum().sum()))
           return df
       else:
           return None
    
    
    def get_file_path(ticker):
       """Get data path for the ticker (Also returns the path for the dummy reference).
    
       :param ticker: str - Customized ticker
       :return: str - Path
       """
       if ticker == 'Reference':
           path = os.path.join(root, 'Reference.txt')
       else:
           path = os.path.join(root, '{}.txt'.format(ticker))
    
       return path
    
    
    class TradeZero(bt.Strategy):
       """Test TradeZero to identify if we can trade at 0!"""
    
       def __init__(self):
           # Internal Strategy Variables
           self.date = self.datas[0].datetime.date  # Reference
           self.order = None  # To keep track of pending orders
           self.names = self.getdatanames()
           # Exit
           self.exit = dt.date(2009, 6, 11)
    
       def log(self, level, message):
           print('{} - {} - {}'.format(level, self.date(0), message))
    
       def notify_order(self, order):
           name = order.data._name
           if order.status in [order.Submitted, order.Accepted]:
               # Buy/Sell order submitted/accepted to/by broker - Nothing to do
               return
           # Check if an order has been completed
           if order.status in [order.Completed]:
               side = 'Buy' if order.isbuy() else 'Sell'
               message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
               self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
           elif order.status in [order.Canceled, order.Margin, order.Rejected]:
               self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))
    
           self.order = None
    
       def notify_trade(self, trade):
           name = trade.data._name
           if not trade.isclosed:
               return
           self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))
    
       def next(self):
           if self.order:
               return
           # Initialize Data
           dta = self.datas[0]
           # Debugging
           # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
           name, pos = self.names[0], self.getpositionbyname(self.names[0])
           # Entry
           if not pos and self.date(0) == dt.date(2009, 2, 3):
               self.log('INFO', '{} - Entry on first bar!'.format(name))
               if self.date(0).day % 2 == 0:  # Sell on even days
                   self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
                   self.order = self.order_target_size(data=dta, target=-1)
               else:
                   self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
                   self.order = self.order_target_size(data=dta, target=1)
    
           # Final Exit
           if pos and self.date(0) == self.exit:
               self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
               self.order = self.close(data=dta, exectype=bt.Order.Close)
    
    
    class TradeZero2(bt.Strategy):
       """Test TradeZero to identify if we can trade at 0!"""
    
       def __init__(self):
           # Internal Strategy Variables
           self.date = self.datas[0].datetime.date  # Reference
           self.order = None  # To keep track of pending orders
           self.names = self.getdatanames()
           del self.names[0]  # Remove Reference
           self.log('INFO', 'Symbols: {}'.format(self.names))
           # Exit
           self.exit = dt.date(2009, 6, 11)
    
       def log(self, level, message):
           print('{} - {} - {}'.format(level, self.date(0), message))
    
       def notify_order(self, order):
           name = order.data._name
           if order.status in [order.Submitted, order.Accepted]:
               # Buy/Sell order submitted/accepted to/by broker - Nothing to do
               return
           # Check if an order has been completed
           if order.status in [order.Completed]:
               side = 'Buy' if order.isbuy() else 'Sell'
               message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
               self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
           elif order.status in [order.Canceled, order.Margin, order.Rejected]:
               self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))
    
           self.order = None
    
       def notify_trade(self, trade):
           name = trade.data._name
           if not trade.isclosed:
               return
           self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))
    
       def prenext(self):
           self.next()
    
       def next(self):
           if self.order:
               return
           if self.date(0) == dt.date(2009, 2, 3):
               self.log('DEBUG', 'This is where the problem is!')
               self.log('DEBUG', '{} {}'.format(self.datas[1].close[0], self.names[0]))
           # Initialize Data
           dta = self.datas[1]
           # Debugging
           # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
           name, pos = self.names[0], self.getpositionbyname(self.names[0])
           # Entry
           if not pos and self.date(0) == dt.date(2009, 2, 3):
               self.log('INFO', '{} - Entry on first bar!'.format(name))
               if self.date(0).day % 2 == 0:  # Sell on even days
                   self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
                   self.order = self.order_target_size(data=dta, target=-1)
               else:
                   self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
                   self.order = self.order_target_size(data=dta, target=1)
           # Final Exit
           if pos and self.date(0) == self.exit:
               self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
               self.order = self.close(data=dta, exectype=bt.Order.Close)
    
    
    def test_trade_zero(ticker, multi):
       cash = 100000.0
       # Setup Cerebro
       cerebro = bt.Cerebro()
       cerebro.broker.setcash(cash)
       # Data
       if multi:
           data = bt.feeds.PandasData(dataname=create_reference_data('2008-06-18', '2009-06-15'))
           cerebro.adddata(data, name='Reference')
       df = get_market_df(ticker)
       data = bt.feeds.PandasData(dataname=df)
       cerebro.adddata(data, name=ticker)
       comms = bt.CommissionInfo(commission=2.75, margin=950, mult=400)
       cerebro.broker.addcommissioninfo(comms, name=ticker)
       # Strategy
       cerebro.addstrategy(TradeZero2 if multi else TradeZero)
       # Backtest
       cerebro.run()
    
    
    if __name__ == '__main__':
       test_trade_zero('LHS1M09', False)
       test_trade_zero('LHS1M09', True)
       test_trade_zero('LHS2M09', True)
    

    Now, there are 3 cases in my test code:
    1. Loading only one time serie of LHS1M09: we can see that the trade at 0 works
    2. Loading 2 time series, the Reference and LHS1M09: the trade at 0 doesn't work, the order is submitted on LHS1M09 but executed on the Reference somehow?
    3. Loading 2 time series, the Reference and LHS2M09 (which is the same as LHS1M09 except that I change the close price of the day where the trade will take place to a non-zero value): the trade now works

    Now below is the output of the 3 cases:

    Case 1:

    Get dataframe for: LHS1M09
    INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
    INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
    INFO - 2009-02-04 - LHS1M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75
    INFO - 2009-06-11 - LHS1M09 - Exit on last bar @ -3.425!
    INFO - 2009-06-12 - LHS1M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
    INFO - 2009-06-12 - LHS1M09 - Trade PnL - Gross: -1220.00, Net: -1225.50
    

    Case 2:

    Get dataframe for: LHS1M09
    INFO - 2009-06-15 - Symbols: ['LHS1M09']
    DEBUG - 2009-02-03 - This is where the problem is!
    DEBUG - 2009-02-03 - 0.0 LHS1M09  # Close price is 0.00
    INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
    INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
    INFO - 2009-02-04 - Reference - Execution (Buy) - Price: 1.00, Cost: 1.00, Comms: 0.00  # This is where the problem is!
    

    Case 3:

    Get dataframe for: LHS2M09
    INFO - 2009-06-15 - Symbols: ['LHS2M09']
    DEBUG - 2009-02-03 - This is where the problem is!
    DEBUG - 2009-02-03 - 0.05 LHS2M09  # Close price is 0.05
    INFO - 2009-02-03 - LHS2M09 - Entry on first bar!
    INFO - 2009-02-03 - LHS2M09 - Buy Order (Front) @ 0.05
    INFO - 2009-02-04 - LHS2M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75  # Now works fine
    INFO - 2009-06-11 - LHS2M09 - Exit on last bar @ -3.425!
    INFO - 2009-06-12 - LHS2M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
    INFO - 2009-06-12 - LHS2M09 - Trade PnL - Gross: -1220.00, Net: -1225.50
    

    I am not sure if I am missing something obvious or if there is a bug here. Any help would be greatly appreciated.

    Thanks!



  • So I have done a bit of debugging and found where the data gets changed to Reference from LHS1M09.

    It happens in the following test:

    data = data or self.datas[0]
    

    Hence why it is not a problem when dealing with only one serie but when dealing with multi-series, we revert back to self.datas[0] which is "Reference" in my case.

    Somehow, even though data is a valid time serie, the test seems to return False when Close is equal to 0.0.

    See below a test:

    self.log('DEBUG', 'Date: {} - Close {} - Test: {}!'.format(self.date(0), dta.close[0], bool(dta)))
    

    With the following results:

    DEBUG - 2009-01-29 - Date: 2009-01-29 - Close 0.075 - Test: True!
    DEBUG - 2009-01-30 - Date: 2009-01-30 - Close 0.05 - Test: True!
    DEBUG - 2009-02-02 - Date: 2009-02-02 - Close -0.175 - Test: True!
    DEBUG - 2009-02-03 - Date: 2009-02-03 - Close 0.0 - Test: False!
    DEBUG - 2009-02-04 - Date: 2009-02-04 - Close -0.05 - Test: True!
    DEBUG - 2009-02-05 - Date: 2009-02-05 - Close -0.25 - Test: True!
    DEBUG - 2009-02-06 - Date: 2009-02-06 - Close -0.6 - Test: True!
    

    I guess this is due to 0.0 evaluated as False... I'll try to change the code to:

    data = data if data is not None else self.datas[0]
    

    @backtrader: Any ideas on this?

    Thanks!


  • administrators

    @laurent-michelizza said in Problem with Trade @ 0 and multi time series:

    data = data or self.datas[0]
    

    This is because when you run this code in next, you are in the Stage 2 mode and a boolean evaluation of a lines object (data feeds, indicators, observers, ...) uses the current [0] point. This is done by overloading the operators and intended to support code patterns like

    class MyStrategy(bt.Strategy):
        def __init__(self):
            self.signal = bt.ind.SMA(self.data) > self.data
    
        def next(self):
            if self.signal:
                self.buy()
    

    The different overloading during Stage 1 and Stage 2 allows for example the lazy evaluation of the logical comparison seen above during __init__

    This is documented here: Docs - Platform Concepts (look for Stage 1 and Stage 2)

    Using the is operator is the way to go.



  • The problem though is that the line of code that I am referencing (see below) is in the backtrader codebase (4 instances in strategy.py) and not in my code as I always reference data explicitly.

    data = data or self.datas[0]
    

    As a hack, I have amended the line in strategy.py to the below:

    data = data if data is not None else self.datas[0]
    

    This is not ideal though. Is there a way around it or would you consider amending this piece of code?

    PS: Thanks for the link, I will look at it in more details.


  • administrators

    Indeed. At some point in time there was an attempt to remove those checks (one operator overloading was implemented) Obviously, some of them survived the attempt.


  • administrators

    Situation which can (probably) only triggered when working with, not so usual, things like spreads (or else the stock/future you have in your portfolio is really worth nothing)