Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Problem with Trade @ 0 and multi time series

    General Code/Help
    2
    6
    1184
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      Laurent Michelizza last edited by Laurent Michelizza

      Hi,

      As I mainly trade spreads on Commodities, I had to find a way to load several time series per commodity. So after some work, I ended up with a setup where for each future I have a chain that is formed of several time series (all of them have different start/end times, one time serie for each spread). I also load first a dummy reference as the first serie to help with the synchronization of all the time series (the dummy reference is a time serie from the minimum start to the maximum end of all the time series).

      This seems to work pretty well, except that a couple of days ago, while doing some testing I ran into a weird issue. Spreads are financial instruments that can be negative, positive or equal to zero, so I need to be able to trade at any price. Which seems to be working fine when loading only one time serie, but a trade a zero on the second serie doesn't seem to work as expected (or I am missing something).

      See below my code (can be ran as is with the data - see link in the code):

      import datetime as dt
      import os
      import pandas as pd
      
      import backtrader as bt
      
      # OneDrive Link: https://1drv.ms/f/s!Ai-XKOnv2eoFg9Fyyo_asLi9_EapqQ
      # Change below to where the data is:
      root = '.\\'
      
      
      def business_date_range(start, end):
         """From a start and end date, returns a list of business days dates
      
         :param start: str/datetime - Start Date
         :param end: str/datetime - End Date
         :return: list - Business Dates
         """
         # Convert to datetime
         if not isinstance(start, dt.datetime):
             start = dt.datetime.strptime(start, '%Y-%m-%d')
         if not isinstance(end, dt.datetime):
             end = dt.datetime.strptime(end, '%Y-%m-%d')
      
         return [d for d in [start + dt.timedelta(days=i) for i in range((end - start).days + 1)] if d.weekday() < 5]
      
      
      def create_reference_data(start, end):
         """Create a dummy file to contains dates for the whole backtest - this is used as a reference for the backtest
         when not all time series are aligned properly.
      
         :param start: str/datetime - Start Date
         :param end: str/datetime - End Date
         """
         path = get_file_path('Reference')
         # Go through all the dates
         dates = business_date_range(start, end)
         ld = []
         fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
         for d in dates:
             ld.append({'Date': d, 'Open': 1, 'High': 2, 'Low': 3, 'Close': 4, 'Volume': 5, 'OI': 6})
         # Create DataFrame
         df = pd.DataFrame(ld)
         df = df.set_index('Date', drop=True)
         df.index.names = [None]
         df = df[fields]
         # Save DataFrame
         df.to_csv(path, header=False)
      
         return df
      
      
      def get_market_df(ticker, start_date=None, end_date=None):
         """From a ticker get the data from the data repository and store into a dataframe
      
         :param ticker: Customized ticker
         :param start_date: Optional parameter start date
         :param end_date: Optional parameter start date
         :return: Dataframe containing the data or None if there's nothing in the database
         """
         print('Get dataframe for: {}'.format(ticker))
         file = get_file_path(ticker)
         if os.path.exists(file):
             fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
             # Read Data
             df = pd.read_csv(file, sep=',', parse_dates=True, header=None, names=fields)
             # Remove week end data
             df = df[df.index.dayofweek < 5]
             # Check if data needs to be truncated
             if start_date is not None:
                 df = df[df.index >= start_date]
             if end_date is not None:
                 df = df[df.index <= end_date]
             # Check if missing data
             if df.isnull().values.any():
                 print('Missing data in: {} - Count: {}, please check!'.format(ticker, df.isnull().sum().sum()))
             return df
         else:
             return None
      
      
      def get_file_path(ticker):
         """Get data path for the ticker (Also returns the path for the dummy reference).
      
         :param ticker: str - Customized ticker
         :return: str - Path
         """
         if ticker == 'Reference':
             path = os.path.join(root, 'Reference.txt')
         else:
             path = os.path.join(root, '{}.txt'.format(ticker))
      
         return path
      
      
      class TradeZero(bt.Strategy):
         """Test TradeZero to identify if we can trade at 0!"""
      
         def __init__(self):
             # Internal Strategy Variables
             self.date = self.datas[0].datetime.date  # Reference
             self.order = None  # To keep track of pending orders
             self.names = self.getdatanames()
             # Exit
             self.exit = dt.date(2009, 6, 11)
      
         def log(self, level, message):
             print('{} - {} - {}'.format(level, self.date(0), message))
      
         def notify_order(self, order):
             name = order.data._name
             if order.status in [order.Submitted, order.Accepted]:
                 # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                 return
             # Check if an order has been completed
             if order.status in [order.Completed]:
                 side = 'Buy' if order.isbuy() else 'Sell'
                 message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
                 self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
             elif order.status in [order.Canceled, order.Margin, order.Rejected]:
                 self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))
      
             self.order = None
      
         def notify_trade(self, trade):
             name = trade.data._name
             if not trade.isclosed:
                 return
             self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))
      
         def next(self):
             if self.order:
                 return
             # Initialize Data
             dta = self.datas[0]
             # Debugging
             # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
             name, pos = self.names[0], self.getpositionbyname(self.names[0])
             # Entry
             if not pos and self.date(0) == dt.date(2009, 2, 3):
                 self.log('INFO', '{} - Entry on first bar!'.format(name))
                 if self.date(0).day % 2 == 0:  # Sell on even days
                     self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
                     self.order = self.order_target_size(data=dta, target=-1)
                 else:
                     self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
                     self.order = self.order_target_size(data=dta, target=1)
      
             # Final Exit
             if pos and self.date(0) == self.exit:
                 self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
                 self.order = self.close(data=dta, exectype=bt.Order.Close)
      
      
      class TradeZero2(bt.Strategy):
         """Test TradeZero to identify if we can trade at 0!"""
      
         def __init__(self):
             # Internal Strategy Variables
             self.date = self.datas[0].datetime.date  # Reference
             self.order = None  # To keep track of pending orders
             self.names = self.getdatanames()
             del self.names[0]  # Remove Reference
             self.log('INFO', 'Symbols: {}'.format(self.names))
             # Exit
             self.exit = dt.date(2009, 6, 11)
      
         def log(self, level, message):
             print('{} - {} - {}'.format(level, self.date(0), message))
      
         def notify_order(self, order):
             name = order.data._name
             if order.status in [order.Submitted, order.Accepted]:
                 # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                 return
             # Check if an order has been completed
             if order.status in [order.Completed]:
                 side = 'Buy' if order.isbuy() else 'Sell'
                 message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
                 self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
             elif order.status in [order.Canceled, order.Margin, order.Rejected]:
                 self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))
      
             self.order = None
      
         def notify_trade(self, trade):
             name = trade.data._name
             if not trade.isclosed:
                 return
             self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))
      
         def prenext(self):
             self.next()
      
         def next(self):
             if self.order:
                 return
             if self.date(0) == dt.date(2009, 2, 3):
                 self.log('DEBUG', 'This is where the problem is!')
                 self.log('DEBUG', '{} {}'.format(self.datas[1].close[0], self.names[0]))
             # Initialize Data
             dta = self.datas[1]
             # Debugging
             # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
             name, pos = self.names[0], self.getpositionbyname(self.names[0])
             # Entry
             if not pos and self.date(0) == dt.date(2009, 2, 3):
                 self.log('INFO', '{} - Entry on first bar!'.format(name))
                 if self.date(0).day % 2 == 0:  # Sell on even days
                     self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
                     self.order = self.order_target_size(data=dta, target=-1)
                 else:
                     self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
                     self.order = self.order_target_size(data=dta, target=1)
             # Final Exit
             if pos and self.date(0) == self.exit:
                 self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
                 self.order = self.close(data=dta, exectype=bt.Order.Close)
      
      
      def test_trade_zero(ticker, multi):
         cash = 100000.0
         # Setup Cerebro
         cerebro = bt.Cerebro()
         cerebro.broker.setcash(cash)
         # Data
         if multi:
             data = bt.feeds.PandasData(dataname=create_reference_data('2008-06-18', '2009-06-15'))
             cerebro.adddata(data, name='Reference')
         df = get_market_df(ticker)
         data = bt.feeds.PandasData(dataname=df)
         cerebro.adddata(data, name=ticker)
         comms = bt.CommissionInfo(commission=2.75, margin=950, mult=400)
         cerebro.broker.addcommissioninfo(comms, name=ticker)
         # Strategy
         cerebro.addstrategy(TradeZero2 if multi else TradeZero)
         # Backtest
         cerebro.run()
      
      
      if __name__ == '__main__':
         test_trade_zero('LHS1M09', False)
         test_trade_zero('LHS1M09', True)
         test_trade_zero('LHS2M09', True)
      

      Now, there are 3 cases in my test code:
      1. Loading only one time serie of LHS1M09: we can see that the trade at 0 works
      2. Loading 2 time series, the Reference and LHS1M09: the trade at 0 doesn't work, the order is submitted on LHS1M09 but executed on the Reference somehow?
      3. Loading 2 time series, the Reference and LHS2M09 (which is the same as LHS1M09 except that I change the close price of the day where the trade will take place to a non-zero value): the trade now works

      Now below is the output of the 3 cases:

      Case 1:

      Get dataframe for: LHS1M09
      INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
      INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
      INFO - 2009-02-04 - LHS1M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75
      INFO - 2009-06-11 - LHS1M09 - Exit on last bar @ -3.425!
      INFO - 2009-06-12 - LHS1M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
      INFO - 2009-06-12 - LHS1M09 - Trade PnL - Gross: -1220.00, Net: -1225.50
      

      Case 2:

      Get dataframe for: LHS1M09
      INFO - 2009-06-15 - Symbols: ['LHS1M09']
      DEBUG - 2009-02-03 - This is where the problem is!
      DEBUG - 2009-02-03 - 0.0 LHS1M09  # Close price is 0.00
      INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
      INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
      INFO - 2009-02-04 - Reference - Execution (Buy) - Price: 1.00, Cost: 1.00, Comms: 0.00  # This is where the problem is!
      

      Case 3:

      Get dataframe for: LHS2M09
      INFO - 2009-06-15 - Symbols: ['LHS2M09']
      DEBUG - 2009-02-03 - This is where the problem is!
      DEBUG - 2009-02-03 - 0.05 LHS2M09  # Close price is 0.05
      INFO - 2009-02-03 - LHS2M09 - Entry on first bar!
      INFO - 2009-02-03 - LHS2M09 - Buy Order (Front) @ 0.05
      INFO - 2009-02-04 - LHS2M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75  # Now works fine
      INFO - 2009-06-11 - LHS2M09 - Exit on last bar @ -3.425!
      INFO - 2009-06-12 - LHS2M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
      INFO - 2009-06-12 - LHS2M09 - Trade PnL - Gross: -1220.00, Net: -1225.50
      

      I am not sure if I am missing something obvious or if there is a bug here. Any help would be greatly appreciated.

      Thanks!

      1 Reply Last reply Reply Quote 0
      • L
        Laurent Michelizza last edited by Laurent Michelizza

        So I have done a bit of debugging and found where the data gets changed to Reference from LHS1M09.

        It happens in the following test:

        data = data or self.datas[0]
        

        Hence why it is not a problem when dealing with only one serie but when dealing with multi-series, we revert back to self.datas[0] which is "Reference" in my case.

        Somehow, even though data is a valid time serie, the test seems to return False when Close is equal to 0.0.

        See below a test:

        self.log('DEBUG', 'Date: {} - Close {} - Test: {}!'.format(self.date(0), dta.close[0], bool(dta)))
        

        With the following results:

        DEBUG - 2009-01-29 - Date: 2009-01-29 - Close 0.075 - Test: True!
        DEBUG - 2009-01-30 - Date: 2009-01-30 - Close 0.05 - Test: True!
        DEBUG - 2009-02-02 - Date: 2009-02-02 - Close -0.175 - Test: True!
        DEBUG - 2009-02-03 - Date: 2009-02-03 - Close 0.0 - Test: False!
        DEBUG - 2009-02-04 - Date: 2009-02-04 - Close -0.05 - Test: True!
        DEBUG - 2009-02-05 - Date: 2009-02-05 - Close -0.25 - Test: True!
        DEBUG - 2009-02-06 - Date: 2009-02-06 - Close -0.6 - Test: True!
        

        I guess this is due to 0.0 evaluated as False... I'll try to change the code to:

        data = data if data is not None else self.datas[0]
        

        @backtrader: Any ideas on this?

        Thanks!

        B 1 Reply Last reply Reply Quote 0
        • B
          backtrader administrators @Laurent Michelizza last edited by

          @laurent-michelizza said in Problem with Trade @ 0 and multi time series:

          data = data or self.datas[0]
          

          This is because when you run this code in next, you are in the Stage 2 mode and a boolean evaluation of a lines object (data feeds, indicators, observers, ...) uses the current [0] point. This is done by overloading the operators and intended to support code patterns like

          class MyStrategy(bt.Strategy):
              def __init__(self):
                  self.signal = bt.ind.SMA(self.data) > self.data
          
              def next(self):
                  if self.signal:
                      self.buy()
          

          The different overloading during Stage 1 and Stage 2 allows for example the lazy evaluation of the logical comparison seen above during __init__

          This is documented here: Docs - Platform Concepts (look for Stage 1 and Stage 2)

          Using the is operator is the way to go.

          1 Reply Last reply Reply Quote 0
          • L
            Laurent Michelizza last edited by

            The problem though is that the line of code that I am referencing (see below) is in the backtrader codebase (4 instances in strategy.py) and not in my code as I always reference data explicitly.

            data = data or self.datas[0]
            

            As a hack, I have amended the line in strategy.py to the below:

            data = data if data is not None else self.datas[0]
            

            This is not ideal though. Is there a way around it or would you consider amending this piece of code?

            PS: Thanks for the link, I will look at it in more details.

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              Indeed. At some point in time there was an attempt to remove those checks (one operator overloading was implemented) Obviously, some of them survived the attempt.

              1 Reply Last reply Reply Quote 1
              • B
                backtrader administrators last edited by

                Situation which can (probably) only triggered when working with, not so usual, things like spreads (or else the stock/future you have in your portfolio is really worth nothing)

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post
                Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors