@eduedix

Hi,

This is what I do to use pyfolio:

        cerebro = bt.Cerebro()
        cerebro.broker.setcash(cash)
        cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')

        for s in symbols:
            df = pd.read_csv(os.path.join(path_dir, '{}.csv'.format(s)), parse_dates=True, index_col=0)
            data = bt.feeds.PandasData(dataname=df)
            cerebro.adddata(data)

        cerebro.addstrategy(strategy, **params)

        results = cerebro.run()
        strat = results[0]
        pyfoliozer = strat.analyzers.getbyname('pyfolio')
        returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items()

Hope this helps.

Hi,

You need a data provider. Look at Esignal or IQFeed.

The problem though is that the line of code that I am referencing (see below) is in the backtrader codebase (4 instances in strategy.py) and not in my code as I always reference data explicitly.

data = data or self.datas[0]

As a hack, I have amended the line in strategy.py to the below:

data = data if data is not None else self.datas[0]

This is not ideal though. Is there a way around it or would you consider amending this piece of code?

PS: Thanks for the link, I will look at it in more details.

So I have done a bit of debugging and found where the data gets changed to Reference from LHS1M09.

It happens in the following test:

data = data or self.datas[0]

Hence why it is not a problem when dealing with only one serie but when dealing with multi-series, we revert back to self.datas[0] which is "Reference" in my case.

Somehow, even though data is a valid time serie, the test seems to return False when Close is equal to 0.0.

See below a test:

self.log('DEBUG', 'Date: {} - Close {} - Test: {}!'.format(self.date(0), dta.close[0], bool(dta)))

With the following results:

DEBUG - 2009-01-29 - Date: 2009-01-29 - Close 0.075 - Test: True!
DEBUG - 2009-01-30 - Date: 2009-01-30 - Close 0.05 - Test: True!
DEBUG - 2009-02-02 - Date: 2009-02-02 - Close -0.175 - Test: True!
DEBUG - 2009-02-03 - Date: 2009-02-03 - Close 0.0 - Test: False!
DEBUG - 2009-02-04 - Date: 2009-02-04 - Close -0.05 - Test: True!
DEBUG - 2009-02-05 - Date: 2009-02-05 - Close -0.25 - Test: True!
DEBUG - 2009-02-06 - Date: 2009-02-06 - Close -0.6 - Test: True!

I guess this is due to 0.0 evaluated as False... I'll try to change the code to:

data = data if data is not None else self.datas[0]

@backtrader: Any ideas on this?

Thanks!

I don't think you should rename the variable params as it is part of the default strategy that you are inheriting. You can also access it simply with .p like below:

self.p.printlog

Hi,

As I mainly trade spreads on Commodities, I had to find a way to load several time series per commodity. So after some work, I ended up with a setup where for each future I have a chain that is formed of several time series (all of them have different start/end times, one time serie for each spread). I also load first a dummy reference as the first serie to help with the synchronization of all the time series (the dummy reference is a time serie from the minimum start to the maximum end of all the time series).

This seems to work pretty well, except that a couple of days ago, while doing some testing I ran into a weird issue. Spreads are financial instruments that can be negative, positive or equal to zero, so I need to be able to trade at any price. Which seems to be working fine when loading only one time serie, but a trade a zero on the second serie doesn't seem to work as expected (or I am missing something).

See below my code (can be ran as is with the data - see link in the code):

import datetime as dt
import os
import pandas as pd

import backtrader as bt

# OneDrive Link: https://1drv.ms/f/s!Ai-XKOnv2eoFg9Fyyo_asLi9_EapqQ
# Change below to where the data is:
root = '.\\'


def business_date_range(start, end):
   """From a start and end date, returns a list of business days dates

   :param start: str/datetime - Start Date
   :param end: str/datetime - End Date
   :return: list - Business Dates
   """
   # Convert to datetime
   if not isinstance(start, dt.datetime):
       start = dt.datetime.strptime(start, '%Y-%m-%d')
   if not isinstance(end, dt.datetime):
       end = dt.datetime.strptime(end, '%Y-%m-%d')

   return [d for d in [start + dt.timedelta(days=i) for i in range((end - start).days + 1)] if d.weekday() < 5]


def create_reference_data(start, end):
   """Create a dummy file to contains dates for the whole backtest - this is used as a reference for the backtest
   when not all time series are aligned properly.

   :param start: str/datetime - Start Date
   :param end: str/datetime - End Date
   """
   path = get_file_path('Reference')
   # Go through all the dates
   dates = business_date_range(start, end)
   ld = []
   fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
   for d in dates:
       ld.append({'Date': d, 'Open': 1, 'High': 2, 'Low': 3, 'Close': 4, 'Volume': 5, 'OI': 6})
   # Create DataFrame
   df = pd.DataFrame(ld)
   df = df.set_index('Date', drop=True)
   df.index.names = [None]
   df = df[fields]
   # Save DataFrame
   df.to_csv(path, header=False)

   return df


def get_market_df(ticker, start_date=None, end_date=None):
   """From a ticker get the data from the data repository and store into a dataframe

   :param ticker: Customized ticker
   :param start_date: Optional parameter start date
   :param end_date: Optional parameter start date
   :return: Dataframe containing the data or None if there's nothing in the database
   """
   print('Get dataframe for: {}'.format(ticker))
   file = get_file_path(ticker)
   if os.path.exists(file):
       fields = ['Open', 'High', 'Low', 'Close', 'Volume', 'OI']
       # Read Data
       df = pd.read_csv(file, sep=',', parse_dates=True, header=None, names=fields)
       # Remove week end data
       df = df[df.index.dayofweek < 5]
       # Check if data needs to be truncated
       if start_date is not None:
           df = df[df.index >= start_date]
       if end_date is not None:
           df = df[df.index <= end_date]
       # Check if missing data
       if df.isnull().values.any():
           print('Missing data in: {} - Count: {}, please check!'.format(ticker, df.isnull().sum().sum()))
       return df
   else:
       return None


def get_file_path(ticker):
   """Get data path for the ticker (Also returns the path for the dummy reference).

   :param ticker: str - Customized ticker
   :return: str - Path
   """
   if ticker == 'Reference':
       path = os.path.join(root, 'Reference.txt')
   else:
       path = os.path.join(root, '{}.txt'.format(ticker))

   return path


class TradeZero(bt.Strategy):
   """Test TradeZero to identify if we can trade at 0!"""

   def __init__(self):
       # Internal Strategy Variables
       self.date = self.datas[0].datetime.date  # Reference
       self.order = None  # To keep track of pending orders
       self.names = self.getdatanames()
       # Exit
       self.exit = dt.date(2009, 6, 11)

   def log(self, level, message):
       print('{} - {} - {}'.format(level, self.date(0), message))

   def notify_order(self, order):
       name = order.data._name
       if order.status in [order.Submitted, order.Accepted]:
           # Buy/Sell order submitted/accepted to/by broker - Nothing to do
           return
       # Check if an order has been completed
       if order.status in [order.Completed]:
           side = 'Buy' if order.isbuy() else 'Sell'
           message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
           self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
       elif order.status in [order.Canceled, order.Margin, order.Rejected]:
           self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))

       self.order = None

   def notify_trade(self, trade):
       name = trade.data._name
       if not trade.isclosed:
           return
       self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))

   def next(self):
       if self.order:
           return
       # Initialize Data
       dta = self.datas[0]
       # Debugging
       # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
       name, pos = self.names[0], self.getpositionbyname(self.names[0])
       # Entry
       if not pos and self.date(0) == dt.date(2009, 2, 3):
           self.log('INFO', '{} - Entry on first bar!'.format(name))
           if self.date(0).day % 2 == 0:  # Sell on even days
               self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=-1)
           else:
               self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=1)

       # Final Exit
       if pos and self.date(0) == self.exit:
           self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
           self.order = self.close(data=dta, exectype=bt.Order.Close)


class TradeZero2(bt.Strategy):
   """Test TradeZero to identify if we can trade at 0!"""

   def __init__(self):
       # Internal Strategy Variables
       self.date = self.datas[0].datetime.date  # Reference
       self.order = None  # To keep track of pending orders
       self.names = self.getdatanames()
       del self.names[0]  # Remove Reference
       self.log('INFO', 'Symbols: {}'.format(self.names))
       # Exit
       self.exit = dt.date(2009, 6, 11)

   def log(self, level, message):
       print('{} - {} - {}'.format(level, self.date(0), message))

   def notify_order(self, order):
       name = order.data._name
       if order.status in [order.Submitted, order.Accepted]:
           # Buy/Sell order submitted/accepted to/by broker - Nothing to do
           return
       # Check if an order has been completed
       if order.status in [order.Completed]:
           side = 'Buy' if order.isbuy() else 'Sell'
           message = '{} - Execution ({}) - Price: {:.2f}, Cost: {:.2f}, Comms: {:.2f}'
           self.log('INFO', message.format(name, side, order.executed.price, order.executed.value, order.executed.comm))
       elif order.status in [order.Canceled, order.Margin, order.Rejected]:
           self.log('WARNING', '{} - Order Cancelled/Margin/Rejected'.format(name))

       self.order = None

   def notify_trade(self, trade):
       name = trade.data._name
       if not trade.isclosed:
           return
       self.log('INFO', '{} - Trade PnL - Gross: {:.2f}, Net: {:.2f}'.format(name, trade.pnl, trade.pnlcomm))

   def prenext(self):
       self.next()

   def next(self):
       if self.order:
           return
       if self.date(0) == dt.date(2009, 2, 3):
           self.log('DEBUG', 'This is where the problem is!')
           self.log('DEBUG', '{} {}'.format(self.datas[1].close[0], self.names[0]))
       # Initialize Data
       dta = self.datas[1]
       # Debugging
       # self.log('DEBUG', 'Date: {} - Close: {}'.format(self.date(0), dta.close[0]))
       name, pos = self.names[0], self.getpositionbyname(self.names[0])
       # Entry
       if not pos and self.date(0) == dt.date(2009, 2, 3):
           self.log('INFO', '{} - Entry on first bar!'.format(name))
           if self.date(0).day % 2 == 0:  # Sell on even days
               self.log('INFO', '{} - Sell Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=-1)
           else:
               self.log('INFO', '{} - Buy Order (Front) @ {}'.format(name, dta.close[0]))
               self.order = self.order_target_size(data=dta, target=1)
       # Final Exit
       if pos and self.date(0) == self.exit:
           self.log('INFO', '{} - Exit on last bar @ {}!'.format(name, dta.close[0]))
           self.order = self.close(data=dta, exectype=bt.Order.Close)


def test_trade_zero(ticker, multi):
   cash = 100000.0
   # Setup Cerebro
   cerebro = bt.Cerebro()
   cerebro.broker.setcash(cash)
   # Data
   if multi:
       data = bt.feeds.PandasData(dataname=create_reference_data('2008-06-18', '2009-06-15'))
       cerebro.adddata(data, name='Reference')
   df = get_market_df(ticker)
   data = bt.feeds.PandasData(dataname=df)
   cerebro.adddata(data, name=ticker)
   comms = bt.CommissionInfo(commission=2.75, margin=950, mult=400)
   cerebro.broker.addcommissioninfo(comms, name=ticker)
   # Strategy
   cerebro.addstrategy(TradeZero2 if multi else TradeZero)
   # Backtest
   cerebro.run()


if __name__ == '__main__':
   test_trade_zero('LHS1M09', False)
   test_trade_zero('LHS1M09', True)
   test_trade_zero('LHS2M09', True)

Now, there are 3 cases in my test code:
1. Loading only one time serie of LHS1M09: we can see that the trade at 0 works
2. Loading 2 time series, the Reference and LHS1M09: the trade at 0 doesn't work, the order is submitted on LHS1M09 but executed on the Reference somehow?
3. Loading 2 time series, the Reference and LHS2M09 (which is the same as LHS1M09 except that I change the close price of the day where the trade will take place to a non-zero value): the trade now works

Now below is the output of the 3 cases:

Case 1:

Get dataframe for: LHS1M09
INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
INFO - 2009-02-04 - LHS1M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75
INFO - 2009-06-11 - LHS1M09 - Exit on last bar @ -3.425!
INFO - 2009-06-12 - LHS1M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
INFO - 2009-06-12 - LHS1M09 - Trade PnL - Gross: -1220.00, Net: -1225.50

Case 2:

Get dataframe for: LHS1M09
INFO - 2009-06-15 - Symbols: ['LHS1M09']
DEBUG - 2009-02-03 - This is where the problem is!
DEBUG - 2009-02-03 - 0.0 LHS1M09  # Close price is 0.00
INFO - 2009-02-03 - LHS1M09 - Entry on first bar!
INFO - 2009-02-03 - LHS1M09 - Buy Order (Front) @ 0.0
INFO - 2009-02-04 - Reference - Execution (Buy) - Price: 1.00, Cost: 1.00, Comms: 0.00  # This is where the problem is!

Case 3:

Get dataframe for: LHS2M09
INFO - 2009-06-15 - Symbols: ['LHS2M09']
DEBUG - 2009-02-03 - This is where the problem is!
DEBUG - 2009-02-03 - 0.05 LHS2M09  # Close price is 0.05
INFO - 2009-02-03 - LHS2M09 - Entry on first bar!
INFO - 2009-02-03 - LHS2M09 - Buy Order (Front) @ 0.05
INFO - 2009-02-04 - LHS2M09 - Execution (Buy) - Price: -0.10, Cost: 950.00, Comms: 2.75  # Now works fine
INFO - 2009-06-11 - LHS2M09 - Exit on last bar @ -3.425!
INFO - 2009-06-12 - LHS2M09 - Execution (Sell) - Price: -3.15, Cost: 950.00, Comms: 2.75
INFO - 2009-06-12 - LHS2M09 - Trade PnL - Gross: -1220.00, Net: -1225.50

I am not sure if I am missing something obvious or if there is a bug here. Any help would be greatly appreciated.

Thanks!

That works thanks!

Hi,

I would like to access the strategy parameters for a project I am working on.

However when trying to do so, I get the following:

ort.Test.params
Out[25]: backtrader.metabase.AutoInfoClass_LineRoot_LineMultiple_LineSeries_LineIterator_DataAccessor_StrategyBase_Strategy_Test1

So as a workaround, I put my parameters in a dictionary "parameters" and then passed these parameters as a tuple to params, see below:

   parameters = {'param1': 1}
   params = tuple(parameters.items())

This seems to be working but I am curious to know if there is a better/more elegant way of doing this?

Thanks!

This sounds suspicious. I use futures data with missing days but don't have this issue.
Do you load only one future when running your tests?

Also, you should include your code.

This hedging support offered by FX Brokers and CFD is really a bad idea and I am glad that backtrader is handling orders the way it is doing it at the moment. Which is the way to do it.

I am not sure what benefits do you get from hedging. If you have a loss, you can always exit it and re-enter, do need to supposedly hedge it by entering the opposite direction. The result is the same.