Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    1. Home
    2. Rapha
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
    R
    • Profile
    • Following 0
    • Followers 0
    • Topics 3
    • Posts 9
    • Best 2
    • Controversial 0
    • Groups 0

    Rapha

    @Rapha

    2
    Reputation
    2
    Profile views
    9
    Posts
    0
    Followers
    0
    Following
    Joined Last Online

    Rapha Unfollow Follow

    Best posts made by Rapha

    • RE: getposition(data).price & self.broker.getcash() return unexpected values

      With my previous remark in mind, I doubled checked the cash formula above under point 1, and get the expected results using the opening prices out of my input (and not getposition...).

      So problem solved.

      posted in General Code/Help
      R
      Rapha
    • RE: getposition(data).price & self.broker.getcash() return unexpected values

      I found the explanation to my second question, so I'll post it there :
      getposition(data).price returns the cost basis. For instance:

      • If all the positions on a given share where bought the same day, it returns the open price of that day.
      • If all the positions on a given share where bought the previous day, it returns the previous day open price.
      • If the positions have been accumulated over multiple periods, it returns the average price.

      Hope it might help some else.

      posted in General Code/Help
      R
      Rapha

    Latest posts made by Rapha

    • Passing params to my strategy

      Hi there,

      I am in the process of segmenting and packaging different part of my code, and the backtrader part containing my strategy (class ML_strategy(bt.Strategy)) is now isolated from the rest of in a package (backtest).

      For this reason, I need to pass the params to my strategy without defining them in the class as I used to do:

      class ML_strategy(bt.Strategy):
          params = (('target_n_positions', 20),
                    ('min_n_positions', 5),
                    ('min_position_value', 2000),
                    ('rebalancing_frequency', 'daily'),
                    ('rebalancing_direction', 'long'),
                    ('verbose', False))
      

      I have tried two methods, which both give me some errors:

      1. Setting the class variables this way from my main.py:
                  backtest.ML_strategy.params = (('target_n_positions', 20),
                                                 ('min_n_positions', 5),
                                                 ('min_position_value', 2000),
                                                 ('rebalancing_frequency','daily'),
                                                 ('rebalancing_direction','long'),
                                                 ('verbose', False))
      
                  cerebro.addstrategy(backtest.ML_strategy)
      

      which gives me the following error:

      File "C:\Users\rapha\Documents\GitHub\algorithms\venv\lib\site-packages\backtrader\metabase.py", line 283, in donew
        params = cls.params()
      TypeError: 'tuple' object is not callable
      
      1. Providing the instance variables in addstrategy:
      cerebro.addstrategy(backtest.ML_strategy,target_n_positions=20,min_n_positions=5,min_position_value=2000,rebalancing_frequency='daily',rebalancing_direction='long',verbose=False)
      

      which gives me the following error:

      File "C:\Users\rapha\Documents\GitHub\algorithms\venv\lib\site-packages\backtrader\metabase.py", line 283, in donew
        params = cls.params()
      TypeError: 'tuple' object is not callable
      

      Additional information: I have switch to Python 3.10. I used to work on python 3.6 for backtrader, not sure if that might be the causing the issue as well.

      Any idea on what I might be doing wrong here?

      Thanks in advance
      Rapha

      posted in General Code/Help
      R
      Rapha
    • RE: getposition(data).price & self.broker.getcash() return unexpected values

      With my previous remark in mind, I doubled checked the cash formula above under point 1, and get the expected results using the opening prices out of my input (and not getposition...).

      So problem solved.

      posted in General Code/Help
      R
      Rapha
    • RE: getposition(data).price & self.broker.getcash() return unexpected values

      I found the explanation to my second question, so I'll post it there :
      getposition(data).price returns the cost basis. For instance:

      • If all the positions on a given share where bought the same day, it returns the open price of that day.
      • If all the positions on a given share where bought the previous day, it returns the previous day open price.
      • If the positions have been accumulated over multiple periods, it returns the average price.

      Hope it might help some else.

      posted in General Code/Help
      R
      Rapha
    • getposition(data).price & self.broker.getcash() return unexpected values

      Hi there,

      There are two things I cannot get my mind around, probably because of wrong assumptions or understanding on my end. The code I am using is shown at end, which is run with data drawn from AlphaVantage.

      1. I do not understand how the cash balance returned by self.broker.getcash() is computed.
        My expectation:
        Cash_t = Cash_t-1 + Value of positions_t-1 closed valued at open price of day t - Value of positions_t opened valued at open price of day t
        My underlying assumptions:
        -Orders are executed daily at market opening at the open prices --> previous day's positions closed & current day's positions opened
        -No commissions are paid as no commission info are delivered through cerebro.broker.addcommissioninfo.

      2. Why does self.getposition(data=ticker).price does not always return the open price.
        I checked for some date references, and there are many occurences where the price returned by getposition(data).price does not correspond to the open price provided in my input (neither the close price). Below is an example for a given date:

      My log:
      9ef7f30e-187e-444a-9995-d4cc5b29dedc-image.png

      My input data:
      2d7a5b18-3d50-4ddb-b757-92bf6b189cc5-image.png

      Additional remark: the price at which the orders have been executed correspond to the open price, according to my log.

      The code:

      from pathlib import Path
      import csv
      from time import time
      import datetime
      import numpy as np
      import pandas as pd
      import pandas_datareader.data as web
      import matplotlib.pyplot as plt
      import seaborn as sns
      
      #import pandas_market_calendars as mcal
      
      import backtrader as bt
      from backtrader.feeds import PandasData
      
      import quantstats as qs
      
      
      pd.set_option('display.expand_frame_repr', False)
      np.random.seed(42)
      sns.set_style('darkgrid')
      
      def format_time(t):
          m_, s = divmod(t, 60)
          h, m = divmod(m_, 60)
          return f'{h:>02.0f}:{m:>02.0f}:{s:>02.0f}'
         
      #----------------- DATAFRAME LOADER -----------------
      OHLCV = ['open', 'high', 'low', 'close', 'volume']
      class SignalData(PandasData):
          """
          Define pandas DataFrame structure
          """
          cols = OHLCV + ['predicted']
      
          # create lines
          lines = tuple(cols)
      
          # define parameters
          params = {c: -1 for c in cols}
          params.update({'datetime': None})
          params = tuple(params.items())
          
      #----------------- STRATEGY -----------------
      #Includes an option to only trade on certain weekdays in lines 39/40.
      class MLStrategy(bt.Strategy):
          params = (('n_positions', 20),
                    ('min_positions', 10),
                    ('verbose', False),
                    ('log_file', 'backtest.csv'))
      
          def log(self, txt, dt=None):
              """ Logger for the strategy"""
              dt = dt or self.datas[0].datetime.datetime(0)
              with Path(self.p.log_file).open('a') as f:
                  log_writer = csv.writer(f)
                  log_writer.writerow([dt.isoformat()] + txt.split(','))
      
          def notify_order(self, order):
              if order.status in [order.Submitted, order.Accepted]:
                  if order.status in [order.Submitted]:
                      self.log(f'{order.data._name},SUBMITTED')
                  if order.status in [order.Accepted]:
                      self.log(f'{order.data._name},ACCEPTED')
                  return
      
              # Check if an order has been completed
              # broker could reject order if not enough cash
              if self.p.verbose:
                  if order.status in [order.Completed]:
                      p = order.executed.price
                      if order.isbuy():
                          self.log(f'{order.data._name},BUY executed,{p:.2f}')
                      elif order.issell():
                          self.log(f'{order.data._name},SELL executed,{p:.2f}')
      
                  elif order.status in [order.Canceled]:
                      self.log(f'{order.data._name},Order Canceled')
                  elif order.status in [order.Margin]:
                      self.log(f'{order.data._name},Order Margin')
                  elif order.status in [order.Rejected]:
                      self.log(f'{order.data._name},Order Rejected')
      
          def prenext(self):
              self.next()
              self.log('prenext')
      
          def next(self):
              self.log('next')
              today = self.datas[0].datetime.date()
      
              positions = [d._name for d, pos in self.getpositions().items() if pos]
              posdata = [d for d, pos in self.getpositions().items() if pos]    
      
              up, down = {}, {}
              missing = not_missing = 0
              for data in self.datas:
                  if data.datetime.date() == today:
                      if data.predicted[0] > 0:
                          up[data._name] = data.predicted[0]
                      elif data.predicted[0] < 0:
                          down[data._name] = data.predicted[0]
      
              self.log(f'{self.broker.getvalue()},PORTFOLIO VALUE')
              self.log(f'{self.broker.getcash()},CASH VALUE')
              
              for ticker in posdata:
                  self.log(f'{ticker._name,self.getposition(data=ticker).size,self.getposition(data=ticker).price,self.getposition(data=ticker).size*self.getposition(data=ticker).price},POSITION')
                             
              shorts = sorted(down, key=down.get)[:self.p.n_positions]
              longs = sorted(up, key=up.get, reverse=True)[:self.p.n_positions]
              n_shorts, n_longs = len(shorts), len(longs)
      
              if n_shorts < self.p.min_positions+1 or n_longs < self.p.min_positions+1:
                  longs, shorts = [], []
              else:
                  short_target = -1 / n_shorts
                  long_target = 1 / n_longs
                  
              for ticker in positions:
                  if ticker not in longs + shorts:
                      self.order_target_percent(data=ticker, target=0)
                      self.log(f'{ticker},CLOSING ORDER CREATED')
                      
              for ticker in longs:
                  self.order_target_percent(data=ticker, target=long_target)
                  self.log(f'{ticker,long_target},LONG ORDER CREATED')
              for ticker in shorts:
                  self.order_target_percent(data=ticker, target=short_target)
                  self.log(f'{ticker,short_target},SHORT ORDER CREATED')
                  
      cerebro = bt.Cerebro()  # create a "Cerebro" instance
      cash = 1000000
      
      cerebro.broker.setcash(cash)
      
      #------------------------------ ADD INPUT DATA --------------------------------
      idx = pd.IndexSlice
      data = pd.read_hdf('data.h5', 'backtest_data').sort_index()
      
      tickers = data.index.get_level_values(0).unique()
      
      for ticker in tickers:
          df = data.loc[idx[ticker, :], :].droplevel('ticker', axis=0)
          df.index.name = 'datetime'
          bt_data = SignalData(dataname=df)
          cerebro.adddata(bt_data, name=ticker)
          
      #---------------------------- RUN STRATEGY BACKTEST ---------------------------
      
      cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
      cerebro.addstrategy(MLStrategy, n_positions=20, min_positions=10,
                          verbose=True, log_file='backtesting_backtrader_log.csv')
      start = time()
      results = cerebro.run()
      ending_value = cerebro.broker.getvalue()
      duration = time() - start
      
      print(f'Final Portfolio Value: {ending_value:,.2f}')
      print(f'Duration: {format_time(duration)}')```
      posted in General Code/Help
      R
      Rapha
    • RE: Strange behavior around holidays

      @ab_trader: I have looked into my data and while I am supposed to only have tickers traded on the NYSE in my feed, 5 of them were actually traded on European and Asian markets, following a slightly different calendar for holidays. Removing them solved the problem rergarding the two calls of next on 24-03-2016, and my holdings are now very closely aligned in Zipline and Backtrader every days! thanks for the help.

      I still have some discrepancies in the returns, apparently caused by uneven weighting of my positions in Backtrader, which should follow a simple 1/N weighting scheme. I have not had time to look into this yet, and it is not even related to my orginal question. But if there is something odd you spot in my code, pls let me know.

      posted in General Discussion
      R
      Rapha
    • RE: Strange behavior around holidays

      @ab_trader good point. 23rd as 1033 tickers. I have a total of 1034 tickers in my data feed, which I have for some days, but apparently not all the time:

      len(data.index.get_level_values(0).unique())
      Out[48]: 1034
      
      len(data.xs('2016-03-23',level = 1, drop_level = False).index.get_level_values(0).unique())
      Out[49]: 1033
      
      len(data.xs('2016-03-03',level = 1, drop_level = False).index.get_level_values(0).unique())
      Out[50]: 1034
      

      I will investigate the reason of these differences and post the explanation back here.

      posted in General Discussion
      R
      Rapha
    • RE: Strange behavior around holidays

      @ab_trader
      Thanks for the heads up.

      As you suggested, I have investigated the data and unfortunately, it does not look like in my case that it is the root cause of the problem.

      On 2016-03-24, I only have unique references:

      len(data.xs('2016-03-24',level = 1, drop_level = False).index)
      Out: 1033
      
      len(data.xs('2016-03-24',level = 1, drop_level = False).index.unique())
      Out: 1033
      

      On 2016-03-28, there are data as well:

      len(data.xs('2016-03-28',level = 1, drop_level = False).index)
      Out: 1030
      
      len(data.xs('2016-03-28',level = 1, drop_level = False).index.unique())
      Out: 1030
      
      posted in General Discussion
      R
      Rapha
    • RE: Strange behavior around holidays

      @ab_trader Thanks for your feedback.

      I removed from my code whatever was not necessary to reproduce the issue (in my environment at least):

      from pathlib import Path
      import csv
      from time import time
      import numpy as np
      import pandas as pd
      import seaborn as sns
      
      import backtrader as bt
      from backtrader.feeds import PandasData
      
      pd.set_option('display.expand_frame_repr', False)
      np.random.seed(42)
      sns.set_style('darkgrid')
      
      def format_time(t):
          m_, s = divmod(t, 60)
          h, m = divmod(m_, 60)
          return f'{h:>02.0f}:{m:>02.0f}:{s:>02.0f}'
          
      #----------------- DATAFRAME LOADER -----------------
      OHLCV = ['open', 'high', 'low', 'close', 'volume']
      class SignalData(PandasData):
          cols = OHLCV + ['predicted']
      
          lines = tuple(cols)
      
          params = {c: -1 for c in cols}
          params.update({'datetime': None})
          params = tuple(params.items())
          
      #----------------- STRATEGY -----------------
      class MLStrategy(bt.Strategy):
          params = (('n_positions', 20),
                    ('min_positions', 10),
                    ('verbose', False),
                    ('log_file', 'backtest.csv'))
      
          def log(self, txt, dt=None):
              """ Logger for the strategy"""
              dt = dt or self.datas[0].datetime.datetime(0)
              with Path(self.p.log_file).open('a') as f:
                  log_writer = csv.writer(f)
                  log_writer.writerow([dt.isoformat()] + txt.split(','))
      
          def notify_order(self, order):
              if order.status in [order.Submitted, order.Accepted]:
                  if order.status in [order.Submitted]:
                      self.log(f'{order.data._name},SUBMITTED')
                  if order.status in [order.Accepted]:
                      self.log(f'{order.data._name},ACCEPTED')
                  return
      
              if self.p.verbose:
                  if order.status in [order.Completed]:
                      p = order.executed.price
                      if order.isbuy():
                          self.log(f'{order.data._name},BUY executed,{p:.2f}')
                      elif order.issell():
                          self.log(f'{order.data._name},SELL executed,{p:.2f}')
      
                  elif order.status in [order.Canceled]:
                      self.log(f'{order.data._name},Order Canceled')
                  elif order.status in [order.Margin]:
                      self.log(f'{order.data._name},Order Margin')
                  elif order.status in [order.Rejected]:
                      self.log(f'{order.data._name},Order Rejected')
      
          def prenext(self):
              self.next()
      
          def next(self):
              self.log('next')
              today = self.datas[0].datetime.date()
      
              positions = [d._name for d, pos in self.getpositions().items() if pos]
              posdata = [d for d, pos in self.getpositions().items() if pos]    
              
              up, down = {}, {}
              for data in self.datas:
                  if data.datetime.date() == today:
                      if data.predicted[0] > 0:
                          up[data._name] = data.predicted[0]
                      elif data.predicted[0] < 0:
                          down[data._name] = data.predicted[0]
                  
              for ticker in posdata:
                  self.log(f'{ticker._name,self.getposition(data=ticker).size},POSITION')
      
              shorts = sorted(down, key=down.get)[:self.p.n_positions]
              longs = sorted(up, key=up.get, reverse=True)[:self.p.n_positions]
              n_shorts, n_longs = len(shorts), len(longs)
              
              if n_shorts < self.p.min_positions or n_longs < self.p.min_positions:
                  longs, shorts = [], []
              else:
                  short_target = -1 / n_shorts
                  long_target = 1 / n_longs
                  
              
              for ticker in positions:
                  if ticker not in longs + shorts:
                      self.order_target_percent(data=ticker, target=0)
                      self.log(f'{ticker},CLOSING ORDER CREATED')
                  
              for ticker in shorts:
                  self.order_target_percent(data=ticker, target=short_target)
                  self.log(f'{ticker},SHORT ORDER CREATED')
              for ticker in longs:
                  self.order_target_percent(data=ticker, target=long_target)
                  self.log(f'{ticker},LONG ORDER CREATED')
      
      
      cerebro = bt.Cerebro()
      
      cash = 1000000
      
      cerebro.broker.setcash(cash)
      
      #------------------------------ ADD INPUT DATA --------------------------------
      idx = pd.IndexSlice
      data = pd.read_hdf('data.h5', 'backtest_data').sort_index()
      tickers = data.index.get_level_values(0).unique()
      
      for ticker in tickers:
          df = data.loc[idx[ticker, :], :].droplevel('ticker', axis=0)
          df.index.name = 'datetime'
          bt_data = SignalData(dataname=df)
          cerebro.adddata(bt_data, name=ticker)
          
      #---------------------------- RUN STRATEGY BACKTEST ---------------------------
      #cerebro.addcalendar(nyse)
      cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
      cerebro.addstrategy(MLStrategy, n_positions=20, min_positions=10,
                          verbose=True, log_file='backtesting_backtrader_log.csv')
      start = time()
      results = cerebro.run()
      ending_value = cerebro.broker.getvalue()
      duration = time() - start
      

      Unfortunately the log generated from 24-03-2016 to 30-03-2016 is too long to be posted here in its entirety. I just left the first 3 lines of each part of the log (3 first tickers) and replaced the rest with 3 dots. I hope it's still helpful enough.

      2016-03-24T00:00:00,CNX,SUBMITTED
      2016-03-24T00:00:00,DDD,SUBMITTED
      2016-03-24T00:00:00,HOV,SUBMITTED
      ...
      2016-03-24T00:00:00,CNX,ACCEPTED
      2016-03-24T00:00:00,DDD,ACCEPTED
      2016-03-24T00:00:00,HOV,ACCEPTED
      ...
      2016-03-24T00:00:00,CNX,BUY executed,10.27
      2016-03-24T00:00:00,DDD,BUY executed,14.14
      2016-03-24T00:00:00,HOV,SELL executed,38.00
      ...
      2016-03-24T00:00:00,next
      2016-03-24T00:00:00,('AG', -6266),POSITION
      2016-03-24T00:00:00,('AKS', -9768),POSITION
      ...
      2016-03-24T00:00:00,AKS,CLOSING ORDER CREATED
      2016-03-24T00:00:00,AU,CLOSING ORDER CREATED
      2016-03-24T00:00:00,DNR,CLOSING ORDER CREATED
      ...
      2016-03-24T00:00:00,HMY,SHORT ORDER CREATED
      2016-03-24T00:00:00,TECK,SHORT ORDER CREATED
      2016-03-24T00:00:00,VAL,SHORT ORDER CREATED
      ...
      2016-03-24T00:00:00,BTU,LONG ORDER CREATED
      2016-03-24T00:00:00,SALT,LONG ORDER CREATED
      2016-03-24T00:00:00,BHC,LONG ORDER CREATED
      ...
      2016-03-24T00:00:00,AKS,SUBMITTED
      2016-03-24T00:00:00,AU,SUBMITTED
      2016-03-24T00:00:00,DNR,SUBMITTED
      ...
      2016-03-24T00:00:00,AKS,ACCEPTED
      2016-03-24T00:00:00,AU,ACCEPTED
      2016-03-24T00:00:00,DNR,ACCEPTED
      ...
      2016-03-24T00:00:00,AKS,BUY executed,4.19
      2016-03-24T00:00:00,AU,BUY executed,12.73
      2016-03-24T00:00:00,DNR,SELL executed,2.25
      ...
      2016-03-24T00:00:00,next
      2016-03-24T00:00:00,('AG', -5817),POSITION
      2016-03-24T00:00:00,('AMRX', 1200),POSITION
      2016-03-24T00:00:00,('AUY', -13857),POSITION
      ...
      2016-03-24T00:00:00,AG,CLOSING ORDER CREATED
      2016-03-24T00:00:00,AMRX,CLOSING ORDER CREATED
      2016-03-24T00:00:00,AUY,CLOSING ORDER CREATED
      ...
      2016-03-29T00:00:00,AG,SUBMITTED
      2016-03-29T00:00:00,AMRX,SUBMITTED
      2016-03-29T00:00:00,AUY,SUBMITTED
      ...
      2016-03-29T00:00:00,AG,ACCEPTED
      2016-03-29T00:00:00,AMRX,ACCEPTED
      2016-03-29T00:00:00,AUY,ACCEPTED
      ...
      2016-03-29T00:00:00,AG,BUY executed,6.42
      2016-03-29T00:00:00,AMRX,SELL executed,31.22
      2016-03-29T00:00:00,AUY,BUY executed,2.80
      ...
      2016-03-29T00:00:00,next
      2016-03-29T00:00:00,VAL,SHORT ORDER CREATED
      2016-03-29T00:00:00,CDE,SHORT ORDER CREATED
      2016-03-29T00:00:00,AG,SHORT ORDER CREATED
      ...
      2016-03-30T00:00:00,VAL,SUBMITTED
      2016-03-30T00:00:00,CDE,SUBMITTED
      2016-03-30T00:00:00,AG,SUBMITTED
      ...
      2016-03-30T00:00:00,VAL,ACCEPTED
      2016-03-30T00:00:00,CDE,ACCEPTED
      2016-03-30T00:00:00,AG,ACCEPTED
      ...
      2016-03-30T00:00:00,VAL,SELL executed,105.20
      2016-03-30T00:00:00,CDE,SELL executed,5.64
      2016-03-30T00:00:00,AG,SELL executed,6.84
      ...
      2016-03-30T00:00:00,next
      2016-03-30T00:00:00,('AG', -5418),POSITION
      2016-03-30T00:00:00,('AMRX', 1143),POSITION
      2016-03-30T00:00:00,('AUY', -12576),POSITION
      ...
      2016-03-30T00:00:00,AMRX,CLOSING ORDER CREATED
      2016-03-30T00:00:00,AZO,CLOSING ORDER CREATED
      2016-03-30T00:00:00,CYH,CLOSING ORDER CREATED
      ...
      

      I hope it helps.
      Best,
      Rapha

      posted in General Discussion
      R
      Rapha
    • Strange behavior around holidays

      Hi there,

      I am in the process of switching from Zipline to Backtrader, and analyze the difference between the two backtrading engines for some algos.

      While the holdings and returns are very close for some periods, I have big variations at some points in time. One source of difference I have identified occurs around holidays where the algo seems to be looping twice on the day preceding the holiday, and behave in a fashion I cannot explain shortly after.

      I have for instance identified this behavior the around Good Friday 2016 (24-03-2016), where the behavior is the following:

      • 24-03-2016 --> algo loops twice through next
      • 25-03-2016 --> Good Friday (market closed)
      • 26-03-2016 & 27-03-2016 --> weekend (market closed)
      • 28-03-2016 --> nothing happens
      • 29-03-2016 --> orders are submitted, accepted and executed, but no position is reported for that day in my log
      • 30-03-2016 --> back to normal

      As you'll see in my code, I tried two different options to add a calendar (one commented) as it seemed to the issue here, but it did not help.

      Below is my code which is a slightly modified version of the the following script.
      https://github.com/PacktPublishing/Machine-Learning-for-Algorithmic-Trading-Second-Edition/blob/master/08_ml4t_workflow/03_backtesting_with_backtrader.ipynb

      Remarks

      • My data feed does not contain any values for the holidays, but does for every other days when the market is open (28-03-2016 for instance)
      • My signal (predicted column of self.datas) is the output of a simple regression.

      My knowledge of backtrader is quite limited yet, and this issue has gotten me stuck for a while now. Any help would be highly appreciated. Please let me know if the log for the days mentionned above would help.

      from pathlib import Path
      import csv
      from time import time
      import datetime
      import numpy as np
      import pandas as pd
      import pandas_datareader.data as web
      import matplotlib.pyplot as plt
      import seaborn as sns
      
      import pandas_market_calendars as mcal
      
      import backtrader as bt
      from backtrader.feeds import PandasData
      
      import quantstats as qs
      
      #nyse = mcal.get_calendar('NYSE')
      class NYSE_2016(bt.TradingCalendar):
          params = dict(
              holidays=[
                  datetime.date(2016, 1, 1),
                  datetime.date(2016, 1, 18),
                  datetime.date(2016, 2, 15),
                  datetime.date(2016, 3, 25),
                  datetime.date(2016, 5, 30),
                  datetime.date(2016, 7, 4),
                  datetime.date(2016, 9, 5),
                  datetime.date(2016, 11, 24),
                  datetime.date(2016, 12, 26),
              ]
          )
      
      pd.set_option('display.expand_frame_repr', False)
      np.random.seed(42)
      sns.set_style('darkgrid')
      
      def format_time(t):
          m_, s = divmod(t, 60)
          h, m = divmod(m_, 60)
          return f'{h:>02.0f}:{m:>02.0f}:{s:>02.0f}'
      
      #----------------------------- BACKTRADER SETUP -------------------------------
      class FixedCommisionScheme(bt.CommInfoBase):
          """
          Simple fixed commission scheme for demo
          """
          params = (
              ('commission', .02),
              ('stocklike', True),
              ('commtype', bt.CommInfoBase.COMM_FIXED),
          )
      
          def _getcommission(self, size, price, pseudoexec):
              return abs(size) * self.p.commission
          
      #----------------- DATAFRAME LOADER -----------------
      OHLCV = ['open', 'high', 'low', 'close', 'volume']
      class SignalData(PandasData):
          """
          Define pandas DataFrame structure
          """
          cols = OHLCV + ['predicted']
      
          # create lines
          lines = tuple(cols)
      
          # define parameters
          params = {c: -1 for c in cols}
          params.update({'datetime': None})
          params = tuple(params.items())
          
      #----------------- STRATEGY -----------------
      class MLStrategy(bt.Strategy):
          params = (('n_positions', 20),
                    ('min_positions', 10),
                    ('verbose', False),
                    ('log_file', 'backtest.csv'))
      
          def log(self, txt, dt=None):
              """ Logger for the strategy"""
              dt = dt or self.datas[0].datetime.datetime(0)
              with Path(self.p.log_file).open('a') as f:
                  log_writer = csv.writer(f)
                  log_writer.writerow([dt.isoformat()] + txt.split(','))
      
          def notify_order(self, order):
              if order.status in [order.Submitted, order.Accepted]:
                  if order.status in [order.Submitted]:
                      self.log(f'{order.data._name},SUBMITTED')
                  if order.status in [order.Accepted]:
                      self.log(f'{order.data._name},ACCEPTED')
                  return
      
              if self.p.verbose:
                  if order.status in [order.Completed]:
                      p = order.executed.price
                      if order.isbuy():
                          self.log(f'{order.data._name},BUY executed,{p:.2f}')
                      elif order.issell():
                          self.log(f'{order.data._name},SELL executed,{p:.2f}')
      
                  elif order.status in [order.Canceled]:
                      self.log(f'{order.data._name},Order Canceled')
                  elif order.status in [order.Margin]:
                      self.log(f'{order.data._name},Order Margin')
                  elif order.status in [order.Rejected]:
                      self.log(f'{order.data._name},Order Rejected')
      
          def prenext(self):
              self.next()
      
          def next(self):
              self.log('next')
              today = self.datas[0].datetime.date()
      
              positions = [d._name for d, pos in self.getpositions().items() if pos]
              posdata = [d for d, pos in self.getpositions().items() if pos]   
              
              up, down = {}, {}
              missing = not_missing = 0
              for data in self.datas:
                  if data.datetime.date() == today:
                      if data.predicted[0] > 0:
                          up[data._name] = data.predicted[0]
                      elif data.predicted[0] < 0:
                          down[data._name] = data.predicted[0]
                  
              for ticker in posdata:
                  self.log(f'{ticker._name,self.getposition(data=ticker).size},POSITION')
                  
              shorts = sorted(down, key=down.get)[:self.p.n_positions]
              longs = sorted(up, key=up.get, reverse=True)[:self.p.n_positions]
              n_shorts, n_longs = len(shorts), len(longs)
      
              if n_shorts < self.p.min_positions or n_longs < self.p.min_positions:
                  longs, shorts = [], []
              else:
                  short_target = -1 / n_shorts
                  long_target = 1 / n_longs
                  
              
              for ticker in positions:
                  if ticker not in longs + shorts:
                      self.order_target_percent(data=ticker, target=0)
                      self.log(f'{ticker},CLOSING ORDER CREATED')
                  
              for ticker in shorts:
                  self.order_target_percent(data=ticker, target=short_target)
                  self.log(f'{ticker},SHORT ORDER CREATED')
              for ticker in longs:
                  self.order_target_percent(data=ticker, target=long_target)
                  self.log(f'{ticker},LONG ORDER CREATED')
      
                  
      #CREATE AND CONFIGURE CEREBRO INSTANCE
      cerebro = bt.Cerebro() 
      cash = 1000000
      cerebro.broker.setcash(cash)
      
      #------------------------------ ADD INPUT DATA --------------------------------
      idx = pd.IndexSlice
      data = pd.read_hdf('data.h5', 'backtest_data').sort_index()
      tickers = data.index.get_level_values(0).unique()
      
      for ticker in tickers:
          df = data.loc[idx[ticker, :], :].droplevel('ticker', axis=0)
          df.index.name = 'datetime'
          bt_data = SignalData(dataname=df)
          cerebro.adddata(bt_data, name=ticker)
          
      #---------------------------- RUN STRATEGY BACKTEST ---------------------------
      #cerebro.addcalendar(nyse)
      cerebro.addcalendar(NYSE_2016)
      cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
      cerebro.addstrategy(MLStrategy, n_positions=20, min_positions=10,
                          verbose=True, log_file='backtesting_backtrader_log.csv')
      start = time()
      results = cerebro.run()
      ending_value = cerebro.broker.getvalue()
      duration = time() - start
      
      print(f'Final Portfolio Value: {ending_value:,.2f}')
      print(f'Duration: {format_time(duration)}')
      
      #GET PYFOLIO INPUTS
      pyfolio_analyzer = results[0].analyzers.getbyname('pyfolio')
      returns, positions, transactions, gross_lev = pyfolio_analyzer.get_pf_items()
      
      returns.rename_axis(index={'index':'date'})
      gross_lev.rename_axis(index={'index':'date'})
      positions.rename_axis(index={'Datetime':'date'})
      
      returns.to_hdf('backtest.h5', 'backtrader/returns')
      positions.to_hdf('backtest.h5', 'backtrader/positions')
      transactions.to_hdf('backtest.h5', 'backtrader/transactions')
      gross_lev.to_hdf('backtest.h5', 'backtrader/gross_lev')
      
      #------------------------------- RUN PYFOLIO ----------------------------------
      returns = pd.read_hdf('backtest.h5', 'backtrader/returns')
      positions = pd.read_hdf('backtest.h5', 'backtrader/positions')
      transactions = pd.read_hdf('backtest.h5', 'backtrader/transactions')
      gross_lev = pd.read_hdf('backtest.h5', 'backtrader/gross_lev')
      
      benchmark = web.DataReader('SP500', 'fred', '2014', '2018').squeeze()
      benchmark = benchmark.pct_change().tz_localize('UTC')
      
      daily_tx = transactions.groupby(level=0)
      longs = daily_tx.value.apply(lambda x: x.where(x>0).sum())
      shorts = daily_tx.value.apply(lambda x: x.where(x<0).sum())
      
      fig, axes = plt.subplots(ncols=2, figsize=(15, 5))
      
      df = returns.to_frame('Strategy').join(benchmark.to_frame('Benchmark (S&P 500)'))
      df.add(1).cumprod().sub(1).plot(ax=axes[0], title='Cumulative Return')
      
      longs.plot(label='Long',ax=axes[1], title='Positions')
      shorts.plot(ax=axes[1], label='Short')
      positions.cash.plot(ax=axes[1], label='PF Value')
      axes[1].legend()
      sns.despine()
      fig.tight_layout()
      
      plt.show()
      plt.close()
      
      posted in General Discussion
      R
      Rapha