Navigation

    Backtrader Community
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Init data gets not recalculated in optstrategy

    General Code/Help
    momentum optstrategy init
    2
    2
    115
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      Jonny8 last edited by

      Hello Everyone,

      I have an issue with the optstrategy. The code seems to reuse the self.ranks dictionary from the init:

      self.ranks = {d: m / v for d, v, m in zip(self.stocks, vs, ms)}

      The Code:

      # <<<Momentum Strategy section>>>
class St(bt.Strategy):
    params = dict(
        selcperc=0.50,  # percentage of stocks to select from the universe
        rperiod=1,  # period for the returns calculation, default 1 period
        vperiod=11,  # lookback period for volatility - default 36 periods
        mperiod=16,  # lookback period for momentum - default 90 periods
        momentum=Momentum, # parametrize the momentum and its period
        reserve=globalparams["reserve"],  # 5% reserve capital
        monthdays=[1],
        monthcarry=True,
        when=bt.timer.SESSION_START,
        benchmarkstop=False, # If true, no stocks will be bought and no rebalancing will be done if benchmark is below SMAperiod
        SMAperiod=200,
        benchmark_bond=True, # Sell all Stocks and buy Bonds
        jump_momentum=True, # If true, after a time of jump_one (30 days x jump_one) in every month, all the money will be directed to the best performing stock. Rule for that:
                            # In Excel, this is a 0.6 x month return of fund with best past 3 month return plus 0.4 x return of fund with best return, month to date.
        jump_one=0.6,
        printlog=True,
    )

    def __init__(self):
        self.bench = self.data0
        self.bond = self.data1
        self.stocks = self.datas[2:]
        # calculate 1st the amount of stocks that will be selected
        self.selnum = int(len(self.stocks) * self.p.selcperc)

        # allocation perc per stock
        # reserve kept to make sure orders are not rejected due to
        # margin. Prices are calculated when known (close), but orders can only
        # be executed next day (opening price). Price can gap upwards
        self.perctarget = (1.0 - self.p.reserve) / self.selnum
        
        # This is the set up of the timer that makes the strategy being executed at the given time
        self.add_timer(
            when=self.p.when,
            monthdays=self.p.monthdays,
            monthcarry=self.p.monthcarry
        )
        
        jump = True

        # returns, volatilities and momentums
        rs = [bt.ind.PctChange(d, period=self.p.rperiod) for d in self.stocks]
        vs = [bt.ind.StdDev(ret, period=self.p.vperiod) for ret in rs]
        #ms = [bt.ind.ROC(d, period=self.p.mperiod) for d in self.datas]
        ms = [self.p.momentum(d, period=self.p.mperiod) for d in self.stocks]
        
        self.bench_sma = bt.ind.SMA(self.data0, period=self.p.SMAperiod)
        
        # simple rank formula: (momentum * net payout) / volatility
        # the highest ranked: low vol, large momentum, large payout
        self.ranks = {d: m / v for d, v, m in zip(self.stocks, vs, ms)}
        #TODO: does it perform better without the volatility?

        self.bench_filter = self.bench < self.bench_sma
        
    def prenext(self):
        # call next() even when data is not available for all tickers
        self.next()     


    def log(self, arg):
        if self.p.printlog:
            print('{} {}'.format(self.datetime.date(), arg))
        
    # This section is for logging of orders in greater detail to figure out whether the strategy is actually having no problem with orders
    
    def notify_order(self, order):
        if order.status in [order.Accepted]:
            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return
        if order.status in [order.Submitted]:
            if order.isbuy():
            
                dt, dn = self.datetime.date(), order.data._name
                print('Buy {} {} {} Price {:.2f} Value {:.2f} Size {} Cash {:.2f}'.format(
                        order.getstatusname(), dt, dn, order.created.price, order.created.size * order.created.price , order.created.size, self.broker.getcash()))
            if order.issell():
                dt, dn = self.datetime.date(), order.data._name
                print('Sell {} {} {} Price {:.2f} Value {:.2f} Size {}'.format(
                        order.getstatusname(), dt, dn, order.created.price, order.created.size * order.created.price, order.created.size))

            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return

        # Check if an order has been completed
        # Attention: broker could reject order if not enough cash
        if order.status in [order.Completed]:
            if order.isbuy():
                dt, dn = self.datetime.date(), order.data._name
                print('Buy {} {} Price {:.2f} Value {:.2f} Size {}'.format(
                    dt, dn, order.executed.price, order.executed.value, order.executed.size))

            if order.issell():# Sell
                dt, dn = self.datetime.date(), order.data._name
                print('Sell {} {} Price {:.2f} Value {:.2f} Size {}'.format(
                    dt, dn, order.executed.price, order.executed.value, order.executed.size))


        elif order.status in [order.Canceled, order.Margin, order.Rejected]:
            self.log('Order Canceled/Margin/Rejected')
           
    
    # This is the function using the timer to execute the rebalance 
    def notify_timer(self, timer, when, *args, **kwargs):
        #print('strategy notify_timer with tid {}, when {} _getminperstatus {}'.
        #      format(timer.p.tid, when, int(self._getminperstatus())))
        if self._getminperstatus() < 0:
            self.rebalance()
  
    def next(self):
        pass # must be filled with a pass

    
    # Actual order giving by a ranking takes place here
    def rebalance(self):
        
        #if jump == True:
        # Enter Jump Code here    
        
        # sort data and current rank
        ranks = sorted(
            self.ranks.items(),  # get the (d, rank), pair
            key=lambda x: x[1][0],  # use rank (elem 1) and current time "0"
            reverse=True,  # highest ranked 1st ... please
        )
        
        # put top ranked in dict with data as key to test for presence
        rtop = dict(ranks[:self.selnum])

        # For logging purposes of stocks leaving the portfolio
        rbot = dict(ranks[self.selnum:])

        # prepare quick lookup list of stocks currently holding a position
        posdata = [d for d, pos in self.getpositions().items() if pos]
        

        if self.p.benchmarkstop:
            for d in (d for d in posdata):
                if "Bond" == d._name and self.bench_filter:
                    return
                else:
                    if "Bond" == d._name and not self.bench_filter:
                        self.order_target_percent("Bond", target=0.0)
                        self.log('Leave {} due to end of down period'.format(d._name))
                        return
            
        # Triple Momentum: If Benchmark index is below SMA, nothing will be bought or rebalanced
        if self.p.benchmarkstop:
            if self.bench_filter:
                #print('SMA {} - Bench {}'.format(self.bench_sma[0], self.bench[0]))
                if self.p.benchmark_bond:
                    for d in posdata:
                        self.log('Leave {} due to switch to Bonds'.format(d._name))
                        self.order_target_percent(d, target=0.0)
                    self.order_target_percent("Bond", target=0.95)
                    self.log('Buy Bond')
                    bond_flag = True
                    return #Code stops here and skips rebalancing und buying
                
        # remove those no longer top ranked
        # do this first to issue sell orders and free cash
        for d in (d for d in posdata if d not in rtop):
            self.log('Leave {} - Rank {:.2f}'.format(d._name, rbot[d][0]))
            self.order_target_percent(d, target=0.0)
        
        # rebalance those already top ranked and still there
        for d in (d for d in posdata if d in rtop):
            self.log('Rebal {} - Rank {:.2f}'.format(d._name, rtop[d][0]))
            self.order_target_percent(d, target=self.perctarget)
            del rtop[d]  # remove it, to simplify next iteration

        # issue a target order for the newly top ranked stocks
        # do this last, as this will generate buy orders consuming cash
        for d in rtop:
            self.log('Enter {} - Rank {:.2f}'.format(d._name, rtop[d][0]))
            self.order_target_percent(d, target=self.perctarget)
            
    def stop(self):
        pnl = round(self.broker.getvalue() - globalparams["cash"],2)
        print('Final PnL: {}'.format(
            pnl))


def run(args=None):
    #optreturn=False otherwise the heatmap doesn't work
    cerebro = bt.Cerebro(maxcpus=1,
                         preload=False,
		                 optdatas=False,
		                 optreturn=False,
		                 stdstats=True) 

    
# <<<Data loading section>>>
    
    # Parse from/to-date
    fromdate = datetime.datetime(2014, 11, 3)
    todate = datetime.datetime(2020, 6, 18)
    
    # Add SPY/QQQ as "Benchmark"
    df0 = pd.read_csv(r'C:\Users\MMD\PycharmProjects\Trading\Data Mining\Data\QQQ.csv', index_col=0, parse_dates=True)
    benchdata = bt.feeds.PandasData(dataname=df0,name="QQQ",fromdate=fromdate, todate=todate,  plot=False)
    cerebro.adddata(benchdata)

    # Add TMF as "Bond"
    df1 = pd.read_csv(r'C:\Users\MMD\PycharmProjects\Trading\Data Mining\Data\TMF.csv', index_col=0, parse_dates=True)
    bonddata = bt.feeds.PandasData(dataname=df1,name="Bond",fromdate=fromdate, todate=todate, plot=False)
    cerebro.adddata(bonddata)    
    
    # add all the data files available in the directory datadir
    for fname in glob.glob(os.path.join(r'C:\Users\MMD\PycharmProjects\Trading\Data Mining\Data\Momentum', '*')):
        df = pd.read_csv(fname, index_col=0, parse_dates=True)

        if len(df)>200:
            cerebro.adddata(bt.feeds.PandasData(dataname=df,name=os.path.basename(fname).replace(".csv", ""),fromdate=fromdate, todate=todate, plot=False))
            #print(os.path.basename(fname).replace(".csv", "")) #prints the name of the added csv file
            
            
# <<<Cerebro loading section>>>

    # add strategy
    #cerebro.addstrategy(HoldAllStrategy, buy_date=datetime.date(2013, 3, 31))
    #cerebro.addstrategy(HSt, buy_date=datetime.date(2014, 11, 3))
    #cerebro.addstrategy(St)
    cerebro.optstrategy(St, mperiod=range(15, 17), vperiod=range(10, 12))
    
    # set the cash, cheat on close and commission
    cerebro.broker.setcash(globalparams["cash"])
    cerebro.broker.set_coc(True)
    cerebro.broker.setcommission(commission=globalparams["commission"])
    
    # Adding Analysers
    #cerebro.addanalyzer(bt.analyzers.SharpeRatio, riskfreerate=0.0, _name='sharpe')
    cerebro.addanalyzer(btanal.PyFolio)                # Needed to use PyFolio
    cerebro.addanalyzer(btanal.TradeAnalyzer)          # Analyzes individual trades
    cerebro.addanalyzer(bt.analyzers.AnnualReturn, _name='annual_return')
    
    
    # If you want to have all data written into a log file
    #cerebro.addwriter(bt.WriterFile, csv=True, out='log.csv')
    
    cerebro.addobserver(bt.observers.Benchmark,
                        data=benchdata,
                        timeframe=bt.TimeFrame.NoTimeFrame)

    results = cerebro.run(maxcpus=1)#maxcpu=1 otherwise pickling multiprocessing errors

    '''
    final_results_list = []
    for run in results:
        for strategy in run:
            PnL = round(strategy.broker.get_value() - globalparams["cash"], 2)
            #my_dict = strategy.analyzers.sharpe.get_analysis()
            my_dict = strategy.analyzers.annual_return.get_analysis()
            annual_returns = [v for _, v in my_dict.items() if v != 0]
            average_annual_return = sum(annual_returns) / len(annual_returns)
            final_results_list.append([strategy.params.mperiod, strategy.params.vperiod, PnL, round(average_annual_return*100, 2)]) # Adjust here the Heatmap variables params y-axis and x-axis #2

    my_heatmap(final_results_list)
    '''
# <<<Performance analysing section section>>>

    #cerebro.plot()

    # Basic performance evaluation ... final value ... minus starting cash
    pnl = cerebro.broker.get_value() - globalparams["cash"]
    print('Profit ... or Loss: {:.2f}'.format(pnl))
    
    # Quantstats thanks to https://algotrading101.com/learn/backtrader-for-backtesting/
    # Does not work with optstrategy
    '''
    returns, positions, transactions, gross_lev = results[0].analyzers.pyfolio.get_pf_items()
    returns.index = returns.index.tz_convert(None)
    qs.reports.html(returns, output='stats.html', title='Momentum')
    webbrowser.open('stats.html')
    '''
    
    # Pyfolio if needed
    '''
    returns, positions, transactions, gross_lev = results[0].analyzers.pyfolio.get_pf_items()
    benchmark_rets = pd.Series([0.00004] * len(returns.index), index=returns.index)     
    pf.create_full_tear_sheet(returns, positions, transactions, benchmark_rets=benchmark_rets)
    '''
    
# <<<Execute starting section>>>    
if __name__ == '__main__':
    run()

      The log (first run is fine... but in the second run the old data starts seeping in):

      2020-05-29 Rebal ARKK - Rank 257.37
2020-05-29 Rebal ARKW - Rank 194.29
Sell Submitted 2020-06-02 ARKK Price 64.80 Value -129.60 Size -2
Sell 2020-06-02 ARKK Price 64.80 Value 107.72 Size -2
Final PnL: 9594.99 
(end of 1. run)
2020-06-18 Enter ARKW - Rank 81.49
2020-06-18 Enter QQQ - Rank 42.31
2015-08-31 Enter ARKK - Rank 5.77
2015-08-31 Enter ARKW - Rank 3.85
Buy Submitted 2015-09-02 ARKK Price 19.37 Value 4745.65 Size 245 Cash 468.19
Buy Submitted 2015-09-02 ARKW Price 21.33 Value 4735.04 Size 222 Cash 468.19
2015-09-02 Order Canceled/Margin/Rejected
2015-09-02 Order Canceled/Margin/Rejected
2020-06-18 Rebal ARKW - Rank 81.49
2020-06-18 Rebal QQQ - Rank 42.31
2015-09-30 Rebal ARKW - Rank 15.38
2015-09-30 Rebal QQQ - Rank 17.73
Buy Submitted 2015-10-02 ARKW Price 21.13 Value 21.13 Size 1 Cash 425.72
Buy 2015-10-02 ARKW Price 21.13 Value 21.13 Size 1
2020-06-18 Rebal ARKW - Rank 81.49
2020-06-18 Rebal QQQ - Rank 42.31
2015-10-30 Leave ARKW - Rank 105.01
2015-10-30 Rebal QQQ - Rank 364.09
2015-10-30 Enter EUNL.DE - Rank 338.88
Sell Submitted 2015-11-03 ARKW Price 23.03 Value -5158.72 Size -224
Sell Submitted 2015-11-03 QQQ Price 114.61 Value -114.61 Size -1
Buy Submitted 2015-11-03 EUNL.DE Price 38.21 Value 5196.56 Size 136 Cash 517.89
Sell 2015-11-03 ARKW Price 23.03 Value 4780.67 Size -224
Sell 2015-11-03 QQQ Price 114.61 Value 101.05 Size -1
Buy 2015-11-03 EUNL.DE Price 38.21 Value 5196.56 Size 136
2020-06-18 Leave EUNL.DE - Rank 0.01
2020-06-18 Rebal ARKW - Rank 81.49
2020-06-18 Rebal QQQ - Rank 42.31
2015-11-30 Leave ARKW - Rank 90.69
2015-11-30 Leave QQQ - Rank 39.61
2015-11-30 Rebal EUNL.DE - Rank 143.75
2015-11-30 Enter ARKK - Rank 108.25

      I am not sure how to deal with it...
      First I tried to change the cerebro settings:
      https://www.backtrader.com/docu/cerebro/

      Didn't work for me...

      I tried to put it in Nextstart(self) ... didn't work

      Then I tried to empty or set the dictionaries to none... also didn't work

      Is there another way?

      Thank you very much in advance!

      vladisld 1 Reply Last reply Reply Quote 0
      • vladisld
        vladisld @Jonny8 last edited by

        @Jonny8 said in Init data gets not recalculated in optstrategy:

        pnl = cerebro.broker.get_value() - globalparams["cash"]

        Just a note:

        I'm not sure it is a good idea to query the info from the original Cerebro instance's broker.

        In multi-CPU case the original Cerebro instance's broker is not updated. In single CPU case (as in the code above), although the same broker instance is used for all the optimization parameter permutation runs, the broker state is reset for each permutation. So querying the broker in this case will return the value for the last optimization run.

        It is better to rely only on the data gathered by the analyzers.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors