Pyfolio Analyzer doesn't get the right positions

[J. Javier Gálvez]

Hi, Backtraders!

I facing an issue retrieving the positions from a backtesting in Backtrader. I'm using Pyfolio analyzer to try to achieve this but it doesn't work properly.

This is what I got in positions for my strategy:

I tried the following function of Pyfolio but it doesn't work either:

The code I am using in backtrader to run the strategy is this one:

    cerebro = bt.Cerebro()

    # Add pyfolio analyzer
    cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')

    # Data feed --------------------------------
    for equity in equity_list:
        data = get_data(
            hp.cd_dospuntos(os.getcwd(), 2) + '/input_gral/backtest_data/' + equity + '.csv',
            from_date,
            to_date
        )
        cerebro.adddata(data, name=equity)

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Comissions
    cerebro.broker.setcommission(commission=0.001)

    # Configuramos el slipage a 0.5%
    cerebro.broker.set_slippage_perc(perc=0.005,
                                     slip_open=True, slip_limit=True,
                                     slip_match=True, slip_out=False)

    # When working with large percentages it would be good to disable checksubmit in the broker. The rationale being
    # that if the current total portfolio is large, the broker might reject some orders if no cash is left.
    # You could order the orders in such a way that this doesn't happen, but disabling checksubmitwould be easier.
    cerebro.broker.set_checksubmit(checksubmit=False)

    # Strategy
    cerebro.addstrategy(
        TestStrategy,
        buy_and_hold=buy_and_hold,
        sellAll_date=sellAll_date,
    )

    start = time.time()
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Execute
    results = cerebro.run(runonce=False)

    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    end = time.time()

    # Obtiene los resultados para el posterior análisis con pyfolio
    strat = results[0]
    pyfoliozer = strat.analyzers.getbyname('pyfolio')
    returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items()

    # Grabamos los resultados a un archivo pickle
    filepath = 'out/BTesting_results.pkl'
    if buy_and_hold:
        filepath = 'out/BTesting_results_buy_and_hold.pkl'
    file = open(filepath, 'wb')
    pikle_dict = {
        'returns': returns,
        'positions': positions,
        'transactions': transactions,
        'gross_lev': gross_lev,
    }
    pickle.dump(pikle_dict, file)

Any help is welcome. Thanks in advance.

[J. Javier Gálvez]

It seems to be a bug in pyfolio.py:

        pss = self.rets['positions']
        ps = [[k] + v[-2:] for k, v in iteritems(pss)]
        cols = ps.pop(0)  # headers are in the first entry
        positions = DF.from_records(ps, index=cols[0], columns=cols)
        positions.index = pandas.to_datetime(positions.index)
        positions.index = positions.index.tz_localize('UTC')

Line 130 is ps = [[k] + v[-2:] for k, v in iteritems(pss)] but it should be ps = [[k] + v[:] for k, v in iteritems(pss)]. With that change now I get the right positions in my strategy:

If the bug could be confirmed, it would be great to get fixed in a future release.

Happy trading!

[backtrader]

See this commit from 2017-03-12: https://github.com/backtrader/backtrader/commit/e407a583d9cf9e873104c90aa5ee8d155bffa307

        -ps = [[k] + v for k, v in iteritems(pss)]
        +ps = [[k] + v[-2:] for k, v in iteritems(pss)]

Your proposal is to turn this

@j-javier-gálvez said in Pyfolio Analyzer doesn't get the right positions:

ps = [[k] + v[-2:] for k, v in iteritems(pss)]

into

@j-javier-gálvez said in Pyfolio Analyzer doesn't get the right positions:

ps = [[k] + v[:] for k, v in iteritems(pss)]

which literally undo that commit. Back then the commit was done to track the changes in the PyFolio API. After a given point in time it was deemed not sensible to try to track it, given that PyFolio is a tool which will be changed to answer to the demands of the platforms related to it.

See here: Docs - PyFolio Overview

Suggestions:

• You keep your changes as an external analyzer (you put the entire analyzer in your project9
• You monkey patch the method which contains the offending (in your case) code

The only alternative within backtrader would be to add a parameter which would let you select that behavior as opposed to the current one.

[J. Javier Gálvez]

@backtrader Thank you very much for your useful answer. Greetings!

[kunalp]

@J-Javier-Gálvez Thanks. Your solution helped.