Hi, @backtrader and Bactraders!

I tried the second option but it didn't go as expected. I read the documentation and review every post about the backfill_from and tried several things from there without success.

The issue is that the timestamp I get from the data when the next() is called jump from 12:00 to 5:00 of the other day instead of continue to 12:01 and so on until 16:00 and then 9:30 of the next day. Here an extract of my prints:

2018-10-01T11:56:00
2018-10-01T11:57:00
2018-10-01T11:58:00
2018-10-01T11:59:00
2018-10-01T12:00:00
2018-10-02T05:31:00
2018-10-02T05:32:00
2018-10-02T05:33:00
2018-10-02T05:34:00
2018-10-02T05:35:00
2018-10-02T05:36:00
2018-10-02T05:37:00

I'm very sure that my local CSVs have data only between 9:30 and 16:00 hrs.

Let me detail about my code. This is how I'm connecting to IB:

cerebro = bt.Cerebro()
store = bt.stores.IBStore(host='127.0.0.1',port=4002, clientId=36, reconnect=-1)

My data is in minute bars and I'm reading it as follow:

data0 = btfeed.GenericCSVData(
        dataname=file_path,

        fromdate=datetime.datetime(2018, 10, 1, 9, 30, 0),
        todate=datetime.datetime(2018, 12, 12, 9, 30, 0),

        timeframe=bt.TimeFrame.Minutes,
        compression=1,

        dtformat='%Y%m%d',
        tmformat='%H:%M:%S',

        datetime=0,
        time=1,
        open=3,
        high=4,
        low=5,
        close=6,
        volume=8,
        openinterest=-1,
)

Then, adding the data:

contract = 'TWTR-STK-SMART-USD'
data = store.getdata(dataname=contract, rtbar=True,
                             timeframe=bt.TimeFrame.Ticks, historical=False,
                             #todate=datetime.datetime.now(),
                             fromdate=datetime.datetime(2018, 10, 1, 9, 30, 0),
                             tz=pytz.timezone('US/Eastern'),
                             backfill_from=data0,
                             #sessionstart=datetime.time(9, 30, 0),
                             #sessionend=datetime.time(16, 0, 0),
                             )

cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=1)

Finally, I run cerebro as:

cerebro.broker = store.getbroker()

cerebro.addstrategy(TestStrategy)

cerebro.run(exactbars=1, runonce=False)

I have tried several combinations of parameters both, in store.getdata() and in btfeed.GenericCSVData() but none of them work. So, any help will be appreciated.

Thanks in advance.

@backtrader Thank you very much for your useful answer. Greetings!

It seems to be a bug in pyfolio.py:

        pss = self.rets['positions']
        ps = [[k] + v[-2:] for k, v in iteritems(pss)]
        cols = ps.pop(0)  # headers are in the first entry
        positions = DF.from_records(ps, index=cols[0], columns=cols)
        positions.index = pandas.to_datetime(positions.index)
        positions.index = positions.index.tz_localize('UTC')

Line 130 is ps = [[k] + v[-2:] for k, v in iteritems(pss)] but it should be ps = [[k] + v[:] for k, v in iteritems(pss)]. With that change now I get the right positions in my strategy:

If the bug could be confirmed, it would be great to get fixed in a future release.

Happy trading!

Hi, Backtraders!

I facing an issue retrieving the positions from a backtesting in Backtrader. I'm using Pyfolio analyzer to try to achieve this but it doesn't work properly.

This is what I got in positions for my strategy:

I tried the following function of Pyfolio but it doesn't work either:

The code I am using in backtrader to run the strategy is this one:

    cerebro = bt.Cerebro()

    # Add pyfolio analyzer
    cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')

    # Data feed --------------------------------
    for equity in equity_list:
        data = get_data(
            hp.cd_dospuntos(os.getcwd(), 2) + '/input_gral/backtest_data/' + equity + '.csv',
            from_date,
            to_date
        )
        cerebro.adddata(data, name=equity)

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Comissions
    cerebro.broker.setcommission(commission=0.001)

    # Configuramos el slipage a 0.5%
    cerebro.broker.set_slippage_perc(perc=0.005,
                                     slip_open=True, slip_limit=True,
                                     slip_match=True, slip_out=False)

    # When working with large percentages it would be good to disable checksubmit in the broker. The rationale being
    # that if the current total portfolio is large, the broker might reject some orders if no cash is left.
    # You could order the orders in such a way that this doesn't happen, but disabling checksubmitwould be easier.
    cerebro.broker.set_checksubmit(checksubmit=False)

    # Strategy
    cerebro.addstrategy(
        TestStrategy,
        buy_and_hold=buy_and_hold,
        sellAll_date=sellAll_date,
    )

    start = time.time()
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Execute
    results = cerebro.run(runonce=False)

    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
    end = time.time()

    # Obtiene los resultados para el posterior análisis con pyfolio
    strat = results[0]
    pyfoliozer = strat.analyzers.getbyname('pyfolio')
    returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items()

    # Grabamos los resultados a un archivo pickle
    filepath = 'out/BTesting_results.pkl'
    if buy_and_hold:
        filepath = 'out/BTesting_results_buy_and_hold.pkl'
    file = open(filepath, 'wb')
    pikle_dict = {
        'returns': returns,
        'positions': positions,
        'transactions': transactions,
        'gross_lev': gross_lev,
    }
    pickle.dump(pikle_dict, file)

Any help is welcome. Thanks in advance.

@backtrader Thank you very much for your response and suggestions. I appreciate it.

Thank you for your prompt response. Any clue how to overcome the issue? Or is there a limited number of assets we can work with Interactive Brokers and Backtrader?

Hi, Backtraders!

I'm trying to run backtrader with Interactive Broker where I fed data for 10 equities. But I get an error related to the length of some of the data lines at the beginning of the simulation. When I reduce the number of equities to three all run ok. I searched on this community and found a related problem but any
solution is yet proposed.

Here the full exception trace:

Traceback (most recent call last):
  File "/home/galvez/repos/research_factory/20180822_PortafolioBacktrader/XBranch_NN/backTesting/run_backtest.py", line 335, in <module>
    run_live(equity)
  File "/home/galvez/repos/research_factory/20180822_PortafolioBacktrader/XBranch_NN/backTesting/run_backtest.py", line 325, in run_live
    cerebro.run()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/cerebro.py", line 1630, in _runnext
    strat._next()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 325, in _next
    super(Strategy, self)._next()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineiterator.py", line 255, in _next
    clock_len = self._clk_update()
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 305, in _clk_update
    newdlens = [len(d) for d in self.datas]
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/strategy.py", line 305, in <listcomp>
    newdlens = [len(d) for d in self.datas]
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineseries.py", line 464, in __len__
    return len(self.lines)
  File "/home/galvez/anaconda3/envs/backtrader/lib/python3.5/site-packages/backtrader/lineseries.py", line 220, in __len__
    return len(self.lines[0])
ValueError: __len__() should return >= 0

Process finished with exit code 1

Thanks in advance and happy trading.

But in the documentation doesn't say that and in practice in backtrader doesn' do that either. self.broker.get_value(data) it shouldn't take into account the cash, because it is the portfolio value, not the account value. Otherwise, it would not work in backtesting with the default broker.

I understand. Thank you.

The get_value() method it supposes to return the portfolio value, not the value of the entire account. But that method doesn't work properly when is connected with IB. I came up with a solution when I'm working with a single instrument. I override the order_target_value in my strategy as follow:

    def order_target_value(self, data=None, target=0.0, price=None, **kwargs):

        # if isinstance(data, string_types):
        #     data = self.getdatabyname(data)
        # elif data is None:
        #    data = self.data

        data = self.data  # My moddification

        possize = self.getposition(data, self.broker).size

        if not target and possize:  # closing a position
            return self.close(data=data, size=possize, price=price, **kwargs)

        else:
            # value = self.broker.getvalue(datas=[data])
            value = possize * self.datas[0].close  # My modification
            comminfo = self.broker.getcommissioninfo(data)

            # Make sure a price is there
            price = price if price is not None else data.close[0]

            if target > value:
                size = comminfo.getsize(price, target - value)
                return self.buy(data=data, size=size, price=price, **kwargs)

            elif target < value:
                size = comminfo.getsize(price, value - target)
                return self.sell(data=data, size=size, price=price, **kwargs)

        return None  # no execution size == possize

I tested it without IB and got the right result. So, I could expect the same with IB. What do you think about this solution? Is it correct?