Transactions happen for times that don't exist in source data

ruiarruda

I'm trying to use the Pandas Data Feed to load data and run a strategy. I load it like this:

import yfinance
a = yfinance.download(tickers="VGAC",period="max",interval="60m", groupby='ticker', prepost=True)
data = bt.feeds.PandasData(dataname=a, name='VGAC')

I add a transactions analyzer and run the strategy like this:

cerebro.addanalyzer(btanalyzers.Transactions, _name = "trans")
back = cerebro.run()

And I then look at the transactions like this:

back[0].analyzers.trans.get_analysis()

To my surprise there are many transactions happening with a datetime that isn't in the source data. For instance, a transaction at (datetime.datetime(2020, 11, 25, 19, 30), when that value isn't in the source data (the "a" dataframe) datetime column.

Does backtrader automatically fill gaps in candles? Or what is happening here? Couldn't find references to this in the documentation. After simplifying my tests by orders of magnitude in an attempt to obtain a result I actually understand and expect, I still can't. Surely it is because I lack in intellectual capacity, as apparently after several years of coding without major hassles, I discover a library that somehow is super easy to use for everyone else but makes me the most lost I've ever been when trying to tackle any problem, ever.

run-out

@ruiarruda Could you include the entire code you are running and we'll sort it out with you.

ruiarruda

@run-out thank you. I couldn't reproduce the problem again (that's how lost I am), but in my attempts I realized some things:

An indicator (RSI) I still had from the template code I had copy-pasted is influencing the strategy even though I don't use it anywhere after declaring it, and in fact I can't even run the code if I comment it out. I get the message "ValueError: min() arg is an empty sequence"
I'm trying to trade when the price hits a 3-hour low, placing a bracket order with that price as the buy price, and a take profit 0.8% higher and a stop loss 5% lower. That buy signal of mine apparently only starts being searched for after the period of the RSI is over... Couldn't be more lost
The order notification (and the transaction analyzer, sometimes) shows <backtrader.linebuffer.LineBuffer object at ...> instead of the price for the buy order only, showing a decimal number, as expected, for the other ones
Also, that price of the buy order, that I can't see as I explained above, is apparently set as the low price of the NEXT candle after the buy signal is found (I expected it to be set to the current candle's low and only buy if the price is met on the next candle)

I'm rather stunned at how little sense I can make of anything that is happening. Here is the full code:

import backtrader as bt
from datetime import datetime
import backtrader.analyzers as btanalyzers
import yfinance

class firstStrategy(bt.Strategy):

    def __init__(self):
        self.min_profit_sell = 0.008
        self.stop_loss = .05
        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=14)
        self.orefs = []

    def notify_order(self, order):
        print('{}: Order ref: {} / Type {} / Status {} / Size {} / Price {}'.format(
            self.data.datetime.datetime(0),
            order.ref, 'Buy' * order.isbuy() or 'Sell',
            order.getstatusname(),
            round(order.size, 2),
            'NA' if not order.price else order.price))

        if order.status == order.Completed:
            self.holdstart = len(self)

        if not order.alive() and order.ref in self.orefs:
            self.orefs.remove(order.ref)

            
    def next(self):
        self.interval_low = min(self.data.low.get(size=3))
        print (self.data.low.get(size=1))
        if not self.position:
            if self.data.low == self.interval_low:
                limitprice = self.data.low * (1 + self.min_profit_sell)
                print (f'Limit buy price: {self.data.low.get(size=1)[0]}')
                print (f'Take profit price: {limitprice}')
                stopprice = self.data.low * (1 - self.stop_loss)
                print (f'Stop loss price: {stopprice}')
                os = self.buy_bracket(size=100, limitprice=limitprice, price=self.data.low, stopprice=stopprice)
                self.orefs = [o.ref for o in os]

#Variable for our starting cash
startcash = 10000

#Create an instance of cerebro
cerebro = bt.Cerebro()

#Add our strategy
cerebro.addstrategy(firstStrategy)

a = yfinance.download(tickers="VGAC",period="max",interval="60m", groupby='ticker', prepost=True)
data = bt.feeds.PandasData(dataname=a, name='VGAC')

#Add the data to Cerebro
cerebro.adddata(data)

# Set our desired cash start
cerebro.broker.setcash(startcash)

# save a CSV with results
cerebro.addwriter(bt.WriterFile, csv=True, out='backtest.csv')

cerebro.addanalyzer(btanalyzers.Transactions, _name = "trans")
cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name = "trades")

# Run over everything
back = cerebro.run()

#Get final portfolio Value
portvalue = cerebro.broker.getvalue()
pnl = portvalue - startcash

Thanks a lot to you or anyone who can point me in the right direction to not give up on this library before even really starting

run-out

@ruiarruda Two major issues and I modified some minor code.

Your code was inputting the entire line into the bracket order for price, you need to use square brackets.

price=self.data.low[0],

The next issue you had was you were using a very very tight limit price, which basically means you are going to sell almost immediately due to volatility. I adjusted it to:

self.min_profit_sell = 0.05

Here is the total code.

import backtrader as bt
from datetime import datetime
import backtrader.analyzers as btanalyzers
import yfinance


class firstStrategy(bt.Strategy):
    def __init__(self):
        self.min_profit_sell = 0.05
        self.stop_loss = 0.05
        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=14)
        self.orefs = []

    def log(self, txt, dt=None):
        """ Logging function fot this strategy"""
        dt = dt or self.data.datetime[0]
        if isinstance(dt, float):
            dt = bt.num2date(dt)
        print("%s, %s" % (dt.date(), txt))

    def notify_order(self, order):
        print(
            "{}: Order ref: {} / Type {} / Status {} / Size {} / Price {:5.4f}".format(
                self.data.datetime.datetime(0),
                order.ref,
                "Buy" * order.isbuy() or "Sell",
                order.getstatusname(),
                round(order.size, 2),
                order.created.price,
            )
        )

        if order.status == order.Completed:
            self.holdstart = len(self)

        if not order.alive() and order.ref in self.orefs:
            self.orefs.remove(order.ref)

    def next(self):
        self.interval_low = min(self.data.low.get(size=3))
        self.log(
            f"Open: {self.data.open[0]:5.2f}, High: {self.data.high[0]:5.2f}, "
            f"Low: {self.data.low[0]:5.2f}, Close: {self.data.close[0]:5.2f} "
        )
        if not self.position:
            if self.data.low == self.interval_low:
                limitprice = self.data.low * (1 + self.min_profit_sell)
                # print(f'Limit buy price: {self.data.low.get(size=1)[0]}')
                stopprice = self.data.low * (1 - self.stop_loss)
                self.log(
                    f"Take profit price: {limitprice:5.2f} "
                    f"Stop loss price: {stopprice:5.2f}"
                )

                os = self.buy_bracket(
                    size=100,
                    limitprice=limitprice,
                    price=self.data.low[0],
                    stopprice=stopprice,
                )
                self.orefs = [o.ref for o in os]


# Variable for our starting cash
startcash = 10000

# Create an instance of cerebro
cerebro = bt.Cerebro()

# Add our strategy
cerebro.addstrategy(firstStrategy)

a = yfinance.download(
    tickers="VGAC", period="max", interval="60m", groupby="ticker", prepost=True
)
data = bt.feeds.PandasData(dataname=a, name="VGAC")

# Add the data to Cerebro
cerebro.adddata(data)

# Set our desired cash start
cerebro.broker.setcash(startcash)

# save a CSV with results
cerebro.addwriter(bt.WriterFile, csv=True, out="backtest.csv")

cerebro.addanalyzer(btanalyzers.Transactions, _name="trans")
cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name="trades")

# Run over everything
back = cerebro.run()

# Get final portfolio Value
portvalue = cerebro.broker.getvalue()
pnl = portvalue - startcash

Here is the first printouts...

2020-11-24, Open:  9.86, High:  9.95, Low:  9.84, Close:  9.91 
2020-11-24, Take profit price: 10.33 Stop loss price:  9.35
2020-11-24 20:30:00: Order ref: 1 / Type Buy / Status Submitted / Size 100 / Price 9.8400
2020-11-24 20:30:00: Order ref: 2 / Type Sell / Status Submitted / Size -100 / Price 9.3480
2020-11-24 20:30:00: Order ref: 3 / Type Sell / Status Submitted / Size -100 / Price 10.3320
2020-11-24 20:30:00: Order ref: 1 / Type Buy / Status Accepted / Size 100 / Price 9.8400
2020-11-24 20:30:00: Order ref: 2 / Type Sell / Status Accepted / Size -100 / Price 9.3480
2020-11-24 20:30:00: Order ref: 3 / Type Sell / Status Accepted / Size -100 / Price 10.3320
2020-11-24, Open:  9.86, High:  9.86, Low:  9.86, Close:  9.86 
2020-11-24, Open:  9.89, High:  9.89, Low:  9.89, Close:  9.89 
2020-11-25, Open:  9.95, High:  9.95, Low:  9.95, Close:  9.95 
2020-11-25, Open:  9.90, High:  9.90, Low:  9.85, Close:  9.89 
2020-11-25, Take profit price: 10.34 Stop loss price:  9.36
2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Submitted / Size 100 / Price 9.8500
2020-11-25 14:30:00: Order ref: 5 / Type Sell / Status Submitted / Size -100 / Price 9.3575
2020-11-25 14:30:00: Order ref: 6 / Type Sell / Status Submitted / Size -100 / Price 10.3425
2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Accepted / Size 100 / Price 9.8500
2020-11-25 14:30:00: Order ref: 5 / Type Sell / Status Accepted / Size -100 / Price 9.3575
2020-11-25 14:30:00: Order ref: 6 / Type Sell / Status Accepted / Size -100 / Price 10.3425
2020-11-25 14:30:00: Order ref: 1 / Type Buy / Status Completed / Size 100 / Price 9.8400
2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Completed / Size 100 / Price 9.8500
2020-11-25, Open:  9.84, High:  9.94, Low:  9.81, Close:  9.85 
2020-11-25, Open:  9.81, High:  9.99, Low:  9.81, Close:  9.90 
2020-11-25 16:30:00: Order ref: 3 / Type Sell / Status Completed / Size -100 / Price 10.3320
2020-11-25 16:30:00: Order ref: 2 / Type Sell / Status Canceled / Size -100 / Price 9.3480
2020-11-25 16:30:00: Order ref: 6 / Type Sell / Status Completed / Size -100 / Price 10.3425
2020-11-25 16:30:00: Order ref: 5 / Type Sell / Status Canceled / Size -100 / Price 9.3575

ruiarruda

@run-out Thanks a lot for your code. Was great to learn that I was setting the buy price incorrectly.

However, the most confusing issue still stands: the RSI indicator, which I initialize but do NOT use (intentionally, at least), is somehow necessary for the code to run (if you comment it out on your code, the code breaks). Also, the script waits for the RSI period (14) to be over before starting to search for the buy signal. I don't understand it...

Regarding the take profit limit price of .8%... I don't get why it would be a problem. In fact it leads to a higher p&l than 5% on this very code. Also, I ran a live bot for some time with that take profit and it was profitable, and I selected it after running my own simple backtest on different options for the price. Can I not trust Backtrader backtesting results if I use .8% as the limit price?

davidavr

@ruiarruda said in Transactions happen for times that don't exist in source data:

However, the most confusing issue still stands: the RSI indicator, which I initialize but do NOT use (intentionally, at least), is somehow necessary for the code to run (if you comment it out on your code, the code breaks).

Here is the issue

self.interval_low = min(self.data.low.get(size=3))

In next you are trying to access the last 3 low values but in the first call to next, there is only one value in self.data.low at this point. When you leave in the line for the RSI indicator, BT buffers the first 14 data points before calling next so the RSI indicator will be available (whether you access it or not).

You could change the offending line to:

self.interval_low = min(self.data.low.get(size=min(3, len(self.data.low))))

or just create some indicator with a minimum period of 3 to work around this.

run-out

@davidavr said in Transactions happen for times that don't exist in source data:

))

@davidavr Nice find.

@ruiarruda Have a look here to add minimum periods.

ruiarruda

@davidavr thanks a lot, the problem wasn't so complicated afterall

@run-out thank you for the pointer, I'll consult it