@backtrader. Yes thank you, dunno how I missed that!
@søren-pallesen I did have some luck doing something in a sort of hacky way, but works exactly for what I need.

class MyStrategy(bt.Strategy):
    def __init__(self):
        self.__special_array_for_later = [[None, 0, 0, 0,...]] #note nested brackets

    def next(self): 
        if some_condition:
            do_some_special_calculations()

    def do_some_special_calculations(self)
        self.__special_array_for_later.append([current_dt, calc1, calc2, calc3, ...])

Then when it's all over I do this little move:

special_array = thestrat._MyStrategy__special_array_for_later

it turns out that thestrat._MyStrategy* has a bunch of tricks for debugging and other internals.

From there I make my dataframe and do all the things I need.
It seems to be working perfectly for the backtest environment. Not sure what's gonna happen when this all switches over to live but working good so far

Another thing to note. I do my calculate AND append work in a separate function that gets called when some_condition is true. I do that because I don't want to weigh down my next(self) function with unnecessary code. Then when the run is over, I line up the special_array with my other rawdata and trades arrays using dt selections in pandas - during the data pre-processing and analysis steps. I don't know the exact overhead I'm saving by doing this. If I were to use indicators, they would calculate every cycle, but would be easier to line up after. I opted to save the overhead, and do the alignment later. But I imagine both would work.

I've read the page though it's not clear to me to get access to the data. The documentation says do it like so.

thestrats = cerebro.run(tradehistory=True)
thestrat = thestrats[0]

And then I've figured out how to get price data, trades, and orders

# raw data
open = thestrat.data_open.array
high = thestrat.data_high.array
...

# trades
trades = [str(trade).splitlines() for trade in list(thestrat._trades.values())[0][0]]

# orders
orders = [str(order).splitlines() for order in thestrat._orders]

But where I'm stuck now is how can I do this same thing with indicators?

Yass. Thank you @Developing-Coder this solves my issue. To get it working properly I had to add transmit=False to market order, after that it worked as desired. Thank you!

The limit order fill behavior is different than the market order fill behavior. With a market order, the fill won't happen until the next printed bar. But for limit orders, the fill happens in the current bar.

I don't understand why these are different. And is there a way I can get the limit orders to behave like the market orders, in terms of waiting until next printed bar to get filled?

Here is a 1Mb data sample NQ data I use in this example.

from datetime import timedelta, datetime as Dt
from types import SimpleNamespace as Sns
import backtrader as bt
import pandas as pd
import os

inst = Sns (symbol     = 'NQ'
            ,stratName = 'lmt_test'
            ,startDate = Dt(2019, 4, 16 )
            ,endDate   = Dt(2019, 4, 17 )
            ,showlogs  = True
            ,datafile  = os.path.expanduser('~/')+'dev/datas/'+'NQ_test.txt'
            )

# Build dataframes
rawdatacolnames = ['ddate','ttime','open','high','low','close','volume']
df1 = pd.read_csv(inst.datafile, names=rawdatacolnames)
df1['dtime'] = df1.ddate+' '+df1.ttime
df1.dtime = pd.to_datetime(df1.dtime, format = '%m/%d/%Y %H:%M')
df1 = df1[(df1.dtime >= inst.startDate) & (df1.dtime <= inst.endDate)]
df1.index = df1.dtime
df1 = df1[['open','high','low','close']]

ohlc_dict = {'open':'first', 'high':'max', 'low':'min', 'close': 'last'}
df15 = df1.resample('15Min').agg(ohlc_dict)

# ------------------------- Lmt Test Strategy ---------------------------
class lmt_test(bt.Strategy): 
    params = dict(inst = dict())
    # ----------------------------------------------------------------------------------#
    def log(self, txt, f_name='', dt=None): 
        dt = dt or self.data.datetime.datetime()
        print(f'{dt.strftime("%y/%m/%d-%H:%M:%S")}:: {f_name:<12}:: {txt}')
    # ----------------------------------------------------------------------------------#
    def barlog(self, d,                     f_name='BARLOG'): 
        self.log(f'lend0:{len(d)}, lend1:{len(self.data1)} - '+
                f'o:{d.open[0]:.2f} h:{d. high[0]:.2f} '+
                f'l:{d. low[0]:.2f} c:{d.close[0]:.2f}', f_name)
    # ----------------------------------------------------------------------------------#
    def notify_order(self, order,           f_name='NOTIFY ORDER'): 
        self.log(f'Order {order.info.orderTag}-{order.ref} '+
                 f'@{order.plen}: {order.getstatusname()}', f_name)
        if order.status in [order.Completed]:
            orderprintstr = f'{["SELL","BUY "][order.isbuy()]} EXECUTED: {order.info.orderTag}: '+\
                            f'${order.executed.price:.2f}'
            self.log(orderprintstr,f_name,)
            self.log('New position: '+str(self.getposition(self.data1).size),f_name)
    # ----------------------------------------------------------------------------------#
    def notify_trade(self, trade,           f_name='NOTIFY TRADE'): 
        if trade.isclosed:
            self.log(f'PNL: {trade.pnl:.2f}',f_name)
    # ----------------------------------------------------------------------------------#
    def __init__(self):
        if not self.p.inst.showlogs:
            self.log    = lambda x: None
            self.barlog = lambda x: None
    # ----------------------------------------------------------------------------------#
    def next(self,                          f_name='NEXT_BB'): 
        self.barlog(self.data0)

        if (len(self.data0)-5)%45 == 0:
            self.placeOrder(len(self.data1)%2, 2)
    # ----------------------------------------------------------------------------------#
    def placeOrder(self, odr, osz,          f_name='PLACE ORDER'): 
        self.log(f'{["Buy","Sell"][odr]} {osz} at {self.data0.close[0]}, '+
                  'Current position = '+str(self.getposition(self.data1).size),f_name)

        # Cancel open orders
        openOrders = self.broker.get_orders_open()
        if openOrders: self.cancel(openOrders[0])

        # Create new orders
        lmtOrderArgs = dict(data=self.data1, size=1, exectype=bt.Order.Limit, orderTag='Lmt')
        if odr: 
            self.sell(data=self.data1, orderTag='Mkt')
            self.sell(**lmtOrderArgs, price = self.data0.close[0])
        else:
            self.buy (data=self.data1, orderTag='Mkt')
            self.buy (**lmtOrderArgs, price = self.data0.close[0])

# ------------------------- Run Cerebro Strategy ---------------------------
print(Dt.now().strftime('%m-%d %H:%M:%S::'),
        'Running:',inst.symbol,'from:',str(inst.startDate)[:-9],'to:',str(inst.endDate)[:-9])

cerebro = bt.Cerebro(runonce=False, stdstats=False)

data0 = bt.feeds.PandasData(dataname = df1)
cerebro.adddata(data0, 'm1')

data1 = bt.feeds.PandasData(dataname = df15)
cerebro.adddata(data1, 'm15')

cerebro.addstrategy(eval(inst.stratName),inst=inst)

thestrat = cerebro.run(tradehistory=False)[0]

print('Final Portfolio Value: {:.2f}' .format(cerebro.broker.getvalue()))
print(Dt.now().strftime('%m-%d %H:%M:%S::'),f'Symbol {inst.symbol} Backtest Complete.')

and here is an excerpt from the logs:

19/04/16-00:43:00:: BARLOG      :: lend0:44, lend1:3 - o:7662.75 h:7662.75 l:7662.50 c:7662.75
19/04/16-00:44:00:: BARLOG      :: lend0:45, lend1:3 - o:7662.75 h:7662.75 l:7662.50 c:7662.50
19/04/16-00:45:00:: BARLOG      :: lend0:46, lend1:4 - o:7662.50 h:7663.00 l:7662.50 c:7662.50
19/04/16-00:46:00:: BARLOG      :: lend0:47, lend1:4 - o:7662.50 h:7662.50 l:7662.00 c:7662.00
19/04/16-00:47:00:: BARLOG      :: lend0:48, lend1:4 - o:7662.00 h:7662.00 l:7661.75 c:7661.75
19/04/16-00:48:00:: BARLOG      :: lend0:49, lend1:4 - o:7661.75 h:7662.00 l:7661.50 c:7661.75
19/04/16-00:49:00:: BARLOG      :: lend0:50, lend1:4 - o:7661.75 h:7662.50 l:7661.75 c:7662.25
19/04/16-00:49:00:: PLACE ORDER :: Buy 2 at 7662.25, Current position = -2
19/04/16-00:50:00:: NOTIFY ORDER:: Order Mkt-3 @4: Submitted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Submitted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Mkt-3 @4: Accepted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Accepted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Completed
19/04/16-00:50:00:: NOTIFY ORDER:: BUY  EXECUTED: Lmt: $7662.25
19/04/16-00:50:00:: NOTIFY ORDER:: New position: -1
19/04/16-00:50:00:: BARLOG      :: lend0:51, lend1:4 - o:7662.25 h:7662.25 l:7662.00 c:7662.00
19/04/16-00:51:00:: BARLOG      :: lend0:52, lend1:4 - o:7662.00 h:7662.50 l:7662.00 c:7662.50
19/04/16-00:52:00:: BARLOG      :: lend0:53, lend1:4 - o:7662.50 h:7663.00 l:7662.50 c:7662.75
19/04/16-00:53:00:: BARLOG      :: lend0:54, lend1:4 - o:7662.75 h:7663.50 l:7662.75 c:7663.50
19/04/16-00:54:00:: BARLOG      :: lend0:55, lend1:4 - o:7663.50 h:7663.75 l:7663.00 c:7663.00
19/04/16-00:55:00:: BARLOG      :: lend0:56, lend1:4 - o:7663.00 h:7663.00 l:7662.50 c:7662.75
19/04/16-00:56:00:: BARLOG      :: lend0:57, lend1:4 - o:7663.00 h:7663.00 l:7663.00 c:7663.00
19/04/16-00:57:00:: BARLOG      :: lend0:58, lend1:4 - o:7663.00 h:7663.50 l:7663.00 c:7663.00
19/04/16-00:58:00:: BARLOG      :: lend0:59, lend1:4 - o:7663.00 h:7663.25 l:7663.00 c:7663.25
19/04/16-00:59:00:: BARLOG      :: lend0:60, lend1:4 - o:7663.25 h:7663.50 l:7663.00 c:7663.25
19/04/16-01:00:00:: NOTIFY ORDER:: Order Mkt-3 @4: Completed
19/04/16-01:00:00:: NOTIFY ORDER:: BUY  EXECUTED: Mkt: $7663.25
19/04/16-01:00:00:: NOTIFY ORDER:: New position: 0
19/04/16-01:00:00:: NOTIFY TRADE:: PNL: -1.50
19/04/16-01:00:00:: BARLOG      :: lend0:61, lend1:5 - o:7663.25 h:7663.50 l:7662.00 c:7662.25
19/04/16-01:01:00:: BARLOG      :: lend0:62, lend1:5 - o:7662.25 h:7662.75 l:7662.25 c:7662.25
19/04/16-01:02:00:: BARLOG      :: lend0:63, lend1:5 - o:7662.25 h:7662.25 l:7661.75 c:7661.75
19/04/16-01:03:00:: BARLOG      :: lend0:64, lend1:5 - o:7661.75 h:7661.75 l:7661.25 c:7661.50
19/04/16-01:04:00:: BARLOG      :: lend0:65, lend1:5 - o:7661.50 h:7662.25 l:7661.50 c:7662.00

As you can see from the log the order placement happens at 00:49(HH:MM) or len(self.data0)=50. Then, at 00:50 (len(self.data0)=51) both market order and limit order are submitted and accepted. But here's what I can't figure out:

Note that for both the limit and the market order I have the parameter data = self.data1

The market order (which is first in the queue) waits until the new 15m bar-print to execute. This occurs just before 01:00 (len(self.data0)=61 & len(self.data1) increases from 4 to 5). That is the behavior I expect. But the limit order executes, just before the call of next at 00:50 (len(self.data0)=51). It's not waiting for a new bar to print, it's executing inside the current self.data1 bar.

Why is this order execution behavior different in this way? Is there a way to make the limit order behave as the market order does?

@backtrader any insight on this?

This would also apply if there were two different products.

For example if I had 1-min NQ data and 5-sec ES data, and I put on a market and limit order at 9:00a in NQ. The limit order would fill based on the 9:00a bar (when 9:00:05 next() is called), but the market order would fill based on the 9:01 bar at 9:01:00 when the next new bar prints.

The limit order fill behavior is different than the market order fill behavior. With a market order, the fill won't happen until the next printed bar. But for limit orders, the fill happens in the current bar.

I don't understand why these are different. And is there a way I can get the limit orders to behave like the market orders, in terms of waiting until next printed bar to get filled?

Here is a 1Mb data sample NQ data I use in this example.

from datetime import timedelta, datetime as Dt
from types import SimpleNamespace as Sns
import backtrader as bt
import pandas as pd
import os

inst = Sns (symbol     = 'NQ'
            ,stratName = 'lmt_test'
            ,startDate = Dt(2019, 4, 16 )
            ,endDate   = Dt(2019, 4, 17 )
            ,showlogs  = True
            ,datafile  = os.path.expanduser('~/')+'dev/datas/'+'NQ_test.txt'
            )

# Build dataframes
rawdatacolnames = ['ddate','ttime','open','high','low','close','volume']
df1 = pd.read_csv(inst.datafile, names=rawdatacolnames)
df1['dtime'] = df1.ddate+' '+df1.ttime
df1.dtime = pd.to_datetime(df1.dtime, format = '%m/%d/%Y %H:%M')
df1 = df1[(df1.dtime >= inst.startDate) & (df1.dtime <= inst.endDate)]
df1.index = df1.dtime
df1 = df1[['open','high','low','close']]

ohlc_dict = {'open':'first', 'high':'max', 'low':'min', 'close': 'last'}
df15 = df1.resample('15Min').agg(ohlc_dict)

# ------------------------- Lmt Test Strategy ---------------------------
class lmt_test(bt.Strategy): 
    params = dict(inst = dict())
    # ----------------------------------------------------------------------------------#
    def log(self, txt, f_name='', dt=None): 
        dt = dt or self.data.datetime.datetime()
        print(f'{dt.strftime("%y/%m/%d-%H:%M:%S")}:: {f_name:<12}:: {txt}')
    # ----------------------------------------------------------------------------------#
    def barlog(self, d,                     f_name='BARLOG'): 
        self.log(f'lend0:{len(d)}, lend1:{len(self.data1)} - '+
                f'o:{d.open[0]:.2f} h:{d. high[0]:.2f} '+
                f'l:{d. low[0]:.2f} c:{d.close[0]:.2f}', f_name)
    # ----------------------------------------------------------------------------------#
    def notify_order(self, order,           f_name='NOTIFY ORDER'): 
        self.log(f'Order {order.info.orderTag}-{order.ref} '+
                 f'@{order.plen}: {order.getstatusname()}', f_name)
        if order.status in [order.Completed]:
            orderprintstr = f'{["SELL","BUY "][order.isbuy()]} EXECUTED: {order.info.orderTag}: '+\
                            f'${order.executed.price:.2f}'
            self.log(orderprintstr,f_name,)
            self.log('New position: '+str(self.getposition(self.data1).size),f_name)
    # ----------------------------------------------------------------------------------#
    def notify_trade(self, trade,           f_name='NOTIFY TRADE'): 
        if trade.isclosed:
            self.log(f'PNL: {trade.pnl:.2f}',f_name)
    # ----------------------------------------------------------------------------------#
    def __init__(self):
        if not self.p.inst.showlogs:
            self.log    = lambda x: None
            self.barlog = lambda x: None
    # ----------------------------------------------------------------------------------#
    def next(self,                          f_name='NEXT_BB'): 
        self.barlog(self.data0)

        if (len(self.data0)-5)%45 == 0:
            self.placeOrder(len(self.data1)%2, 2)
    # ----------------------------------------------------------------------------------#
    def placeOrder(self, odr, osz,          f_name='PLACE ORDER'): 
        self.log(f'{["Buy","Sell"][odr]} {osz} at {self.data0.close[0]}, '+
                  'Current position = '+str(self.getposition(self.data1).size),f_name)

        # Cancel open orders
        openOrders = self.broker.get_orders_open()
        if openOrders: self.cancel(openOrders[0])

        # Create new orders
        lmtOrderArgs = dict(data=self.data1, size=1, exectype=bt.Order.Limit, orderTag='Lmt')
        if odr: 
            self.sell(data=self.data1, orderTag='Mkt')
            self.sell(**lmtOrderArgs, price = self.data0.close[0])
        else:
            self.buy (data=self.data1, orderTag='Mkt')
            self.buy (**lmtOrderArgs, price = self.data0.close[0])

# ------------------------- Run Cerebro Strategy ---------------------------
print(Dt.now().strftime('%m-%d %H:%M:%S::'),
        'Running:',inst.symbol,'from:',str(inst.startDate)[:-9],'to:',str(inst.endDate)[:-9])

cerebro = bt.Cerebro(runonce=False, stdstats=False)

data0 = bt.feeds.PandasData(dataname = df1)
cerebro.adddata(data0, 'm1')

data1 = bt.feeds.PandasData(dataname = df15)
cerebro.adddata(data1, 'm15')

cerebro.addstrategy(eval(inst.stratName),inst=inst)

thestrat = cerebro.run(tradehistory=False)[0]

print('Final Portfolio Value: {:.2f}' .format(cerebro.broker.getvalue()))
print(Dt.now().strftime('%m-%d %H:%M:%S::'),f'Symbol {inst.symbol} Backtest Complete.')

and here is an excerpt from the logs:

19/04/16-00:43:00:: BARLOG      :: lend0:44, lend1:3 - o:7662.75 h:7662.75 l:7662.50 c:7662.75
19/04/16-00:44:00:: BARLOG      :: lend0:45, lend1:3 - o:7662.75 h:7662.75 l:7662.50 c:7662.50
19/04/16-00:45:00:: BARLOG      :: lend0:46, lend1:4 - o:7662.50 h:7663.00 l:7662.50 c:7662.50
19/04/16-00:46:00:: BARLOG      :: lend0:47, lend1:4 - o:7662.50 h:7662.50 l:7662.00 c:7662.00
19/04/16-00:47:00:: BARLOG      :: lend0:48, lend1:4 - o:7662.00 h:7662.00 l:7661.75 c:7661.75
19/04/16-00:48:00:: BARLOG      :: lend0:49, lend1:4 - o:7661.75 h:7662.00 l:7661.50 c:7661.75
19/04/16-00:49:00:: BARLOG      :: lend0:50, lend1:4 - o:7661.75 h:7662.50 l:7661.75 c:7662.25
19/04/16-00:49:00:: PLACE ORDER :: Buy 2 at 7662.25, Current position = -2
19/04/16-00:50:00:: NOTIFY ORDER:: Order Mkt-3 @4: Submitted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Submitted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Mkt-3 @4: Accepted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Accepted
19/04/16-00:50:00:: NOTIFY ORDER:: Order Lmt-4 @4: Completed
19/04/16-00:50:00:: NOTIFY ORDER:: BUY  EXECUTED: Lmt: $7662.25
19/04/16-00:50:00:: NOTIFY ORDER:: New position: -1
19/04/16-00:50:00:: BARLOG      :: lend0:51, lend1:4 - o:7662.25 h:7662.25 l:7662.00 c:7662.00
19/04/16-00:51:00:: BARLOG      :: lend0:52, lend1:4 - o:7662.00 h:7662.50 l:7662.00 c:7662.50
19/04/16-00:52:00:: BARLOG      :: lend0:53, lend1:4 - o:7662.50 h:7663.00 l:7662.50 c:7662.75
19/04/16-00:53:00:: BARLOG      :: lend0:54, lend1:4 - o:7662.75 h:7663.50 l:7662.75 c:7663.50
19/04/16-00:54:00:: BARLOG      :: lend0:55, lend1:4 - o:7663.50 h:7663.75 l:7663.00 c:7663.00
19/04/16-00:55:00:: BARLOG      :: lend0:56, lend1:4 - o:7663.00 h:7663.00 l:7662.50 c:7662.75
19/04/16-00:56:00:: BARLOG      :: lend0:57, lend1:4 - o:7663.00 h:7663.00 l:7663.00 c:7663.00
19/04/16-00:57:00:: BARLOG      :: lend0:58, lend1:4 - o:7663.00 h:7663.50 l:7663.00 c:7663.00
19/04/16-00:58:00:: BARLOG      :: lend0:59, lend1:4 - o:7663.00 h:7663.25 l:7663.00 c:7663.25
19/04/16-00:59:00:: BARLOG      :: lend0:60, lend1:4 - o:7663.25 h:7663.50 l:7663.00 c:7663.25
19/04/16-01:00:00:: NOTIFY ORDER:: Order Mkt-3 @4: Completed
19/04/16-01:00:00:: NOTIFY ORDER:: BUY  EXECUTED: Mkt: $7663.25
19/04/16-01:00:00:: NOTIFY ORDER:: New position: 0
19/04/16-01:00:00:: NOTIFY TRADE:: PNL: -1.50
19/04/16-01:00:00:: BARLOG      :: lend0:61, lend1:5 - o:7663.25 h:7663.50 l:7662.00 c:7662.25
19/04/16-01:01:00:: BARLOG      :: lend0:62, lend1:5 - o:7662.25 h:7662.75 l:7662.25 c:7662.25
19/04/16-01:02:00:: BARLOG      :: lend0:63, lend1:5 - o:7662.25 h:7662.25 l:7661.75 c:7661.75
19/04/16-01:03:00:: BARLOG      :: lend0:64, lend1:5 - o:7661.75 h:7661.75 l:7661.25 c:7661.50
19/04/16-01:04:00:: BARLOG      :: lend0:65, lend1:5 - o:7661.50 h:7662.25 l:7661.50 c:7662.00

As you can see from the log the order placement happens at 00:49(HH:MM) or len(self.data0)=50. Then, at 00:50 (len(self.data0)=51) both market order and limit order are submitted and accepted. But here's what I can't figure out:

Note that for both the limit and the market order I have the parameter data = self.data1

The market order (which is first in the queue) waits until the new 15m bar-print to execute. This occurs just before 01:00 (len(self.data0)=61 & len(self.data1) increases from 4 to 5). That is the behavior I expect. But the limit order executes, just before the call of next at 00:50 (len(self.data0)=51). It's not waiting for a new bar to print, it's executing inside the current self.data1 bar.

Why is this order execution behavior different in this way? Is there a way to make the limit order behave as the market order does?

@backtrader I have the same question as @Michael172.

For a long trade with a trailing stop, the stop price should be based off the high and not the close for the normal implementation.

For example if my trailing stop is set to trailamount=20, if the high bar has a close of 2800, and a high of 2810, the stop price should be 2790. That's how IB implements a trailing stop.

Why is the trailing stop based off close and is there a way to change it?

Yass. Thank you @Developing-Coder this solves my issue. To get it working properly I had to add transmit=False to market order, after that it worked as desired. Thank you!

I've narrowed down the problem: My input data are 1m candles. In addition, I have an upsampled "dummy" data-feed - it simply replay's the candle 3 times per minute - that I use for a special indicator. In the code below there is data_1m and data_20s. I want all the logic including the trailing stop to operate from the 1m data in the strat code is self.data1. However, the trailing stop seems to be getting filled based on self.data0 as you can see from the logs, even though I specified data=self.data1 when I created the order.

import backtrader as bt
import pandas as pd
from datetime import timedelta, datetime as Dt
from types import SimpleNamespace as Sns

inst = Sns( filename      = 'datas/NQ_test.txt'
           ,startDate     = Dt(2019, 5, 1)
           ,endDate       = Dt(2019, 5, 6)
           ,showlogs      = True
           ,trailstop_int = 10
           )

# ************ Test Strat ************
class test(bt.Strategy): 
    params = dict(showlogs = True, ts_int = 10)
    # ----------------------------------------------------------------------------------#
    def Log(self, txt, f_name = '', dt = None): 
        if self.p.showlogs:
            dt = dt or self.data.datetime.datetime()
            print(f'{dt.strftime("%y/%m/%d-%H:%M:%S")}:: {f_name:<12}:: {txt}')
    # ----------------------------------------------------------------------------------#
    def candle_log(self,                    f_name='CAND_LOG'): 
        self.Log(f'o:{self.data1.open[0]:.2f} h:{self.data1. high[0]:.2f} '+
                 f'l:{self.data1. low[0]:.2f} c:{self.data1.close[0]:.2f}', f_name)     
    # ----------------------------------------------------------------------------------#
    def __init__(self): 
        self.new_1m_print = self.data1.datetime() != self.data1.datetime()(-1)
    # ----------------------------------------------------------------------------------#
    def next(self,                          f_name='NEXT'): 
        # self.candle_log()
        if self.new_1m_print:
            if   (((self.data1.close[0] - self.data1.low[0]) > self.p.ts_int) and 
                            (self.getposition(self.data1).size <= 0)):
                self.positionB = self.getposition(self.data1).size
                odr=0 # Order Direction is BUY
                osz = abs(self.positionB) + 1
                self.placeOrder(odr, osz)
            
            elif (((self.data1.high[0] - self.data1.open[0]) > self.p.ts_int) and
                            (self.getposition(self.data1).size >= 0)):
                self.positionB = self.getposition(self.data1).size
                odr=1 # Order Direction is SELL
                osz = abs(self.positionB) + 1
                self.placeOrder(odr, osz)
    # ----------------------------------------------------------------------------------#
    def placeOrder(self, odr, osz,          f_name = 'PLACE ORDER'):
        self.Log(f'{["Buy","Sell"][odr]} {osz}',f_name)

        # Cancel open orders
        openOrders = self.broker.get_orders_open()
        if openOrders: 
            for o in openOrders:
                self.cancel(o)

        # Create market orders
        orderArgs = dict(data = self.data1, orderTag = 'Entry', size = osz)
        if odr: 
            self.sell(**orderArgs)
        else:   
            self.buy (**orderArgs)
            
        # Create trailing orders
        orderArgs = dict(data = self.data1, orderTag = 'TrailStop', 
                         trailamount = self.p.ts_int + 1, exectype = bt.Order.StopTrail)
        if odr: 
            self.buy (**orderArgs)
        else: 
            self.sell(**orderArgs)
    # ----------------------------------------------------------------------------------#
    def notify_order(self, order,           f_name='NOTIFY ORDER'): 
        # self.Log(f'Order {order.ref} Status: {order.getstatusname()}', f_name)
        if order.status in [order.Completed]:
            if order.isbuy():
                self.Log(f'BUY  EXECUTED {order.info.orderTag}, '+
                         f'Price: {order.executed.price:.2f}', f_name,)
            else:  # is Sell
                self.Log(f'SELL EXECUTED {order.info.orderTag}, '+
                         f'Price: {order.executed.price:.2f}', f_name,)
            self.Log('New position: '+str(self.getposition(self.data1).size),f_name)
    # ----------------------------------------------------------------------------------#
    def notify_trade(self, trade,           f_name='NOTIFY TRADE'): 
        if trade.isclosed:
            self.Log(f'PNL: {trade.pnl:.2f}', f_name)

# ************  RUN  ************
def upsample(): 
    rawdatacolnames = ['ddate','ttime','open','high','low','close','volume']
    dfraw = pd.read_csv(inst.filename, names=rawdatacolnames)
    dfraw['dtime'] = dfraw.ddate+' '+dfraw.ttime
    dfraw.dtime = pd.to_datetime(dfraw.dtime,format='%m/%d/%Y %H:%M')
    dfraw = dfraw[(dfraw.dtime >= inst.startDate) & (dfraw.dtime <= inst.endDate)]
    dfraw.set_index('dtime',drop=True,inplace=True)
    dfraw = dfraw[['open','high','low','close','volume']]
    df_upsamp = dfraw.resample('20s').ffill()
    return dfraw, df_upsamp

dfraw1m, df_upsamp = upsample()

cerebro = bt.Cerebro(runonce=False)

data_20s = bt.feeds.PandasData(dataname = df_upsamp, 
                               timeframe = bt.TimeFrame.Seconds, 
                               compression = 20)
cerebro.adddata(data_20s)

data_1m = bt.feeds.PandasData(dataname = dfraw1m, 
                              timeframe = bt.TimeFrame.Minutes)
cerebro.adddata(data_1m)

cerebro.addstrategy(test,showlogs = inst.showlogs,ts_int = inst.trailstop_int)

cerebro.run()

And here is a sample of the logs that shows the problem:

19/05/01-07:00:00:: PLACE ORDER :: Buy 1
19/05/01-07:01:00:: NOTIFY ORDER:: BUY  EXECUTED Entry, Price: 7835.75
19/05/01-07:01:00:: NOTIFY ORDER:: New position: 1
19/05/01-07:25:00:: NOTIFY ORDER:: SELL EXECUTED TrailStop, Price: 7842.00
19/05/01-07:25:00:: NOTIFY ORDER:: New position: 0
19/05/01-07:25:00:: NOTIFY TRADE:: PNL: 6.25
19/05/01-10:00:00:: PLACE ORDER :: Sell 1
19/05/01-10:01:00:: NOTIFY ORDER:: SELL EXECUTED Entry, Price: 7842.25
19/05/01-10:01:00:: NOTIFY ORDER:: New position: -1
19/05/01-10:56:00:: NOTIFY ORDER:: BUY  EXECUTED TrailStop, Price: 7844.00
19/05/01-10:56:00:: NOTIFY ORDER:: New position: 0
19/05/01-10:56:00:: NOTIFY TRADE:: PNL: -1.75
19/05/01-11:04:00:: PLACE ORDER :: Buy 1
19/05/01-11:04:20:: NOTIFY ORDER:: SELL EXECUTED TrailStop, Price: 7849.25
19/05/01-11:04:20:: NOTIFY ORDER:: New position: -1
19/05/01-11:05:00:: NOTIFY ORDER:: BUY  EXECUTED Entry, Price: 7860.75
19/05/01-11:05:00:: NOTIFY ORDER:: New position: 0
19/05/01-11:05:00:: NOTIFY TRADE:: PNL: -11.50
19/05/01-11:50:00:: PLACE ORDER :: Buy 1
19/05/01-11:50:20:: NOTIFY ORDER:: SELL EXECUTED TrailStop, Price: 7818.75
19/05/01-11:50:20:: NOTIFY ORDER:: New position: -1
19/05/01-11:51:00:: NOTIFY ORDER:: BUY  EXECUTED Entry, Price: 7830.00
19/05/01-11:51:00:: NOTIFY ORDER:: New position: 0
19/05/01-11:51:00:: NOTIFY TRADE:: PNL: -11.25
19/05/02-06:32:00:: PLACE ORDER :: Sell 1
19/05/02-06:33:00:: NOTIFY ORDER:: SELL EXECUTED Entry, Price: 7769.00
19/05/02-06:33:00:: NOTIFY ORDER:: New position: -1
19/05/02-06:35:00:: NOTIFY ORDER:: BUY  EXECUTED TrailStop, Price: 7769.50
19/05/02-06:35:00:: NOTIFY ORDER:: New position: 0
19/05/02-06:35:00:: NOTIFY TRADE:: PNL: -0.50

So as you can see e.g. in the 3rd trade and the 4th trade, the trailing stop is getting filled in the 20 second mark in-between minutes. I would like to figure out how to prevent that, so I'm only getting filled on the 1m data that is specified.

One interesting aspect is that it looks like the Market order respects data= parameter, but not the StopTrail.

For completeness, here is the 1Mb NQ data I use in this example.

When my strategy's trigger gets hit, I create a market order and a trailing stop.

For example at 10:00a I enter a long market order and a short trailing stop - let's say the close price at 10a is 2700 my trailstop amount is 15. If the 10a candle has a low of 2685 or lower, the trailing stop will trigger ON the 10a candle. Despite that the market order doesn't fill until 10:01a (since I'm using 1m data). One of the strange artifacts is that the position actually goes to -1, and then back to 0 when the market order fills.

Is this normal behavior? Shouldn't it be that the stop order doesn't get an opportunity to fill until 10:01a, just like the market order.

I don't have any cheat on open or cheat on close. Has anyone seen this before?

It seems like maybe the search feature has lost some functionality. It used to be very thorough, or so it seemed. Today I tried searching for "quicknotify" which is a parameter in Cerebro, but nothing came up in the search box. I use the search function a lot, so I wanted to point out this issue.

Here is an example of the time slice problem.

You can see that the Renko prints one brick at a time, and eventually catches up with close price.

@ab_trader
The implementation of bt.filters.Renko, only allows for a single brick to print in a given time-slice. If the candle in that time-slice is greater than 2 boxes, the Renko filter still only prints one brick. Then, at the next time slice the Renko filter will look at the new close price, and print a maximum of one brick. It will continue like that until the Renko price eventually "catches up' with the current close price. (this explanation assumes we're using hilo=False filter parameter).

But this is not a correct Renko implementation. Let's call this problem the "time slice problem" (There are a few other problems with the bt.filters.Renko implementation, which are not discussed here). A proper Renko has the ability to print multiple bricks in a single "time slice". However because of the internals of Backtrader, that is not possible. But, we can simulate the opportunity to print multiple bricks in a single time slice, by upsampling the main data set, and setting the Renko filter on the upsampled data. This effectively solves the "time slice problem".

It seems like both resampledata and replaydata are designed for manipulating raw data into a lower resolution/samplerate, but there is nothing built-in the system to replay data to a higher resolution and force cerebro to sample at the high rate. So for the sake of closing out this thread I'll post the code that I used to do this using pandas.

strat = STRAT_OBJECT
symbol    = 'symbol_string'
datafile  = './file_location_string'
startDate = datetime.datetime(2019,1,1)
endDate = datetime.datetime(2019,1,2)
boxsize   = 1

# Sample Size: 5s gives opportunity for 12 renko boxes to print in single minute (which is probably overkill)
sampSize = 5   
 
print(' '.join([Dt.now().strftime('%m-%d %H:%M::'),
                'Running:', symbol, 'at: 1m candles from:',
                str(startDate)[:-9], 'to:', str(endDate)[:-9]]))

if not 60 % sampSize == 0: raise ValueError('sampSize must be a factor of 60')

# my data provider has csv files with separate fields for date and time
columns        = ['ddate','ttime','open','high','low','close','volume']
dfraw          = pd.read_csv(datafile, names = columns)
dfraw['dtime'] = dfraw.ddate + ' ' + dfraw.ttime
dfraw.dtime    = pd.to_datetime(dfraw.dtime, format='%m/%d/%Y %H:%M')
dfraw          = dfraw[(dfraw.dtime >= startDate) & (dfraw.dtime < endDate)]

# this upsample method was chosen for optimized speed and efficiency
timeserieslist = []
for t in dfraw.dtime:
    daterange = pd.date_range(t , t + DT.timedelta(seconds = 60-sampSize), periods = 60//sampSize)
    timeserieslist.extend(daterange.to_list())
df_upsamp = pd.DataFrame(timeserieslist, columns = ['dtime'])
# print(df_upsamp.head())

# map and forward fill the 1m data to replay across the upsampled datetime field
dfraw_ri = dfraw.set_index('dtime') # ri = re-indexed (helps w Series.map efficiency)
df_upsamp['open']  = df_upsamp.dtime.map(dfraw_ri.open)
df_upsamp['high']  = df_upsamp.dtime.map(dfraw_ri.high)
df_upsamp['low']   = df_upsamp.dtime.map(dfraw_ri.low)
df_upsamp['close'] = df_upsamp.dtime.map(dfraw_ri.close)
df_upsamp.fillna(method='ffill', inplace=True)
# print(df_upsamp.head())

cerebro = bt.Cerebro()
cerebro.addstrategy(strat)

data_5s = bt.feeds.PandasData(
                        dataname = df_upsamp,
                        timeframe = bt.TimeFrame.Seconds,
                        compression = sampSize,
                        datetime = 'dtime',
                        )
data_5s.plotinfo.plot = False

data_1m = bt.feeds.PandasData(
                        dataname = dfraw,
                        timeframe = bt.TimeFrame.Minutes,
                        datetime = 'dtime',
                        )

renko_args = dict(
                hilo     = False,
                align    = 1,
                size     = boxsize)

cerebro.adddata(data_5s, '5s_dummy')
cerebro.adddata(data_1m, '1m_standard')
    
data_Rk = data_5s.clone()
data_Rk.addfilter(bt.filters.Renko, **renko_args)
cerebro.adddata(data_Rk, 'renko')

thestrat = cerebro.run(tradehistory=False, stdstats=False)[0]

print(Dt.now().strftime('%m-%d %H:%M::'),f'Symbol {symbol} Backtest Complete')
os.system('say "backtest completed"')

figs = cerebro.plot(style='candle', barup= 'green', volume = False, tickrotation = 15, 
                        fmt_x_ticks = '%m-%d %H:%M', fmt_x_data = '%H:%M:%S')