A question about next()



  • The basic data I'm working with:

    2017-10-12,12.5,12.75,12.4,12.45,579788.0,0.0,1.0,12.5,12.75,12.4,12.45,579788.0
    2017-10-13,12.5,13.025,12.4,12.6,943564.0,0.0,1.0,12.5,13.025,12.4,12.6,943564.0
    2017-10-16,12.6,12.85,12.25,12.3,275900.0,0.0,1.0,12.6,12.85,12.25,12.3,275900.0
    2017-10-17,12.25,12.375,11.9,12.0,378103.0,0.0,1.0,12.25,12.375,11.9,12.0,378103.0
    2017-10-18,12.05,12.4,11.75,12.1,325309.0,0.0,1.0,12.05,12.4,11.75,12.1,325309.0
    2017-10-19,12.05,12.375,11.9,12.25,540820.0,0.0,1.0,12.05,12.375,11.9,12.25,540820
    

    Data feed created with:

                sessionend=datetime.time(17, 30),
                timeframe=bt.TimeFrame.Days,
    
    2017-10-12, 2017-10-12: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Submitted
    2017-10-12, 2017-10-12: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Accepted
    2017-10-12, Next
    2017-10-13, 2017-10-13: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Canceled
    2017-10-13, Order EXPIRED or CANCELED
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Submitted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Accepted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Completed
    2017-10-13, BUY EXECUTED, Price: 12.89, Cost: 1288.71, Comm 1.29
    2017-10-13, Next
    

    Within next(), on 2017-10-12, I place a buy order at 12.98, I'm expecting this order to be filled on 2017-10-13, and then be able to operate on the new data within that bar. I am currently running cancel() at the beginning of the next() function to ensure that there aren't orders piling up, as I would like to reevaluate the data at this point. I am then working through the logic on whether to place another buy/sell order. Since I wasn't filled, I recalculate the buy order at 12.88 and am filled at that price. If I change my logic to use the values of the lines from yesterday (-1), this seems to give me what I'm looking for, but this can't be correct, can it?

    I've messed around with various values of valid=, and don't seem to be able to get the result I'm looking for. For those that are familiar with Multicharts or Tradestation, I'm trying to get the equivalent of buy next bar at {price} stop, or more generically, I am expecting the next() function to execute my code within this function after it has decided to fill any orders from what I've executed the previous day. Do I have the wrong expectation? How do I simulate this behavior?



  • Following up to clarify.

    https://www.backtrader.com/blog/posts/2015-08-08-order-creation-execution/order-creation-execution.html#stop

    "The trigger price set at order creation if the data touches it, starting with the next price bar."

    This is the behavior I would expect and I am not seeing it. Can anyone else confirm?

    Thanks...



  • @beejaysea Hey mate, just trying to answer as I know Daniel isn't around, usually I wouldn't attempt because I'm still learning BT myself!

    Ok, I don't understand why you your buy order (I assume a buy stop order) of 12.98 wasn't filled on 2017-10-13. The high was 13.025 which is obviously greater than 12.98.
    "If the open price has not penetrated above the stop price but the high price is above the stop price, then the stop price has been seen during the session and the order can be executed
    "

    I know you cancel order in the next(), I assume right at the start before you place any new orders. But for 2017-10-13, the next() code is executed after the day has completed, so the buy order for 12.98 should have been filled.

    Please correct me anything I am missing.
    Your actual full code would help but appreciate you may want to protect your IP.



  • deleted...



  • @beejaysea

    With no code available I can only guess. Looks like your dates are somehow shifted. If you issue an order on 10-12-2017, then on 10-13-2017 bt should show you that this order is submitted and accepted, but there is no any indication about it.

    Based on you output: the original 12.98 stop order #389 was issued on 10-11-2017, submitted and accepted on 10-12-2017, checked against 10-12-2017 prices - no execution, then cancellation was sent on 10-12-2017 next(). It was cancelled on 10-13-2017 as it should be. Also on 10-12-2017 next() you issued stop order # 390, which was submitted and accepted on 10-13-2017. And executed at the same day of 10-13-2017.

    Everything works correctly.



  • @ab_trader seems right on spot. The dates are simply off and the 12.98 is actually being checked with the prices 2017-12-12 which are:

    2017-10-12,12.5,12.75,12.4,12.45,579788.0,0.0,1.0,12.5,12.75,12.4,12.45,579788.0
    

    For which an execution at 12.98 is impossible.

    But without any real input (the code which executes the buy and how logging is happening), it's just a long shot.



  • @Richard-O-Regan , @ab_trader , @Paska-Houso - Thanks to all of you for your feedback, I'll try to get a better sample together.

    @ab_trader - I added 2017-10-11, I don't believe that Order #389 is being issued the day prior. I'm not sure why the dates would be shifted either, but there is certainly something wonky somewhere. I am going to strip this whole thing down and see if I can come closer to understanding what's up here.

    Is there a known bug/feature that may cause this sort of date shift?

    2017-10-11, 2017-10-11: Order ref: 387 / Type Buy @ 13.223332970591805 / Status Canceled
    2017-10-11, Order EXPIRED or CANCELED
    2017-10-11, 2017-10-11: Order ref: 388 / Type Buy @ 13.09181788502966 / Status Submitted
    2017-10-11, 2017-10-11: Order ref: 388 / Type Buy @ 13.09181788502966 / Status Accepted
    2017-10-11, Next
    2017-10-12, 2017-10-12: Order ref: 388 / Type Buy @ 13.09181788502966 / Status Canceled
    2017-10-12, Order EXPIRED or CANCELED
    2017-10-12, 2017-10-12: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Submitted
    2017-10-12, 2017-10-12: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Accepted
    2017-10-12, Next
    2017-10-13, 2017-10-13: Order ref: 389 / Type Buy @ 12.984214633206085 / Status Canceled
    2017-10-13, Order EXPIRED or CANCELED
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Submitted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Accepted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Completed
    2017-10-13, BUY EXECUTED, Price: 12.89, Cost: 1288.71, Comm 1.29
    2017-10-13, Next
    


  • @beejaysea said in A question about next():

    Is there a known bug/feature that may cause this sort of date shift?

    That's what your code (how you use the platform) will show. For example.

    @beejaysea said in A question about next():

    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Submitted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Accepted
    2017-10-13, 2017-10-13: Order ref: 390 / Type Buy @ 12.887084699895887 / Status Completed

    How do you get an order submitted, accepted and executed on the same date? Simple output cannot let anyone know what you are doing. It can be as simple as wrong logging.



  • @beejaysea please show the code that creates orders and log strings. Otherwise it is hard to make any suggestions.

    Please also add a log print out in the next() when you issue the order.



  • @ab_trader, @Paska-Houso, yep, I totally get this. I have broken out the code (and included here). Most of the logging code and whatnot was taken from various examples, so I'm expecting that portion of it is fine.

    import backtrader as bt
    import datetime
    
    from dateutil.relativedelta import relativedelta
    
    
    class test_strategy(bt.Strategy):
        params = (
            ('printlog', True),
        )
    
        def log(self, txt, dt=None, doprint=False):
            if self.params.printlog or doprint:
                dt = dt or self.datas[0].datetime.date(0)
                print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
            super().__init__()
    
            self.dataclose = self.datas[0].close
            self.datahigh = self.datas[0].high
            self.datalow = self.datas[0].low
    
            self.orders = []
            self.order = None
    
            self.buyprice = None
            self.buycomm = None
    
            self.bars = 0
    
            self.atr = bt.indicators.AverageTrueRange(self.datas[0], period=10)
            self.ema = bt.indicators.EMA(self.datas[0], period=10)
    
    
        def next(self):
            super().next()
            self.log('Next')
    
            if self.position:
                self.bars += 1
    
            if self.order:
                self.cancel_all()
    
            offset = 0
    
            close = self.dataclose[offset]
            atr = self.atr[offset]
            ema = self.ema[offset]
    
            eofthisday = datetime.datetime.combine(self.data.datetime.date(), datetime.time(23, 59, 59, 999))
            eofnextday = eofthisday + datetime.timedelta(days=1)
            valid_date = eofnextday # This was so I could test valid= in the orders instead of cancelling the existing ones.
    
            if not self.position:
                if close < ema:
                    self.buy(exectype=bt.Order.Stop, price=ema)
            else:
                sell_at = ema - (atr * 1.5)
    
                if close > self.position.price + (atr * 0.5):
                    sell_at = ema
    
                if close > ema + (atr * 1.5):
                    sell_at = ema + (atr * 1.5)
    
                sl=self.sell(exectype=bt.Order.Stop, price=sell_at)
    
    
        def stop(self):
            super().stop()
            self.log('Ending Value %.2f' % (self.broker.getvalue()))
    
        def notify_trade(self, trade):
            super().notify_trade(trade)
            if not trade.isclosed:
                return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm))
    
        def notify_order(self, order):
            super().notify_order(order)
            self.log('{}: Order ref: {} / Type {} @ {} / Status {}'.format(
                self.data.datetime.date(0),
                order.ref, 'Buy' * order.isbuy() or 'Sell',
                order.price,
                order.getstatusname()))
    
            if order.status in [order.Submitted, order.Accepted]:
                # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                self.order = order
                self.orders.append(order)
                return
    
            if order.status in [order.Expired, order.Canceled]:
                self.log('Order EXPIRED or CANCELED')
            elif order.status in [order.Completed, order.Margin]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
            self.orders.clear()
            self.order = None
    
        def cancel_all(self):
            for o in self.orders:
                self.cancel(o)
            self.order = None
            self.orders.clear()
    
    if __name__ == '__main__':
        data = bt.feeds.QuandlCSV(
            dataname='TISI.csv',
            reverse=False,
            fromdate=datetime.datetime.now() - relativedelta(years=2),
            todate=datetime.datetime.now(),
            sessionend=datetime.time(17, 30),
            timeframe=bt.TimeFrame.Days)
    
        cerebro = bt.Cerebro()
        cerebro.addstrategy(test_strategy)
        cerebro.broker.setcommission(commission=0.001)
        cerebro.broker.setcash(10000)
        cerebro.addsizer(bt.sizers.FixedSize, stake=100)
        cerebro.adddata(data)
    
        try:
            c = cerebro.run()
            cerebro.plot(style='candle')
        except Exception as e:
            print(e)
    

    Obviously, the meat is in next(). The data is from Quandl EOD. I have executed the same code on other tickers, and see the same-sort of behavior. It's probably worthwhile to note that this behavior is not just on the buy() orders, but also the sell()s.



  • @beejaysea said in A question about next():

    Most of the logging code and whatnot was taken from various examples, so I'm expecting that portion of it is fine

    It's not.

    @beejaysea said in A question about next():

    ...
            elif order.status in [order.Completed, order.Margin]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
    ...
    

    Completed and Margin have obviously nothing to do with each other.

    And

    @beejaysea said in A question about next():

        self.log('{}: Order ref: {} / Type {} @ {} / Status {}'.format(
            self.data.datetime.date(0),
    

    You are not logging the dates related to the order statuses. You are logging the actual time at which things are being seen. And this should let you easily answer the question posed above

    @Paska-Houso said in A question about next():

    How do you get an order submitted, accepted and executed on the same date?

    Simply: you are logging things at the same point in time, but not logging the time at which they happened. This is what @ab_trader suggested as the most obvious explanation for the apparent mismatch in prices and execution.The mismatch happens because you compare the order price with the wrong prices.



  • @Paska-Houso said in A question about next():

    @beejaysea said in A question about next():

        self.log('{}: Order ref: {} / Type {} @ {} / Status {}'.format(
            self.data.datetime.date(0),
    

    You are not logging the dates related to the order statuses. You are logging the actual time at which things are being seen. And this should let you easily answer the question posed above

    @Paska-Houso said in A question about next():

    How do you get an order submitted, accepted and executed on the same date?

    Simply: you are logging things at the same point in time, but not logging the time at which they happened. This is what @ab_trader suggested as the most obvious explanation for the apparent mismatch in prices and execution.The mismatch happens because you compare the order price with the wrong prices.

    Okay. I have changed a few things around, and I think I see what's going on here, and where I need to make changes.

    Thank you.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.