@backtrader said in Matching of stopLimit orders:

@sandro-böhme said in Matching of stopLimit orders:

Somehow from the notify methods I assumed that everything will be handled asynchronously but it doesn't

That's a huge mistake. You may think it's an advantage short-term ... but it isn't. The notify_xxx do simply let you know how things have happened. They imply nothing.

That was indeed a big mistake. Thanks for your support @backtrader!

@backtrader
TL;DR: From my observation it all seems to work now as simple as it can and get exactly the way I wanted it.

Longer Version:
You wrote

the logic is supposed to be located in next

So I thought about a rough idea of extending the broker to have some kind of updateOrder(oldOrder, newArgs) method that cancels the old order, really synchronously waits for for the result and then sends the new stop-limit-sell order. When I looked into the broker I use to see if I can add an order type - surprise, surprise - I recognized that the methods are already synchronous to a big degree. E.g. when the cancel() method returns the order is already canceled on the exchange and this is already reflected in the returning order object. It is the same with the stop-limit-sell order. When the sell() method returns the order is already placed at the exchange.
Now this makes things incredibly simple. I can really just call cancel() and then send my new sell() order with the raised stop-loss price in the next() method directly afterwards because I know that the previous sell order will immediately get canceled before I send the raised stop loss and I actually don't have to wait for any notify_xxx() method to confirm that.
Somehow from the notify methods I assumed that everything will be handled asynchronously but it doesn't. At least not for the Broker implementation I use.
I tested it live on an exchange and it works like a charm!
Of course I still use the notify_xxx methods to get called when the buy or sell orders got matched.

@backtrader said in Matching of stopLimit orders:

@sandro-böhme said in Matching of stopLimit orders:

If I would like to make use of it as a stop price as in (Stop And Reverse) I guess I need to find a way to get the order send and accepted at the beginning of a candle or more precisely during a candle with the current PSAR value

This is NOT possible. Because the PSAR value is only known when the candle is closed.

• In regular mode you get the final PSAR value with the final bar prices

• In replay mode you get the temporary PSAR value with the known bar prices.

In no case can you have the PSAR value at the beginning of the bar. At the beginning of the bar you have the previous PSAR value.

I wondered what would be the use of it if I only have the PSAR stop-loss value of the elapsed bar represented in next() but I figured that of course one can place a market sell order if the market price in next() is below the PSAR one can access in next(). But that might not save the investment very well as the price can have headed south way more than with a stop loss. Especially in Cryptos.

It feels wrong to write that now but my understanding is, that in the current next() call I should get the PSAR value for the subsequent candle.
The Wikipedia page about the Parabolic SAR says

That is, tomorrow's SAR value is built using data available today.

Also if I use the indicator in TradingView (my charting tool of choice) it is set already at the beginning of the candle and doesn't change even though the final high and low values of that candle are obviously not yet established.

At the end of course Backtrader can have used the previous high and low values to generate the current PSAR value at the end of the candle in next(). This is why I logged this values in the indicator and in next() to see if the input of the PSAR is taken from the high and low values available in next(). Every first line comes from the indicator and obviously the every second from the next() call.

2019-02-15T09:27:00, PSAR High: 9.3882 Low: 9.3877
2019-02-15T09:27:00, next() Open: 9.3877, High: 9.3882, Low: 9.3877, Close: 9.3882, PSAR: 9.3505 in next()
...
2019-02-15T09:28:00, PSAR High: 9.3919 Low: 9.3919
2019-02-15T09:28:00, next() Open: 9.3919, High: 9.3919, Low: 9.3919, Close: 9.3919, PSAR: 9.351254 in next()
...
2019-02-15T09:29:00, PSAR High: 9.3839 Low: 9.3839
2019-02-15T09:29:00, next() Open: 9.3839, High: 9.3839, Low: 9.3839, Close: 9.3839, PSAR: 9.35287984 in next()
...
2019-02-15T09:30:00, PSAR High: 9.3893 Low: 9.3893
2019-02-15T09:30:00, next() Open: 9.3893, High: 9.3893, Low: 9.3893, Close: 9.3893, PSAR: 9.3544406464 in next()

I don't know yet why the ohlc values are equal so often (probably an issue specific to my setup) but it seems to me that the PSAR uses the current close values from next(). Which is perfect for me as I can use this value then in next() for the stop-limit-sell order that will have the order placed before the method finished which should mean that it is available quite early in the next candle.

But in the end of course the PSAR indicator or Backtrader platform might do something I don't recognize that leads to next() indeed having the PSAR of the current candle and not the one for the next candle as I expect it.

For me it least it now looks like it works exactly as I want it.

@backtrader said in Matching of stopLimit orders:

@sandro-böhme said in Matching of stopLimit orders:

If you see other problems please tell me as well as I plan to trade real money with that code.

You need to see the problems and think twice about your money.

Thanks for taking the time to respond @Backtrader. That's much appreciated!

Are you expecting people to read and debug your code for you? Problem #1.

No, I don't but I wanted to give the opportunity that people can do that if they want.
I wasn't sure why you was assuming 6.9994 is the Limit price. I worried that you actually had a look in the code and I made a mistake there and mixed something up.

Log things properly, because I can have a quick look at the log and you can today relate the code to the log. When you look at the log in the future you won't know what it relates too. If there are ohlc values and trigger and limit prices, log it all. And log the version of the code you are working with if you ever want to relate the log to the actual code. (you may log a git sha hash or something, but something that lets you relate the log to the actual code)

Thanks for the hint. I agree that it makes sense to at least log the ohlc values and trigger and limit prices.

In any case, setting the Limit price of a sell-StopLimit below the trigger price means you want to give money away for free.

Let me explain why I do that so you can correct me if I'm wrong.

Assume I set the limit order to have the same price as the trigger price to not give away money for free.
As I send a stop limit sell order I expect the market price to drop when the order is triggered and the limit sell order is added to the order book.
But if the price drops very fast not all the other sell orders before mine could have been bought away by buyers before the market price is below my limit sell order. Then my sell order would be too expensive for the buyers and not get filled.
The lower my limit price is compared to the trigger (stop) price the higher the chance that my sell order gets filled.
This is why I would say I give away the money to pay for lowering the risk of the sell order not to get filled.

@sandro-böhme said in Matching of stopLimit orders:

Thanks a lot, that's a very good point!

Problem #2.

It's not a good point. It is simply the point. It doesn't matter if 6.9994 is your trigger or limit price. It is never reached when your order is alive. Your order can be neither triggered nor executed.

Of course you are right again.
I just haven't checked it by myself at that time and wasn't sure if there is maybe (also) a bug in my code which is why I didn't know yet it was the point.

I'll compare it with the other bar and I guess I'll see the difference.
If I don't I'll try to find the matching logic and debug the code to better understand the behavior.

You don't need to understand the matching logic. It is as easy as: it cannot be matched if the order prices are not in the range of the bar.

You are absolutely right. Again... :-)

You need TRADING experience.

Absolutely. I'm working on that as well. To be precise I would say I'm towards the end of the "Conscious Incompetence" stage of the stages of competence. So at least I should have left the Dunning-Kruger effect behind me ;-).
In this thread I already learned quite a bit thanks to your feedback!

When you have TRADING experience, how and when orders are matched will be obvious to you. My suggestion: spend several hours see how things work in a market.

There is no doubt that will happen. Not only regarding trading in general but also when I run my code with real money agains an exchange. I plan to set breakpoints, examine the actual values I get from the exchange and thus always have the chance to stop the code from running. Even though I cannot stop the exchange to execute the order I have send some buy orders already with prices well below the market price to see if they show up on the exchange the way I expected. I also did that with careful debugging and examining the actual values of the objects.

See this other recent thread: Community - Stop Order execution price? Why does the poster believe he has found a flaw in the logic for the execution go a Stop order?

• Simple: the poster has NO trading experience.

Yes I've seen that as I read some threads (and docs of course) before posting. This thread was actually the reason I was specifically talking about a stop-limit order and not only about stop-loss.

@sandro-böhme said in Matching of stopLimit orders:

Basically submitting the sell-StopLimit takes the time of a candle

Problem #3. This is a completely wrong statement. And very dangerous. The candle you are working with contains PAST prices. Here and China. Order submission DOES NOT take the time of a candle. The candle DOES NOT longer exist. You work and can only work with PAST information.

The order is submitted and can ONLY be applied and evaluated with the next set of incoming prices.

Sure I'm sorry that I wasn't more specific to be really clear.
I'll try to describe how I understand that more clearly now so you can correct me if I'm wrong.

The next() method runs after a candle has been closed when all final ohlc data for that candle is available. Judging from the logs the notify_...() methods seem to get called directly after the first next() and before the subsequent next(). One could see this as the time during the candle.
I submit the sell order when I get the "cancel" notification in notify_order(). This is after the previous next() call and before the subsequent next() call. As written above I think this is the time the ohlc data for the subsequent next() candle is created. Technically I cannot really know that it takes the full time of the candle as I could as well get the notify_order() call at the last minute of the 15m candle. So of course you are right that this is a wrong statement.

@sandro-böhme said in Matching of stopLimit orders:

I think the ParabolicSAR assumes that I can get the order accepted at the same candle

Problem #4

The ParabolicSAR assumes nothing. The indicator simply delivers a value. It is a math formula. You are making the assumption. Don't put the the blame on the math. And yes wording is important because it shows how you perceive things.

If I would like to make use of it as a stop price as in (Stop And Reverse) I guess I need to find a way to get the order send and accepted at the beginning of a candle or more precisely during a candle with the current PSAR value. Currently I always have a candle where there is no sell-stop-limit order set as I submit the order during notify_order() which is before next() and get it accepted after next().
But of course you are right. The indicator simply delivers a value as every other indicator.

@sandro-böhme said in Matching of stopLimit orders:

Is there a way I can get the next() and notify_...() methods called multiple times during a candle independent of the time frame e.g. in a 1m timeframe during backtesting and live trading?

Problem #5

Is there any way in which you can get future information? Yes, by entering the vortex which is generated with tachyon particles and a thoron emitter whilst the frequency of the shields is remodulated to match the rotating harmonics of the reverse polarity of the feedback impulse traveling over the tractor beam.

Oh yes, you will read about replaying: Docs - Data Replay. But of course, you will then have to decide when to make the trade within the bar and even with replay ... the prices you see are already PAST prices. What you see is always PAST data.

Thanks for that information that is very helpful! I'll have a look into data-replay.

@backtrader said in Matching of stopLimit orders:

Summary

• You have a sell-StopLimit order with a Limit at 6.9994 and the suspect bars are far away from that price
  You are absolutely right. I've enhanced my log output:

2019-01-28T13:30:00, Order #16 Submitted
2019-01-28T13:30:00, Order #16 Accepted
2019-01-28T13:30:00, Open: 6.9704, High: 6.99, Low: 6.8757, Close: 6.9, Psar: 6.87364 in next()
2019-01-28T13:30:00, Order #16 canceled in next()
Trend switched. Sar: 6.9994 (fbsl: 6.41, prev. ep: 6.9994)
2019-01-28T13:45:00, Order #16 Canceled
2019-01-28T13:45:00, Order #17 submitting stop limit sell raise from 6.8422 to 6.9994
2019-01-28T13:45:00, Open: 6.9054, High: 6.94, Low: 6.8302, Close: 6.8351, Psar: 6.9994 in next()
2019-01-28T14:00:00, Order #17 Submitted
2019-01-28T14:00:00, Order #17 Accepted
2019-01-28T14:00:00, Open: 6.835, High: 6.8753, Low: 6.6972, Close: 6.7233, Psar: 6.996016 in next()
2019-01-28T14:00:00, Order #17 canceled in next()

Basically submitting the sell-StopLimit takes the time of a candle and matching will be done in the next one. At the time the order got accepted and triggered below 6.9994 the highest market price (6.835) was already too low compared to what I was ready to sell for (6.9994 - 0.1 slippage = 6.8994) with my limit order.
I think the ParabolicSAR assumes that I can get the order accepted at the same candle I get it's value.
If the sell order would have been accepted on the same bar it would have been sold as the High value was 6.94 at 13:45 which would have been triggered the stop price (6.9994) and which would have resulted in my sell offer of 6.8994 of my limit price to be cheaper than the highest market price of 6.94.
I'll also rethink about raising my slippage to allow for cheaper prices to still match my limit prices but that has to fit to real market conditions.

Is there a way I can get the next() and notify_...() methods called multiple times during a candle independent of the time frame e.g. in a 1m timeframe during backtesting and live trading? It should neither be a problem for the exchange nor for the library I use for the exchange (CCXT).

@backtrader said in Matching of stopLimit orders:

@sandro-böhme said in Matching of stopLimit orders:

2019-01-28T13:45:00, Order #17 submitting stop limit sell raise from 6.8422 to 6.9994
2019-01-28T13:45:00, Close, 6.8351 psar: 6.9994 in next()
2019-01-28T14:00:00, Order #17 Submitted
2019-01-28T14:00:00, Order #17 Accepted
2019-01-28T14:00:00, Close, 6.7233 psar: 6.9960 in next()

Your order is a sell-StopLimit with a Limit price of 6.9994 (we don't know what the Stop trigger price is, but it doesn't really play a role in this case)

The only thing we know about the candle at 14:00 is that the close price is 6.7233, well below the Limit price you have set for the sell-StopLimit.

Judging from the code according to the link above (some formatting aside):

self.sell_order = self.sell(size=self.params.buy_pos_size, exectype=Order.StopLimit,
              price=self.psar.lines.psar[0],
              plimit=self.psar.lines.psar[0] - self.params.slippage)
self.log(f'Order #{self.sell_order.ref} submitting stop limit sell raise '
              f'from {round(prev_sell_order_price, 4)} to {round(self.psar.lines.psar[0], 4)}'
)

I would expect 6.9994 to be the stop price and plimit=self.psar.lines.psar[0] - self.params.slippage to be the Limit price or did I mix something up?

Because we don't know the actual range of the bar and the chart you post is obviously not interactive, one can only assume that a couple of bars match that closeand none of them actually has a high which is close to 6.9994.

Thanks a lot, that's a very good point! self.params.slippage is 0.1 I'll check the bar.
I should have mentioned that earlier but at the end a sell-StopLimit order matched.

2019-01-28T17:15:00, Order #23 Canceled
2019-01-28T17:15:00, Order #24 submitting stop limit sell raise from 6.3618 to 6.2666
2019-01-28T17:15:00, Close, 6.0953 psar: 6.2666 in next()
2019-01-28T17:30:00, Order #24 Submitted
2019-01-28T17:30:00, Order #24 Accepted
2019-01-28T17:30:00, Order #24 Completed
2019-01-28T17:30:00, Trade #1 Closed
2019-01-28T17:30:00, Trade closed. Hit p_sl (6.26661325220262). Sold at: 6.1666132522026205.
2019-01-28T17:30:00, Close, 6.1843 psar: 6.2274 in next()

I'll compare it with the other bar and I guess I'll see the difference.
If I don't I'll try to find the matching logic and debug the code to better understand the behavior.
Any pointers about where I can find it are much appreciated if you have them handy.

Summary

• You have a sell-StopLimit order with a Limit at 6.9994 and the suspect bars are far away from that price

The question

• Why are you expecting execution to happen?

Your order will only be executed at the Limit price or better (for a sell order, a better price means a higher price)

Thanks a lot!

I haven't read that somewhere but I would have expected that e.g. the order #17 gets matched after notify_orders() in candle "14:00" and before next() or after next() of candle "14:00" as the cancellation did not hit the exchange yet (no "cancel" notification).
But judging from your feedback I guess I have to find an other place where I cancel the sell order. Is there a best practice for that?

When does the matching actually happen?

First of all, thanks for this amazing work on Backtrader. It looks like being exactly what I was searching for and I'm really looking forward to use it more!
I have to say that I'm just starting out with Backtrader and trading and Python in general which is why there might be some basic errors.

++ The goal ++
I want to implement a stop loss mechanism based on the Parabolic SAR.
After buying I want to raise the stop limit order according to the value of the indicator. This is why I cancel the sell order on every next() and submit a new one after I get a canceled notification with a raised price according to the ParabolicSAR. It should look like that:

++ The order management ++
I call the cancel() method in next() (lets call it "next-1") and the order is actually canceled after the bar is finished and before the subsequent next() method. Lets call it "pre-next-2".
In notify_order() when I get the canceled notification ("pre-next-2") I immediately submit a raised stop limit order. It gets accepted after the current bar and before next() of the next bar. This would be "pre-next-3".
This is in line with what I read in the "Order Management" documentation.
The log output looks like that:

2019-01-28T13:15:00, Order #15 Canceled
2019-01-28T13:15:00, Order #16 submitting stop limit sell raise from 6.7965 to 6.8422
2019-01-28T13:15:00, Close, 6.9770 psar: 6.8422 in next()
2019-01-28T13:30:00, Order #16 Submitted
2019-01-28T13:30:00, Order #16 Accepted
2019-01-28T13:30:00, Close, 6.9000 psar: 6.8736 in next()
2019-01-28T13:30:00, Order #16 canceled in next()
Trend switched. Sar: 6.9994 (fbsl: 6.41, prev. ep: 6.9994)
2019-01-28T13:45:00, Order #16 Canceled
2019-01-28T13:45:00, Order #17 submitting stop limit sell raise from 6.8422 to 6.9994
2019-01-28T13:45:00, Close, 6.8351 psar: 6.9994 in next()
2019-01-28T14:00:00, Order #17 Submitted
2019-01-28T14:00:00, Order #17 Accepted
2019-01-28T14:00:00, Close, 6.7233 psar: 6.9960 in next()
2019-01-28T14:00:00, Order #17 canceled in next()
2019-01-28T14:15:00, Order #17 Canceled
2019-01-28T14:15:00, Order #18 submitting stop limit sell raise from 6.9994 to 6.9841
2019-01-28T14:15:00, Close, 6.7137 psar: 6.9841 in next()

But the order is not matched as I expected which is why I guess I miss some information. Can you tell me when it is supposed to match by Backtrader?

If needed the strategy can be found here: https://github.com/sandroboehme/cryptotrader/blob/master/cryptotrader/strategy/trailing_stop_loss_v2.py

And it can be debugged using this script (no api-keys of the exchange needed):
https://github.com/sandroboehme/cryptotrader/blob/master/backtestPsarStopLoss.sh

If you see other problems please tell me as well as I plan to trade real money with that code.
Thanks a lot in advance for your feedback!

I think the most up to date repo is this one:
https://github.com/Dave-Vallance/bt-ccxt-store