Realistic execution price for market orders with intraday data

[brunof]

Hello,

Regarding the execution of Market orders with the opening price of the next bar, it makes sense when using daily bars but maybe not when using intraday data. I'm using 10 minute bars and, with rare exceptions, the open price of a bar is always the same as the close price of the previous bar. In a live trading scenario, by the time a new bar is received and processed, the opening price of the next bar is already in the past. For that reason, it seems unrealistic to execute market orders at the open price of the next bar.

Questions:

1. Does that make sense to you or am I missing something?

2. When backtesting using intraday data, is there a way to use the close price of the next bar for the execution of market orders?

Thank you.

[EMR]

@brunof Maybe you do not have small intraday gaps on the market that you trade, but on futures I do see some of them.
When the trading system receive the close of a bar, this information is already past information. When the market order is sent to the exchange, event with the smallest delay, you have at least a few milliseconds delay (except of course if you are citadel and trade HFT in which case it will be smaller)
So : you receive a price, then (time delay) issue a market order, then (delay) the market order hit the bid / ask at the exchange. At this point, you have no mean to know at which price your market order will be filled.
Given the constraints of BT (no bid / ask, no tick data level slippage) the most realistic is that it is filled at the first next price encountered : the open of the following candle.

[EMR]

Sorry, small mistake, please read :
"So : you receive a price, then (time delay) issue a market order. At this point, you have no mean to know at which price your market order will be filled. Then (delay) the market order hit the bid / ask at the exchange. "

[davidavr]

@brunof This sounds to me like an issue with the source data bars, not with Backtrader. If you are truly aggregating tick data into bars, then a given tick would only be included in one bar. So if the close price of one bar is the same as the open price of the next bar, that would only occur if there were two separate ticks at the same price (which is quite common, btw). It would be interesting to look at the underlying tick data for some of your ten-minute bars and see if they are consistent with what you'd expect.

I think Backtrader's assumption of filling at the open of the next bar is as good of an assumption as you can make. If indeed, there were two ticks at the same level that crossed the boundary of a bar, then you would expect to get filled at the same level as the previous bar's close.

If you want a much more realistic simulation of execution behavior on an exchange, I think you'd ultimately need to use tick data.

[brunof]

Hello @davidavr and @EMR,

Thank you very much for your replies.

I see that the issue is more specific to my use case than I initially thought.

My application receives tick data (from a high quality data provider) and creates the OHLCV bars. I'm trading a highly liquid futures contract, which often has a few dozen ticks (i.e., trades) per millisecond. The fact that the close price of a bar is often the same as the open price of the next bar is not important (sorry if my initial message was confusing). Still, I can think of two scenarios where, in my opinion, using the open price of the next bar for order execution is not very realistic:

Scenario 1: When trades are manually executed. It may take a few minutes for someone to be notified about a trade opportunity and enter the order.

Scenario 2: Even with automated execution, some systems (such as mine) take over 10 seconds between receiving the ticks required to build a bar, processing them and making a trade decision. Python is not the fastest language, so I'd guess that most systems are unable to receive a bar, make a decision, and enter an order, all within the first tick of the next bar.

I understand that Backtrader has a slippage feature that can help with that, but I would rather use real prices than a fixed slippage amount or percentage. Executing the trade with the close price of the next bar, which in my case is the price from 10 minutes later, sounded like a simple solution.

It just occurred to me that using the VWAP of the next bar should be more realistic. Of course, it could only be used as the execution price and not to calculate indicators or make trade decisions (otherwise it would introduce a massive look-ahead bias). I've found a couple of possible solutions by searching for the "VWAP" keyword:

• Option 1: Suggested by ab_trader in this thread: Use a StopLimit order and specify the VWAP as both the price and plimit parameters.

• Option 2: Since I'm not using open prices for anything else in Backtrader, I could pass the VWAP as the open prices, which would trick Backtrader into executing Market orders with the VWAP (inspired by this thread).

I'm not considering solutions that check the tick data to get the exact amount of slippage because, in my case, it would be too slow (my tick data does not fit into memory, doing anything with it is slow).

Thanks again.

[davidavr]

@brunof I think Option 2 sounds pretty reasonable. It may seem a bit kludgy to replace the Open field with a different value, but I looked at the execute method in BackBroker and it's pretty involved so subclassing and overriding would be complicated. Note you could replace the Open with whatever you like: VWAP, the first tick 10 seconds after the open, etc. - whatever you think is a reasonable fill price.

[brunof]

Thanks for checking the code, @davidavr. I agree, both solutions are a bit hacky, but I was unable to think of a better solution so far. I will go with "Option 2" as it seems simpler.

[run-out]

@brunof said in Realistic execution price for market orders with intraday data:

Executing the trade with the close price of the next bar, which in my case is the price from 10 minutes later, sounded like a simple solution.

Use cheat-on-open and get the the self.data.close[0] in next open. This will be the close on the next bar and you can use that for setting your order prices before the bar opens.

It's my civic duty to mention being cautious about look ahead bias, the number one reason for great performance.

Using the example in the link I provide above, and removing operate since we are not making orders here, you get the following:

2006-12-20 FROM NEXT,      open 4108.3 close 4118.54
2006-12-21 FROM NEXT_OPEN, open 4111.85 close 4112.1 


2006-12-21 FROM NEXT,      open 4111.85 close 4112.1
2006-12-22 FROM NEXT_OPEN, open 4109.86 close 4073.5 


2006-12-22 FROM NEXT,      open 4109.86 close 4073.5
2006-12-27 FROM NEXT_OPEN, open 4079.7 close 4134.86 


2006-12-27 FROM NEXT,      open 4079.7 close 4134.86
2006-12-28 FROM NEXT_OPEN, open 4137.44 close 4130.66 


2006-12-28 FROM NEXT,      open 4137.44 close 4130.66
2006-12-29 FROM NEXT_OPEN, open 4130.12 close 4119.94 

[brunof]

Hello @run-out,

Thank you for your reply. I will try that as well.

BTW, that's some interesting stuff you have on RunBacktest.com :)

[run-out]

@brunof Thanks! It's a most satisfying way to make a living!