Replaying Data - Effects on Signal Generators

ThatBlokeDave

I have some indicators/signal generators which spot certain candle types e.g Hammer, Hanging Man etc and then produce by and sell signals.

This relies on checking the ratios of the body and wicks using data.open, data.high, data.low and data.close values.

When replaying data I find that I am generating signals more signals than intended whilst the higher timeframe candle is forming. This is because it looks like each call of next() provides the same open price but is still updating the high, low and close based on the new data as it comes in.

An example is below of open, high, low and close values for a 1 minute data set using 15 minute replay compression. The opens stay the same whilst the highs, low and closes are updating (and rightly so since the candle is still forming)

O: 1.45253 H: 1.4526 L: 1.45251 C: 1.45258
O: 1.45253 H: 1.45268 L: 1.45251 C: 1.45267
O: 1.45253 H: 1.45272 L: 1.45251 C: 1.45271
O: 1.45253 H: 1.45278 L: 1.45251 C: 1.45272
O: 1.45253 H: 1.45278 L: 1.45251 C: 1.45274
O: 1.45253 H: 1.45278 L: 1.45251 C: 1.45271
O: 1.45253 H: 1.4528 L: 1.45251 C: 1.4528

My problem is that since the bt.Indicator's also call the next() method, it causes me to generate signals whilst the candle is still forming on each minute.

What is the best way to handle this? Should I take prevent calculations until the bar has finished forming and then take the highest high and lowest low from the series and then final close value to make the calculations?

Or perhaps I missed something in the docs which would make this task much simpler? I don't see a data1 being generated that I can look into instead of the lower timeframe.

backtrader

The suspicion here is that you are looking for resampledata instead of replaydata.

The former delivers the timeframe/compression bar once it is complete. If you resample minutes to days, the system will process the minute-bars and don't deliver anything until a day is complete.

With replaydata you get the daily-bar but seeing the development and hence you will get called many times. With this model you can for example watch the development of a daily prices/indicator value and issue an order 5 minutes before the close.

If you still want to use replaydata, even if it doesn't seem so, you can control when the next bar comes in by checking the len(self.data)

Replay data is the reason why all indicators have formulas that are idempotent (if called again with no changed values, next will deliver the same value)

ThatBlokeDave

@backtrader

Thanks. I was previously using resample data. It was working fine in backtesting but when testing live I noticed something that I was trying to resolve by using replay :)

Basically all the order.status notifications are not received immediately but are delayed to the next bar.

So for example if I wanted to create a stop loss order only once the order is completed (to avoid having the stop order hit being hit before the initial order is opened or having an order rejected but the stop accepted), I would need to wait a whole bar before I know whether the order is open/completed. If I am on the higher timeframes (hourly, daily etc) that can be a long time to wait to put a stop loss order in.

So I thought I could get the notifications earlier by replaying the data. I should get the notification within a tick.

So my currently workflow with resample is:

Bar 1: Buy Sell signal received
Bar 2: Place order (buy/sell)
Bar 3: Order Completed notification received, create stop loss order
Bar 4: SL order accepted notification received

One thing I can think of is to re-sample with two timeframes. What do you think?

backtrader

@ThatBlokeDave said in Replaying Data - Effects on Signal Generators:

I was previously using resample data ... Basically all the order.status notifications are not received immediately but are delayed to the next bar.

That's inaccurate. The non-resampled bars are not seen by the broker. You know they are there, but the broker doesn't. The simulated broker can only act on the bars it sees.

So I thought I could get the notifications earlier by replaying the data. I should get the notification within a tick.

In this case the broker sees all the ticks and will execute when possible. You get the notification exactly as fast as in the previous case. The difference being that the ticks accumulate on the same bar.

One thing I can think of is to re-sample with two timeframes. What do you think?

That will of course speed up the notifications to your strategy.

ThatBlokeDave

@backtrader said in Replaying Data - Effects on Signal Generators:

That's inaccurate. The non-resampled bars are not seen by the broker. You know they are there, but the broker doesn't. The simulated broker can only act on the bars it sees.

Perhaps delayed is the wrong word to use. However when trading live, resampling means that I cannot confirm my position was opened until the next bar which could be a very long time. It would be useful to allow notifications to be received as soon as they are provided by the live broker, when ever that happens to be. (A bit like we have an option to cheat on close for those who need it).

The broker has already provided important information which is actionable regardless of whether you are primarily working of a particular timeframe.

Another helpful alternative would be to extend the recent OCO, brackets etc order execution types to the live stores. This would mean I don't need to wait for confirmation before placing a stoploss order .

Anyway, these are just my thoughts and suggestions. I do appreciate the hard work, care and thought that goes into developing the platform.

backtrader

@ThatBlokeDave said in Replaying Data - Effects on Signal Generators:

Perhaps delayed is the wrong word to use. However when trading live, resampling means that I cannot confirm my position was opened until the next bar which could be a very long time. It would be useful to allow notifications to be received as soon as they are provided by the live broker, when ever that happens to be. (A bit like we have an option to cheat on close for those who need it).

It is a valid use case and can be considered. There will be a difference between a real broker and a the simulation

The real broker will get your stop-loss order and the execution can take place in the same bar before it is fully resampled
The simulated broker doesn't see the ticks during resampling and can only execute the stop-loss order when the next bar is complete.

It would be a valid use case too, to pass the ticks to the broker, but the changes involved in doing that are not possible with simply some changes.

Another helpful alternative would be to extend the recent OCO, brackets etc order execution types to the live stores. This would mean I don't need to wait for confirmation before placing a stoploss order

Interactive Brokers has support OCO and Brackets and it is included.

backtrader

Immediate broker notifications have been added. See: Community - Release 1.9.44.116

Add the parameter quicknotify=True to cerebro. See: Docs - Cerebro

ThatBlokeDave

@backtrader

Amazing! Thank you, that will be a big help.