Bug building bars in live mode regarding last tick?

vbs

Hello gentlemen,

I am new to this place and I am not quite experienced with automated trading. Currently I am experimenting with Backtrader's live mode and I have a problem (which I think it is a bug):

When feeding live ticks into Backtrader then bars are built in real-time as ticks fly in. I might be wrong but I think there is a bug about how BT handles the first/last tick of a bar.

As far as I know it is the standard way, that when (for example) using M1 bars then the bar at, let's say, "14:00"includes tick data from 13:00.000 to 13:59.999. So a tick at exactly 14:00.000 would be the first tick for the following "15:00"-bar. Is this correct so far?

I think BT actually is treading a tick at 14:00.000 as the last tick of "14:00"-bar (instead as the first tick of the "15:00"-bar).

An example output generated by my test code to illustrate the problem:

{'Time': datetime.datetime(2017, 12, 22, 20, 55, 57, tzinfo=<UTC>), 'Last': 142.0, 
{'Time': datetime.datetime(2017, 12, 22, 20, 55, 58, tzinfo=<UTC>), 'Last': 143.0, 
{'Time': datetime.datetime(2017, 12, 22, 20, 55, 59, tzinfo=<UTC>), 'Last': 144.0, 
{'Time': datetime.datetime(2017, 12, 22, 20, 56, tzinfo=<UTC>), 'Last': 145.0, 
2017-12-22, Bar finished: Time: 2017-12-22 20:56:00 O:101.0 H:145.0 L:101.0 C:145.0 - Len: 1

The first 4 lines in the log are ticks that are coming in with values starting at 142 incrementing with every tick. Then at 20:56:00 the bar finishes. The close price is 145 but the tick which included the value 145 had the timestamp 20:56.000. So I had expected BT to not included that tick into the 20:56-bar but to be the opening tick for the following 20:57-bar. So the 20:56-bar's close price would be 144.

So, do I misunderstand something or is it really a bug? If yes, is there any chance that it could be fixed?
Thanks in advance!

Paska Houso

@vbs said in Bug building bars in live mode regarding last tick?:

As far as I know it is the standard way,

After reading your post ... I think that's the key in your question. You are describing that standard way as your expectation, and the platform is giving you something slightly different.

As such ... it doesn't seem like a bug, but rather a mismatched expectation. You may also try to use the parameters available for resampling:

https://www.backtrader.com/docu/data-resampling/data-resampling.html

rightedge

(Of course I cannot personally guarantee that it will deliver to your expectations, but it seems the closest thing to what you seem to be asking)

vbs

Ok I see, maybe it is not as standardized as I thought. :) Great conincident btw, I was just about to click the "Submit" button when your answer popped up ;)

I think rightedge does again mean something different though.

Behaviors for H1 bars (example with bar timestamp 14:00:00):

Current (rightedge enabled): 13:00:01 to 14:00:00
rightedge disabled: 14:00:00 to 14:59:59 (or maybe 14:00:01 to 15:00:00?)
my assumption was: 13:00:00 to 13:59:59

When writing the following paragraph I assumed the current behavior was wrong. I have just learned it is not wrong but just different from by expectations:
Anyway, I digged my way several hours through the code of BT trying to understand the resampler mechanics. I think I have build a a bit of comprehension and found the issue (which might not be an issue as I have just learned).

The case is about detecting if a tick (or a bar) is on the ending edge of the current bar and should actually close the bar. Then the question is if the tick (or bar) in question should be included in the current bar which is about to be closed or not.
Current behavior is correct when upsampling for example M1 bars to M5 as the last M1 bar should be fully included in the M5 bar.
But this behavior is wrong when upsampling ticks since for example a tick at 14:00:00 should not be included into the bar ending at 14:00:00. Instead a new bar should be opened and the tick should be included there.

I have commited a fix into my repo here:
https://github.com/verybadsoldier/backtrader/commit/76d52d72c550606eb06b84138dca8f08d3b85130

I think the variable consumed controls if the current data should be included before or after switching bars.

So, maybe I should make my "fix" configurable so both ways are possible? Maybe someone being more sound with the BT code could give a comment if I am on the right path with my patch or if there is a more cleaner or better way to do it?

Thank you guys & merry xmas!

ab_trader

@vbs another parameter boundoff can be used to shift the boundaries of the bar. It seems to me that it can work in your case.

vbs

@ab_trader

Thanks for the hint. I fiddled with that parameter before but in my understanding it is not exactly the same (please correct me if I am wrong).
As far as I can see that boundoff can be used to shift in target units and only is useful when using compression values greater than 1 (shifting M1 boundaries by minutes makes no difference). But I need to shift the source units, e.g. 14:00:00.000 to 13:59:59.999.

I still tried it and when setting boundoff to 1 when upsampling ticks to M1 bars then I rarely get bars at all. I think something wents quite wrong in that case.

ab_trader

@vbs
no problem, but based on description of the boundoff parameter it uses number of source units to shift the new bar. You need to shift your 1 min bar 1 tick forward.

bt approach - 14:00:01 - 15:00:00
you need (if I understand correctly) - 14:00:00 - 14:59:59 - 1 tick forward shift

Description shows same case but on 1 min to 15 min transformation. Maybe it doesn't work for ticks as source units.

vbs

@ab_trader
Hm, I have to admit that I cannot find that in the documentation of boundoff. The example is a bit different since both source and target units are minutes (unlike in my case ticks->minutes).
Also judging from the source code, I think the code in question is solely working on resampler target units.

Paska Houso

https://www.backtrader.com/docu/data-resampling/data-resampling.html

boundoff (default: 0)

Push the boundary for resampling/replaying by an amount of units.

If for example the resampling is from 1 minute to 15 minutes, the default behavior is to take the 1-minute bars from 00:01:00 until 00:15:00 to produce a 15-minutes replayed/resampled bar.

If boundoff is set to 1, then the boundary is pushed 1 unit forward. In this case the original unit is a 1-minute bar.
Consequently the resampling/replaying will now:

Use the bars from 00:00:00 to 00:14:00 for the generation of the 15-minutes bar

vbs

@Paska-Houso
Thank you but I didn't mean that I cannot the documentation in general :). When I said "I cannot find that in the documentation" I meant this statement from @ab_trader :

based on description of the boundoff parameter it uses number of source units to shift the new bar

Sorry for confusion...

Paska Houso

Misread your message. After re-reading ... I understand why @ab_trader believes it does so. From the doc:

If boundoff is set to 1, then the boundary is pushed 1 unit forward. In this case the original unit is a 1-minute bar.

The "source unit" (in your description) seems to be the "original unit" (in the documentation). That doesn't mean that it is actually doing what you are expecting.

ab_trader

I've played a bit with boundoff, daily and resampled weekly data trying to get shifted weekly bar, but wasn't able to make it. Maybe it should be same timeframe specified for both original and target bars in order to force it to work.