Thanks for your quick reply @backtrader
You can't claim backtrader
is perfect, but I can claim backtrader
is ... awesome!
I can understand the complexity using tickString
, so let's go with RealTimeBars
data:
Because I do not want bars as be delivered as much in real-time as possible and I perfer to "wait" a while I've set the following (qcheck=2
, timeoffset=True
and rtbar=True
):
ibstore = bt.stores.IBStore(
host=args.host, port=args.port,
clientId=args.clientId, timeoffset=True,
reconnect=args.reconnect, timeout=args.timeout,
notifyall=args.notifyall, _debug=args.debug)
cerebro.setbroker(ibstore.getbroker())
data0 = ibstore.getdata(dataname=args.data0,
backfill_start=True,
timeframe=bt.TimeFrame.Minutes,
compression=1,
name='d0',
qcheck=2,
rtbar=True)
First run: check resampled data with debug mode enabled:
...........
<realtimeBar reqId=16777217, time=1497608880, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608885, open=164.35, high=164.35, low=164.35, close=164.35, volume=4, wap=164.35, count=3>
<realtimeBar reqId=16777217, time=1497608890, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608895, open=164.35, high=164.35, low=164.34, close=164.34, volume=9, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608900, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608905, open=164.34, high=164.34, low=164.34, close=164.34, volume=205, wap=164.34, count=2>
<currentTime time=1497608915>
<realtimeBar reqId=16777217, time=1497608910, open=164.34, high=164.34, low=164.34, close=164.34, volume=133, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497608915, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497608920, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608925, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608930, open=164.35, high=164.35, low=164.35, close=164.35, volume=79, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608935, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
2017-06-16T12:29:03.091121: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
<realtimeBar reqId=16777217, time=1497608940, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608945, open=164.35, high=164.35, low=164.35, close=164.35, volume=30, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608950, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608955, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608960, open=164.35, high=164.35, low=164.34, close=164.34, volume=13, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497608965, open=164.34, high=164.34, low=164.34, close=164.34, volume=15, wap=164.34, count=1>
<currentTime time=1497608975>
<realtimeBar reqId=16777217, time=1497608970, open=164.34, high=164.34, low=164.34, close=164.34, volume=13, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608975, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608980, open=164.35, high=164.35, low=164.35, close=164.35, volume=13, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608985, open=164.35, high=164.36, low=164.35, close=164.36, volume=48, wap=164.35, count=15>
<realtimeBar reqId=16777217, time=1497608990, open=164.35, high=164.35, low=164.35, close=164.35, volume=129, wap=164.35, count=19>
<realtimeBar reqId=16777217, time=1497608995, open=164.35, high=164.35, low=164.35, close=164.35, volume=26, wap=164.35, count=2>
2017-06-16T12:30:03.100416: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 289
<realtimeBar reqId=16777217, time=1497609000, open=164.34, high=164.34, low=164.34, close=164.34, volume=21, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609005, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609010, open=164.34, high=164.34, low=164.34, close=164.34, volume=104, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497609015, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609020, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609025, open=164.33, high=164.33, low=164.33, close=164.33, volume=1, wap=164.33, count=1>
<currentTime time=1497609035>
<realtimeBar reqId=16777217, time=1497609030, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609035, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609040, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609045, open=164.33, high=164.33, low=164.33, close=164.33, volume=286, wap=164.33, count=24>
<realtimeBar reqId=16777217, time=1497609050, open=164.33, high=164.33, low=164.33, close=164.33, volume=51, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609055, open=164.33, high=164.34, low=164.33, close=164.34, volume=243, wap=164.34, count=5>
2017-06-16T12:31:03.092756: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 687
<realtimeBar reqId=16777217, time=1497609060, open=164.33, high=164.33, low=164.33, close=164.33, volume=160, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497609065, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609070, open=164.33, high=164.33, low=164.33, close=164.33, volume=86, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609075, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609080, open=164.34, high=164.34, low=164.34, close=164.34, volume=33, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609085, open=164.33, high=164.33, low=164.33, close=164.33, volume=48, wap=164.33, count=6>
<currentTime time=1497609095>
<realtimeBar reqId=16777217, time=1497609090, open=164.33, high=164.33, low=164.33, close=164.33, volume=24, wap=164.33, count=5>
<realtimeBar reqId=16777217, time=1497609095, open=164.33, high=164.33, low=164.33, close=164.33, volume=43, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609100, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609105, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609110, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609115, open=164.33, high=164.33, low=164.33, close=164.33, volume=54, wap=164.33, count=3>
2017-06-16T12:32:03.096349: Frompre: 0 data0 2017-06-16T10:32:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 288
Second run: The previously resampled bars are now backfilled bars:
2017-06-16T12:33:44.813167: Frompre: 0 data0 2017-06-16T10:28:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
2017-06-16T12:33:44.814699: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 290
2017-06-16T12:33:44.816210: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 708
2017-06-16T12:33:44.817950: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 448
My conclusion: First rtbar is not taken into account
After checking ibdata.py
source code and debug data I think the problem arises when datetime for a rtbar
is equal to the just emitted resampled bar. I seems those minor changes solves the problem.
Test again:
2017-06-16T13:27:02.523913: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
<realtimeBar reqId=16777217, time=1497612420, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612425, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612430, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612435, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612440, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612445, open=164.34, high=164.34, low=164.34, close=164.34, volume=18, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612450, open=164.33, high=164.34, low=164.33, close=164.34, volume=11, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497612455, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612460, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<currentTime time=1497612466>
<realtimeBar reqId=16777217, time=1497612465, open=164.34, high=164.34, low=164.34, close=164.34, volume=349, wap=164.34, count=42>
<realtimeBar reqId=16777217, time=1497612470, open=164.34, high=164.34, low=164.34, close=164.34, volume=152, wap=164.34, count=17>
<realtimeBar reqId=16777217, time=1497612475, open=164.34, high=164.34, low=164.34, close=164.34, volume=20, wap=164.34, count=5>
2017-06-16T13:28:02.429118: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
<realtimeBar reqId=16777217, time=1497612480, open=164.34, high=164.34, low=164.34, close=164.34, volume=61, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612485, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612490, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612495, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612500, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612505, open=164.34, high=164.34, low=164.34, close=164.34, volume=4, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497612510, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<realtimeBar reqId=16777217, time=1497612515, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612520, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<currentTime time=1497612526>
<realtimeBar reqId=16777217, time=1497612525, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612530, open=164.34, high=164.35, low=164.34, close=164.34, volume=256, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612535, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:29:02.331917: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
<realtimeBar reqId=16777217, time=1497612540, open=164.35, high=164.35, low=164.35, close=164.35, volume=2, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497612545, open=164.35, high=164.35, low=164.35, close=164.35, volume=352, wap=164.35, count=25>
<realtimeBar reqId=16777217, time=1497612550, open=164.35, high=164.35, low=164.35, close=164.35, volume=39, wap=164.35, count=6>
<realtimeBar reqId=16777217, time=1497612555, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612560, open=164.35, high=164.35, low=164.35, close=164.35, volume=10, wap=164.35, count=5>
<realtimeBar reqId=16777217, time=1497612565, open=164.35, high=164.35, low=164.35, close=164.35, volume=36, wap=164.35, count=4>
<realtimeBar reqId=16777217, time=1497612570, open=164.35, high=164.35, low=164.35, close=164.35, volume=143, wap=164.35, count=12>
<realtimeBar reqId=16777217, time=1497612575, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612580, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<currentTime time=1497612586>
<realtimeBar reqId=16777217, time=1497612585, open=164.35, high=164.35, low=164.35, close=164.35, volume=3, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497612590, open=164.34, high=164.34, low=164.34, close=164.34, volume=240, wap=164.34, count=6>
<realtimeBar reqId=16777217, time=1497612595, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:30:02.806358: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
<realtimeBar reqId=16777217, time=1497612600, open=164.35, high=164.35, low=164.34, close=164.34, volume=15, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612605, open=164.35, high=164.35, low=164.35, close=164.35, volume=182, wap=164.35, count=32>
<realtimeBar reqId=16777217, time=1497612610, open=164.35, high=164.35, low=164.35, close=164.35, volume=203, wap=164.35, count=17>
<realtimeBar reqId=16777217, time=1497612615, open=164.36, high=164.36, low=164.36, close=164.36, volume=8, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612620, open=164.36, high=164.37, low=164.36, close=164.37, volume=372, wap=164.36, count=23>
<realtimeBar reqId=16777217, time=1497612625, open=164.36, high=164.37, low=164.36, close=164.37, volume=347, wap=164.37, count=27>
<realtimeBar reqId=16777217, time=1497612630, open=164.37, high=164.37, low=164.37, close=164.37, volume=27, wap=164.37, count=6>
<realtimeBar reqId=16777217, time=1497612635, open=164.37, high=164.37, low=164.36, close=164.37, volume=88, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612640, open=164.36, high=164.36, low=164.36, close=164.36, volume=7, wap=164.36, count=1>
<currentTime time=1497612646>
<realtimeBar reqId=16777217, time=1497612645, open=164.36, high=164.36, low=164.36, close=164.36, volume=20, wap=164.36, count=1>
<realtimeBar reqId=16777217, time=1497612650, open=164.37, high=164.37, low=164.37, close=164.37, volume=2, wap=164.37, count=2>
<realtimeBar reqId=16777217, time=1497612655, open=164.36, high=164.36, low=164.36, close=164.36, volume=14, wap=164.36, count=1>
2017-06-16T13:31:03.093858: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285
Second run: The previously resampled bars are now backfilled bars:
2017-06-16T13:32:02.206769: Frompre: 0 data0 2017-06-16T11:26:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
2017-06-16T13:32:02.208600: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
2017-06-16T13:32:02.210083: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
2017-06-16T13:32:02.211503: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
2017-06-16T13:32:02.212988: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285
2017-06-16T13:32:02.219781: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.360000 high 164.370000 low 164.360000 close 164.370000 vol 285
Now in my opinion data seems to be correct.
Thanks for your time