Backtrader Community

borodiliz

Thanks for your quick reply @backtrader

You can't claim backtrader is perfect, but I can claim backtrader is ... awesome!

I can understand the complexity using tickString, so let's go with RealTimeBars data:

Because I do not want bars as be delivered as much in real-time as possible and I perfer to "wait" a while I've set the following (qcheck=2 , timeoffset=True and rtbar=True):

    ibstore = bt.stores.IBStore(
                                host=args.host, port=args.port,
                                clientId=args.clientId, timeoffset=True,
                                reconnect=args.reconnect, timeout=args.timeout,
                                notifyall=args.notifyall, _debug=args.debug)

    cerebro.setbroker(ibstore.getbroker())

    data0 = ibstore.getdata(dataname=args.data0,
                            backfill_start=True,
                            timeframe=bt.TimeFrame.Minutes,
                            compression=1,
                            name='d0',
                            qcheck=2,
                            rtbar=True)

First run: check resampled data with debug mode enabled:

...........
<realtimeBar reqId=16777217, time=1497608880, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608885, open=164.35, high=164.35, low=164.35, close=164.35, volume=4, wap=164.35, count=3>
<realtimeBar reqId=16777217, time=1497608890, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608895, open=164.35, high=164.35, low=164.34, close=164.34, volume=9, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608900, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608905, open=164.34, high=164.34, low=164.34, close=164.34, volume=205, wap=164.34, count=2>
<currentTime time=1497608915>
<realtimeBar reqId=16777217, time=1497608910, open=164.34, high=164.34, low=164.34, close=164.34, volume=133, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497608915, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497608920, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608925, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608930, open=164.35, high=164.35, low=164.35, close=164.35, volume=79, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608935, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
2017-06-16T12:29:03.091121: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
<realtimeBar reqId=16777217, time=1497608940, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608945, open=164.35, high=164.35, low=164.35, close=164.35, volume=30, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608950, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497608955, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608960, open=164.35, high=164.35, low=164.34, close=164.34, volume=13, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497608965, open=164.34, high=164.34, low=164.34, close=164.34, volume=15, wap=164.34, count=1>
<currentTime time=1497608975>
<realtimeBar reqId=16777217, time=1497608970, open=164.34, high=164.34, low=164.34, close=164.34, volume=13, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497608975, open=164.35, high=164.35, low=164.35, close=164.35, volume=1, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497608980, open=164.35, high=164.35, low=164.35, close=164.35, volume=13, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497608985, open=164.35, high=164.36, low=164.35, close=164.36, volume=48, wap=164.35, count=15>
<realtimeBar reqId=16777217, time=1497608990, open=164.35, high=164.35, low=164.35, close=164.35, volume=129, wap=164.35, count=19>
<realtimeBar reqId=16777217, time=1497608995, open=164.35, high=164.35, low=164.35, close=164.35, volume=26, wap=164.35, count=2>
2017-06-16T12:30:03.100416: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 289
<realtimeBar reqId=16777217, time=1497609000, open=164.34, high=164.34, low=164.34, close=164.34, volume=21, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609005, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609010, open=164.34, high=164.34, low=164.34, close=164.34, volume=104, wap=164.34, count=9>
<realtimeBar reqId=16777217, time=1497609015, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609020, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609025, open=164.33, high=164.33, low=164.33, close=164.33, volume=1, wap=164.33, count=1>
<currentTime time=1497609035>
<realtimeBar reqId=16777217, time=1497609030, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609035, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609040, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497609045, open=164.33, high=164.33, low=164.33, close=164.33, volume=286, wap=164.33, count=24>
<realtimeBar reqId=16777217, time=1497609050, open=164.33, high=164.33, low=164.33, close=164.33, volume=51, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609055, open=164.33, high=164.34, low=164.33, close=164.34, volume=243, wap=164.34, count=5>
2017-06-16T12:31:03.092756: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 687
<realtimeBar reqId=16777217, time=1497609060, open=164.33, high=164.33, low=164.33, close=164.33, volume=160, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497609065, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609070, open=164.33, high=164.33, low=164.33, close=164.33, volume=86, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609075, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609080, open=164.34, high=164.34, low=164.34, close=164.34, volume=33, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497609085, open=164.33, high=164.33, low=164.33, close=164.33, volume=48, wap=164.33, count=6>
<currentTime time=1497609095>
<realtimeBar reqId=16777217, time=1497609090, open=164.33, high=164.33, low=164.33, close=164.33, volume=24, wap=164.33, count=5>
<realtimeBar reqId=16777217, time=1497609095, open=164.33, high=164.33, low=164.33, close=164.33, volume=43, wap=164.33, count=6>
<realtimeBar reqId=16777217, time=1497609100, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609105, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609110, open=164.33, high=164.33, low=164.33, close=164.33, volume=0, wap=164.33, count=0>
<realtimeBar reqId=16777217, time=1497609115, open=164.33, high=164.33, low=164.33, close=164.33, volume=54, wap=164.33, count=3>
2017-06-16T12:32:03.096349: Frompre: 0 data0 2017-06-16T10:32:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 288

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T12:33:44.813167: Frompre: 0 data0 2017-06-16T10:28:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.350000 vol 434
2017-06-16T12:33:44.814699: Frompre: 0 data0 2017-06-16T10:29:00.000000 open 164.350000 high 164.360000 low 164.340000 close 164.350000 vol 290
2017-06-16T12:33:44.816210: Frompre: 0 data0 2017-06-16T10:30:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 708
2017-06-16T12:33:44.817950: Frompre: 0 data0 2017-06-16T10:31:00.000000 open 164.330000 high 164.340000 low 164.330000 close 164.330000 vol 448

My conclusion: First rtbar is not taken into account

After checking ibdata.py source code and debug data I think the problem arises when datetime for a rtbar is equal to the just emitted resampled bar. I seems those minor changes solves the problem.

Test again:

2017-06-16T13:27:02.523913: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
<realtimeBar reqId=16777217, time=1497612420, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612425, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612430, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612435, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612440, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612445, open=164.34, high=164.34, low=164.34, close=164.34, volume=18, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612450, open=164.33, high=164.34, low=164.33, close=164.34, volume=11, wap=164.33, count=2>
<realtimeBar reqId=16777217, time=1497612455, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612460, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<currentTime time=1497612466>
<realtimeBar reqId=16777217, time=1497612465, open=164.34, high=164.34, low=164.34, close=164.34, volume=349, wap=164.34, count=42>
<realtimeBar reqId=16777217, time=1497612470, open=164.34, high=164.34, low=164.34, close=164.34, volume=152, wap=164.34, count=17>
<realtimeBar reqId=16777217, time=1497612475, open=164.34, high=164.34, low=164.34, close=164.34, volume=20, wap=164.34, count=5>
2017-06-16T13:28:02.429118: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
<realtimeBar reqId=16777217, time=1497612480, open=164.34, high=164.34, low=164.34, close=164.34, volume=61, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612485, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612490, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
<realtimeBar reqId=16777217, time=1497612495, open=164.34, high=164.34, low=164.34, close=164.34, volume=1, wap=164.34, count=1>
<realtimeBar reqId=16777217, time=1497612500, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612505, open=164.34, high=164.34, low=164.34, close=164.34, volume=4, wap=164.34, count=4>
<realtimeBar reqId=16777217, time=1497612510, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<realtimeBar reqId=16777217, time=1497612515, open=164.34, high=164.34, low=164.34, close=164.34, volume=2, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612520, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=3>
<currentTime time=1497612526>
<realtimeBar reqId=16777217, time=1497612525, open=164.34, high=164.34, low=164.34, close=164.34, volume=3, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612530, open=164.34, high=164.35, low=164.34, close=164.34, volume=256, wap=164.34, count=10>
<realtimeBar reqId=16777217, time=1497612535, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:29:02.331917: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
<realtimeBar reqId=16777217, time=1497612540, open=164.35, high=164.35, low=164.35, close=164.35, volume=2, wap=164.35, count=2>
<realtimeBar reqId=16777217, time=1497612545, open=164.35, high=164.35, low=164.35, close=164.35, volume=352, wap=164.35, count=25>
<realtimeBar reqId=16777217, time=1497612550, open=164.35, high=164.35, low=164.35, close=164.35, volume=39, wap=164.35, count=6>
<realtimeBar reqId=16777217, time=1497612555, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612560, open=164.35, high=164.35, low=164.35, close=164.35, volume=10, wap=164.35, count=5>
<realtimeBar reqId=16777217, time=1497612565, open=164.35, high=164.35, low=164.35, close=164.35, volume=36, wap=164.35, count=4>
<realtimeBar reqId=16777217, time=1497612570, open=164.35, high=164.35, low=164.35, close=164.35, volume=143, wap=164.35, count=12>
<realtimeBar reqId=16777217, time=1497612575, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<realtimeBar reqId=16777217, time=1497612580, open=164.35, high=164.35, low=164.35, close=164.35, volume=0, wap=164.35, count=0>
<currentTime time=1497612586>
<realtimeBar reqId=16777217, time=1497612585, open=164.35, high=164.35, low=164.35, close=164.35, volume=3, wap=164.35, count=1>
<realtimeBar reqId=16777217, time=1497612590, open=164.34, high=164.34, low=164.34, close=164.34, volume=240, wap=164.34, count=6>
<realtimeBar reqId=16777217, time=1497612595, open=164.34, high=164.34, low=164.34, close=164.34, volume=0, wap=164.34, count=0>
2017-06-16T13:30:02.806358: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
<realtimeBar reqId=16777217, time=1497612600, open=164.35, high=164.35, low=164.34, close=164.34, volume=15, wap=164.34, count=2>
<realtimeBar reqId=16777217, time=1497612605, open=164.35, high=164.35, low=164.35, close=164.35, volume=182, wap=164.35, count=32>
<realtimeBar reqId=16777217, time=1497612610, open=164.35, high=164.35, low=164.35, close=164.35, volume=203, wap=164.35, count=17>
<realtimeBar reqId=16777217, time=1497612615, open=164.36, high=164.36, low=164.36, close=164.36, volume=8, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612620, open=164.36, high=164.37, low=164.36, close=164.37, volume=372, wap=164.36, count=23>
<realtimeBar reqId=16777217, time=1497612625, open=164.36, high=164.37, low=164.36, close=164.37, volume=347, wap=164.37, count=27>
<realtimeBar reqId=16777217, time=1497612630, open=164.37, high=164.37, low=164.37, close=164.37, volume=27, wap=164.37, count=6>
<realtimeBar reqId=16777217, time=1497612635, open=164.37, high=164.37, low=164.36, close=164.37, volume=88, wap=164.36, count=4>
<realtimeBar reqId=16777217, time=1497612640, open=164.36, high=164.36, low=164.36, close=164.36, volume=7, wap=164.36, count=1>
<currentTime time=1497612646>
<realtimeBar reqId=16777217, time=1497612645, open=164.36, high=164.36, low=164.36, close=164.36, volume=20, wap=164.36, count=1>
<realtimeBar reqId=16777217, time=1497612650, open=164.37, high=164.37, low=164.37, close=164.37, volume=2, wap=164.37, count=2>
<realtimeBar reqId=16777217, time=1497612655, open=164.36, high=164.36, low=164.36, close=164.36, volume=14, wap=164.36, count=1>
2017-06-16T13:31:03.093858: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T13:32:02.206769: Frompre: 0 data0 2017-06-16T11:26:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 15
2017-06-16T13:32:02.208600: Frompre: 0 data0 2017-06-16T11:27:00.000000 open 164.340000 high 164.340000 low 164.330000 close 164.340000 vol 553
2017-06-16T13:32:02.210083: Frompre: 0 data0 2017-06-16T11:28:00.000000 open 164.340000 high 164.350000 low 164.340000 close 164.340000 vol 338
2017-06-16T13:32:02.211503: Frompre: 0 data0 2017-06-16T11:29:00.000000 open 164.350000 high 164.350000 low 164.340000 close 164.340000 vol 825
2017-06-16T13:32:02.212988: Frompre: 0 data0 2017-06-16T11:30:00.000000 open 164.350000 high 164.370000 low 164.340000 close 164.360000 vol 1285
2017-06-16T13:32:02.219781: Frompre: 0 data0 2017-06-16T11:31:00.000000 open 164.360000 high 164.370000 low 164.360000 close 164.370000 vol 285

Now in my opinion data seems to be correct.

Thanks for your time

borodiliz

Hi all,

I'm doing some tests with backtrader + IB + live data feeds and I'm getting some differences between resampled data and backfilled data.

Initially my goal is simple: A log with exactly the same bars/data than the TWS chart shows.

Given the following code:

class Test(bt.Strategy):

    ######################
    #
    ######################
    def notify_data(self, data, status, * args, ** kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), * args)


    ######################
    #
    ######################
    def notify_store(self, msg, * args, ** kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    ######################
    #
    ######################
    def __init__(self):
        ''''''
    ######################
    #
    ######################
    def prenext(self):
        ''''''
        #self.next(frompre=True)



    ######################
    #
    ######################
    def log(self, txt):
        now = datetime.datetime.now()
        print('%s:  %s' % (
              now.strftime(dtfmt), txt))

    def next (self, frompre=False):

        self.log("Frompre: %d data0 %s open %f high %f low %f close %f vol %d" %
                 (frompre,
                 self.data0.datetime.datetime(0).strftime(dtfmt),
                 self.data0.open[0],
                 self.data0.high[0],
                 self.data0.low[0],
                 self.data0.close[0],
                 self.data0.volume[0]))

def parse_args():
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

    parser.add_argument('--notifyall',
                        required=False, action='store_true',
                        help='Notify all messages to strategy as store notifs')

    parser.add_argument('--debug',
                        required=False, action='store_true',
                        help='Display all info received form IB')


    parser.add_argument('--host', default='127.0.0.1',
                        required=False, action='store',
                        help='Host for the Interactive Brokers TWS Connection')

    parser.add_argument('--port', default=4002, type=int,
                        required=False, action='store',
                        help='Port for the Interactive Brokers TWS Connection')


    parser.add_argument('--clientId', default=None, type=int,
                        required=False, action='store',
                        help='Client Id to connect to TWS (default: random)')

    parser.add_argument('--no-timeoffset',
                        required=False, action='store_true',
                        help=('Do not Use TWS/System time offset for non '
                        'timestamped prices and to align resampling'))

    parser.add_argument('--reconnect', default=3, type=int,
                        required=False, action='store',
                        help='Number of recconnection attempts to TWS')

    parser.add_argument('--timeout', default=3.0, type=float,
                        required=False, action='store',
                        help='Timeout between reconnection attempts to TWS')

    parser.add_argument('--data0', default=None,
                        required=False, action='store',
                        help='data 0 into the system')

    return parser.parse_args()

if __name__ == '__main__':

    dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
    now = datetime.datetime.now();
    print ("Now is %s" % (now.strftime(dtfmt)))


    args = parse_args()

    # Create a cerebro entity
    cerebro = bt.Cerebro(exactbars=True, runonce=False, preload=False)


    ibstore = bt.stores.IBStore(
                                host=args.host, port=args.port,
                                clientId=args.clientId, timeoffset=not args.no_timeoffset,
                                reconnect=args.reconnect, timeout=args.timeout,
                                notifyall=args.notifyall, _debug=args.debug)


    cerebro.setbroker(ibstore.getbroker())



    data0 = ibstore.getdata(dataname=args.data0,
                            backfill_start=True,
                            timeframe=bt.TimeFrame.Minutes,
                            compression=1,
                            name='d0',
                            qcheck=20)

    # New resampler
    cerebro.resampledata(
                         data0,
                         timeframe=bt.TimeFrame.Minutes,
                         compression=1)

    # Add a strategy

    cerebro.addstrategy(Test)

    cerebro.run()

First run: backfilled data + resampled data:
resampled_vs_backfilled.py --port 4002 --clientId 15 --data0 GBL-201709-DTB

2017-06-16T09:48:19.251639: Frompre: 0 data0 2017-06-16T07:44:00.000000 open 164.41 high 164.41 low 164.34 close 164.35 vol 2712
2017-06-16T09:48:19.257446: Frompre: 0 data0 2017-06-16T07:45:00.000000 open 164.35 high 164.36 low 164.33 close 164.35 vol 1530
2017-06-16T09:48:19.259503: Frompre: 0 data0 2017-06-16T07:46:00.000000 open 164.34 high 164.35 low 164.32 close 164.34 vol 1330
2017-06-16T09:48:19.261535: Frompre: 0 data0 2017-06-16T07:47:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 409
2017-06-16T09:48:19.263397: Frompre: 0 data0 2017-06-16T07:48:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 165
***** DATA NOTIF: LIVE
2017-06-16T09:49:02.211599: Frompre: 0 data0 2017-06-16T07:49:00.000000 open 164.35 high 164.39 low 164.35 close 164.38 vol 798
2017-06-16T09:50:03.389798: Frompre: 0 data0 2017-06-16T07:50:00.000000 open 164.38 high 164.38 low 164.36 close 164.37 vol 538
2017-06-16T09:51:02.751163: Frompre: 0 data0 2017-06-16T07:51:00.000000 open 164.37 high 164.38 low 164.36 close 164.37 vol 780
2017-06-16T09:52:04.579715: Frompre: 0 data0 2017-06-16T07:52:00.000000 open 164.36 high 164.38 low 164.36 close 164.37 vol 747
2017-06-16T09:53:02.244969: Frompre: 0 data0 2017-06-16T07:53:00.000000 open 164.37 high 164.37 low 164.36 close 164.36 vol 246
2017-06-16T09:54:00.565664: Frompre: 0 data0 2017-06-16T07:54:00.000000 open 164.36 high 164.37 low 164.34 close 164.36 vol 896
2017-06-16T09:55:11.430551: Frompre: 0 data0 2017-06-16T07:55:00.000000 open 164.35 high 164.36 low 164.34 close 164.34 vol 414
2017-06-16T09:56:04.272170: Frompre: 0 data0 2017-06-16T07:56:00.000000 open 164.34 high 164.34 low 164.31 close 164.32 vol 2116
2017-06-16T09:57:01.123514: Frompre: 0 data0 2017-06-16T07:57:00.000000 open 164.33 high 164.35 low 164.33 close 164.35 vol 931

Second run: The previously resampled bars are now backfilled bars:

2017-06-16T09:57:59.194470: Frompre: 0 data0 2017-06-16T07:44:00.000000 open 164.41 high 164.41 low 164.34 close 164.35 vol 2712
2017-06-16T09:57:59.196351: Frompre: 0 data0 2017-06-16T07:45:00.000000 open 164.35 high 164.36 low 164.33 close 164.35 vol 1530
2017-06-16T09:57:59.201949: Frompre: 0 data0 2017-06-16T07:46:00.000000 open 164.34 high 164.35 low 164.32 close 164.34 vol 1330
2017-06-16T09:57:59.204162: Frompre: 0 data0 2017-06-16T07:47:00.000000 open 164.34 high 164.35 low 164.34 close 164.35 vol 409
2017-06-16T09:57:59.206197: Frompre: 0 data0 2017-06-16T07:48:00.000000 open 164.34 high 164.39 low 164.34 close 164.38 vol 961
2017-06-16T09:57:59.207977: Frompre: 0 data0 2017-06-16T07:49:00.000000 open 164.38 high 164.38 low 164.36 close 164.37 vol 540
2017-06-16T09:57:59.210999: Frompre: 0 data0 2017-06-16T07:50:00.000000 open 164.37 high 164.38 low 164.36 close 164.37 vol 780
2017-06-16T09:57:59.215024: Frompre: 0 data0 2017-06-16T07:51:00.000000 open 164.37 high 164.38 low 164.35 close 164.37 vol 747
2017-06-16T09:57:59.216797: Frompre: 0 data0 2017-06-16T07:52:00.000000 open 164.37 high 164.37 low 164.36 close 164.36 vol 246
2017-06-16T09:57:59.218651: Frompre: 0 data0 2017-06-16T07:53:00.000000 open 164.36 high 164.37 low 164.34 close 164.36 vol 890
2017-06-16T09:57:59.220877: Frompre: 0 data0 2017-06-16T07:54:00.000000 open 164.36 high 164.36 low 164.34 close 164.34 vol 415
2017-06-16T09:57:59.223913: Frompre: 0 data0 2017-06-16T07:55:00.000000 open 164.34 high 164.34 low 164.31 close 164.33 vol 2114
2017-06-16T09:57:59.229085: Frompre: 0 data0 2017-06-16T07:56:00.000000 open 164.33 high 164.35 low 164.32 close 164.35 vol 934
2017-06-16T09:57:59.230990: Frompre: 0 data0 2017-06-16T07:57:00.000000 open 164.35 high 164.35 low 164.33 close 164.35 vol 427

Problems detected:

1.- Some bars are different. e.g: the bar with 211X volume

resampled data -> close 164.32 vol 2116
backfilled data -> close 164.33 vol 2114

2.- Different timestamps, e.g: the bar with volume 246:

resampled data time-> 07:53
backfilled data time -> 07:52

I've tried increasing the qcheck up-to 20 without success

Thanks in advance.

borodiliz

Thanks @backtrader. That will cover most cases. Maybe the SAREXT indicator should also be added?

In my case I'm working with a very large datasheet so to avoid performance issues I've coded something like:

    def next(self):
        self.psar0._lookback = min(100, len(self.data0))

so I have a "dynamic" _lookback from 1 to 100

And why 100?... newbie here! just by observation, it seems to be a good value to calculate a real ParabolicSAR value.

borodiliz

For those interested I've found a temporally solution:

                self.psar0 = bt.talib.SAR(self.datas[0].high, self.datas[0].low)
                self.psar0._lookback = None

As I understand setting _lookback to None on a TA-Lib indicator we're forcing backtrader to pass TA-Lib len(self) i.e: "as much data as possible"

I'm not sure if this is the right solution and what performance issues could arise when working with very large data feeds... but it seems to work. Now the 60-minute TA-Lib ParabolicSAR indicator seems OK on all scenarios (resampled data and runonce=False)

borodiliz

Thanks for your reply @backtrader

Just focusing on using TA-Lib and why the 60-minute re-sampled data is different to the 60-minute non resampled data I've done some tests:

psar-talib-vs-backtrader.py --adddata --data data/60m.txt --compression 60 --strat talib=True --cerebro runonce=True --plot

psar-talib-vs-backtrader.py --adddata --data data/60m.txt --compression 60 --strat talib=True --cerebro runonce=False --plot

And it make sense for me, because as I understand:

If runonce=True all data is passed to TA-Lib (Using talib.py:once() ), so TA-Lib keeps the status for the parabolic acceleration
If runonce=False backtrader only pass 2 values to TA-Lib (Using talib.py:next() ), so TA-Lib can't keep the status

So, my question:

Is there any way to "force" the number of datas (i.e: the narrays array on talib.py:next() ) backtrader should pass to TA-Lib ? I've tried the following without success:

self.psar0 = bt.talib.SAR(self.datas[0].high, self.datas[0].low, timeperiod=30)

My idea is to set timeperiod to maximum acceleration / acceleration = 0.2 / 0.02= 10 periods (default)

borodiliz

Continuing with my PSAR tests now I have problems using TA-Lib

Given this test https://github.com/borodiliz/backtrader/tree/psar-talib/samples/psar-talib

In which there are two CSV files with the same information but in two different timeframes:

5-minutes
60-minutes

I've done the following tests:

Testing the built-in PSAR indicator

data0 => 60-minute. No resampling.

psar-talib-vs-backtrader.py --adddata --data data/60m.txt --compression 60 --plot --strat talib=False

data0 => 5-minute . data1 => resample data0 to 60-minute:

psar-talib-vs-backtrader.py --adddata --data data/5m.txt --compression 5 --addresample60m --plot --strat talib=False

OK! The problem previously explained before on this thread is solved now, the 60-minute data is the same on both tests

Testing the TA-Lib PSAR indicator

Now same tests but using TA-Lib

data0 => 60-minute. No resampling.

psar-talib-vs-backtrader.py --adddata --data data/60m.txt --compression 60 --plot --strat talib=True

psar-talib-vs-backtrader.py --adddata --data data/5m.txt --compression 5 --addresample60m --plot --strat talib=True

data0 => 5-minute . data1 => resample data0 to 60-minute:

Wops!, Why are so different the 60-minute PSAR calculations when resampling?

Another question: Even without data resampling, PSAR results are different on the PSAR built-in to TA-Lib. Any idea?

Thanks again for your time

borodiliz

I realized that it is only in the development branch. Sometimes I sync this branch and test my code.
Should we contact you if we detect any problems on this branch?

borodiliz

Hi @backtrader,

This code has stopped working for me after commit db80f8472fc1aeb05d84c5b52b86412e3f89632

pypy bc-break.py --adddata1m -> OK
pypy bc-break.py --addresample10m -> OK
pypy bc-break.py --addresample15m -> OK
pypy bc-break.py --adddata1m --addresample10m --addresample15m -> FAIL

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)


import argparse
import datetime

import backtrader as bt


class St(bt.Strategy):
    params = dict(

                  talib=False
                  )

    def __init__(self):
        ''''''
       

    def next(self):
        txt = []
        txt.append('{:04d}'.format(len(self)))
        txt.append('{:04d}'.format(len(self.data0)))
        txt.append(self.data0.datetime.datetime())
        txt.append('{:.2f}'.format(self.data0.close[0]))

        try:
            if len(self.datas[1]):
                txt.append('{:04d}'.format(len(self.data1)))
                txt.append(self.data1.datetime.datetime())
                txt.append('{:.2f}'.format(self.data1.close[0]))

        except IndexError:
            False #Nothing to do

        try:
            if len(self.datas[2]):
                txt.append('{:04d}'.format(len(self.data2)))
                txt.append(self.data2.datetime.datetime())
                txt.append('{:.2f}'.format(self.data2.close[0]))
        except IndexError:
            False #Nothing to do

        print(','.join(str(x) for x in txt))


def runstrat(args=None):
    args = parse_args(args)

    cerebro = bt.Cerebro()

    # Data feed kwargs
    kwargs = dict(
                  timeframe=bt.TimeFrame.Minutes,
                  compression=5,
                  )

    # Parse from/to-date
    dtfmt, tmfmt = '%Y-%m-%d', 'T%H:%M:%S'
    for a, d in ((getattr(args, x), x) for x in ['fromdate', 'todate']):
        if a:
            strpfmt = dtfmt + tmfmt * ('T' in a)
            kwargs[d] = datetime.datetime.strptime(a, strpfmt)

    # Data feed
    data0 = bt.feeds.BacktraderCSVData(dataname=args.data0, ** kwargs)

    if args.adddata1m:
        cerebro.adddata(data0)

    if args.addresample10m:
        cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=10)

    if args.addresample15m:
        cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=15)

    # Broker
    cerebro.broker = bt.brokers.BackBroker(** eval('dict(' + args.broker + ')'))

    # Sizer
    cerebro.addsizer(bt.sizers.FixedSize, ** eval('dict(' + args.sizer + ')'))

    # Strategy
    cerebro.addstrategy(St, ** eval('dict(' + args.strat + ')'))

    # Execute
    cerebro.run(** eval('dict(' + args.cerebro + ')'))

    if args.plot:  # Plot if requested to
        cerebro.plot(** eval('dict(' + args.plot + ')'))


def parse_args(pargs=None):
    parser = argparse.ArgumentParser(
                                     formatter_class=argparse.ArgumentDefaultsHelpFormatter,
                                     description=(
                                     'Sample Skeleton'
                                     )
                                     )

    parser.add_argument('--data0', default='../../datas//2006-min-005.txt',
                        required=False, help='Data to read in')

    # Defaults for dates
    parser.add_argument('--fromdate', required=False, default='',
                        help='Date[time] in YYYY-MM-DD[THH:MM:SS] format')

    parser.add_argument('--todate', required=False, default='',
                        help='Date[time] in YYYY-MM-DD[THH:MM:SS] format')

    parser.add_argument('--cerebro', required=False, default='',
                        metavar='kwargs', help='kwargs in key=value format')

    parser.add_argument('--broker', required=False, default='',
                        metavar='kwargs', help='kwargs in key=value format')

    parser.add_argument('--sizer', required=False, default='',
                        metavar='kwargs', help='kwargs in key=value format')

    parser.add_argument('--strat', required=False, default='',
                        metavar='kwargs', help='kwargs in key=value format')

    parser.add_argument('--plot', required=False, default='',
                        nargs='?', const='{}',
                        metavar='kwargs', help='kwargs in key=value format')

    parser.add_argument('--adddata1m', action='store_true', required=False,
                        help='Add 1 minute data')

    parser.add_argument('--addresample10m', action='store_true', required=False,
                        help='Add resample for 10 minute data')

    parser.add_argument('--addresample15m', action='store_true', required=False,
                        help='Add resample for 15 minute data')


    return parser.parse_args(pargs)


if __name__ == '__main__':
    runstrat()

It seems a problem while combining adddata and resampledata

Thanks in advance!

borodiliz

@backtrader said in PSAR indicator and multiple timeframes:

This commit in the development branch addresses the initial status:

https://github.com/mementum/backtrader/commit/f097b7b316b9552e33a5069cee69625b32d52f1c

Now it works like a charm. Thanks for solve this @backtrader !

borodiliz

Just for testing purposes I've modified your psar-indraday.py to accept the following parameters:

--adddata1m to add the 1 minute data (no data feed is added by default)
--addresample10m to add a resample for 10 mins
--addresample15m to add a resample for 15 mins

I've executed the following commands:

psar-intraday.py --adddata1m
psar-intraday.py --addresample10m
psar-intraday.py --addresample15m
psar-intraday.py --adddata1m addresample10m addresample15m

On the following branches:

master@mementum/backtrader
development@mementum/backtrader
PSAR_multiple_timeframes@borodiliz/backtrader

Please check all results here (only first 30 lines)

Keep in mind I'm not sure what are the correct values for even a unique data feed, I'm only checking for differences between all the implementations!.

My conclusions:

PSAR calculations for a single data has changed with your new implementation (diffs between master to development)
PSAR calculations for a single data has NOT changed from master to borodiliz (diffs between master to borodiliz)
PSAR calculations for a multiple data feeds seems "weird" in development@mementum/backtrader

Thanks again for your time!

borodiliz

@borodiliz

Latest posts made by borodiliz

Testing the built-in PSAR indicator

Testing the TA-Lib PSAR indicator