datas time indication off and .resampledata()

RandyT

I put a pairs trading system in test mode today and am seeing the following output on datas time which is off.
I also had a very hard time getting IB to allow this system to start which may be an IB data issue, but still raises concern. I had to try to start the system no less than 20 times before it finally got past DELAYED state for data. Not until I went to the IB server to tail the log did it actually start. The solution was just to keep retrying. I've seen this in the other system I am testing.

First, I am creating separate data feed requests for 2 symbols, one in bt.timeframe.Days and the second pair of feeds in bt.timeframe.Minutes. The use case is that I want to provide daily timeframe data to the indicators in the system and I want to be able to execute a trade some time before close of the day.

Here is what I am seeing, with one of the instruments reporting UTC time vs the exchange time it was configured for.

-- 0671 2017-03-09 14:55:00
Data0	SSO	0251	2017-03-08 16:00:00	85.07
Data1	SDS	0251	2017-03-08 16:00:00	13.46
Data2	SSO-trade	0051	2017-03-09 14:55:00	85.2
Data3	SDS-trade	0169	2017-03-09 09:55:00	13.43
-- 0672 2017-03-09 14:56:00
Data0	SSO	0251	2017-03-08 16:00:00	85.07
Data1	SDS	0251	2017-03-08 16:00:00	13.46
Data2	SSO-trade	0052	2017-03-09 14:56:00	85.13
Data3	SDS-trade	0170	2017-03-09 09:56:00	13.44
-- 0673 2017-03-09 14:57:00
Data0	SSO	0251	2017-03-08 16:00:00	85.07
Data1	SDS	0251	2017-03-08 16:00:00	13.46
Data2	SSO-trade	0053	2017-03-09 14:57:00	85.12
Data3	SDS-trade	0171	2017-03-09 09:57:00	13.44

Here is code used to output that data:

            print('-- %004d' % len(self), str(self.datetime.datetime()))
            for i, d in enumerate(di for di in self.datas if len(di)):
                out = ['Data%d' % i, d._name, '%004d' % len(d),
                       str(d.datetime.datetime()), str(d.close[0])]
                print('\t'.join(out))

Here are the data feed configurations:

        data0 = ibstore.getdata(dataname=args.live_data0,
                                timeframe=bt.TimeFrame.Days,
                                compression=1,
                                sessionstart=dt.time(9, 30),
                                sessionend=dt.time(16, 0),
                                tz='EST5CDT')
        cerebro.resampledata(data0, name=dname0,
                             timeframe=bt.TimeFrame.Days,
                             compression=1)

        data1 = ibstore.getdata(dataname=args.live_data1,
                                timeframe=bt.TimeFrame.Days,
                                compression=1,
                                sessionstart=dt.time(9, 30),
                                sessionend=dt.time(16, 0),
                                tz='EST5EDT')
        cerebro.resampledata(data1, name=dname1,
                             timeframe=bt.TimeFrame.Days,
                             compression=1)

        data2 = ibstore.getdata(dataname=args.live_data0,
                                timeframe=bt.TimeFrame.Minutes,
                                compression=1,
                                sessionstart=dt.time(9, 30),
                                sessionend=dt.time(16, 0),
                                tz='EST5CDT')
        cerebro.resampledata(data2, name=(dname0 + '-trade'),
                             timeframe=bt.TimeFrame.Minutes,
                             compression=1)

        data3 = ibstore.getdata(dataname=args.live_data1,
                                timeframe=bt.TimeFrame.Minutes,
                                compression=1,
                                sessionstart=dt.time(9, 30),
                                sessionend=dt.time(16, 0),
                                tz='EST5EDT')
        cerebro.resampledata(data3, name=(dname1 + '-trade'),
                             timeframe=bt.TimeFrame.Minutes,
                             compression=1)

Some follow-on questions:

Where are we at with ability to .resampledata() from IB to larger timeframes? It was my most recent understanding that we are better off creating separate feed requests rather than resample bt.timeframe.Minutes to bt.timeframe.Days in this case.
It also would probably be more appropriate for me to be using .replaydata() here as I realize as I type this that I am not going to get a 15:55 bar on the daily feeds, nor will I have updated indicator data to look at at 15:55 without .replaydata(). Is .replaydata() working? I seem to remember that either we should not mix .replaydata() with use of .resampledata() or some question if it was working.

Thanks once again for your patience with me @backtrader . :-)

backtrader

@RandyT said in datas time indication off and .resampledata():

Where are we at with ability to .resampledata() from IB to larger timeframes? It was my most recent understanding that we are better off creating separate feed requests rather than resample bt.timeframe.Minutes to bt.timeframe.Days in this case.

Not very far away. Too many other things in the backlog. It is still better, due to how backfilling is handled, to create a separate data feed.

It also would probably be more appropriate for me to be using .replaydata() here as I realize as I type this that I am not going to get a 15:55 bar on the daily feeds, nor will I have updated indicator data to look at at 15:55 without .replaydata(). Is .replaydata() working? I seem to remember that either we should not mix .replaydata() with use of .resampledata() or some question if it was working.

The too many corner cases that the core code tries to catch is the key. Your combinations are stretching the limits of what actually can be done within reasonable limits with live feeds. Synchronizing frozen data feeds was easy and an easy route was chosen for it, rather than putting a more complex mechanism in place.

It's a matter of trial and error. Unfortunately.

A rework of the platform (probably with bcolz although still undecided)

Thanks once again for your patience ...

The real patience is on your side, by checking the unchecked and things that even been committed and not yet tested (you find the bugs even before they are there)