How to get only end of day data when live trading

jaikumarm

I have simple test strategy based on the IB live trading blog post, it works as expected in back test but in live trading for the current day, it returns a bar as of now and does not wait till eod to give a complete bar

0001, 2016-12-01T18:00:00.000000, 2185.5, 2191.75, 2179.25, 2186.75, 9369.0, 1, nan
0002, 2016-12-04T18:00:00.000000, 2179.0, 2203.75, 2174.25, 2198.875, 23728.0, 1, nan
0003, 2016-12-05T18:00:00.000000, 2199.75, 2206.5, 2195.0, 2204.75, 39177.0, 1, nan
0004, 2016-12-06T18:00:00.000000, 2205.75, 2235.75, 2201.5, 2231.375, 111509.0, 1, nan
0005, 2016-12-07T18:00:00.000000, 2231.25, 2246.0, 2227.75, 2242.375, 392866.0, 1, nan
0006, 2016-12-08T18:00:00.000000, 2241.5, 2255.75, 2239.75, 2254.75, 1013273.0, 1, nan
0007, 2016-12-11T18:00:00.000000, 2255.25, 2264.75, 2246.75, 2250.375, 1287827.0, 1, nan
0008, 2016-12-12T18:00:00.000000, 2250.25, 2273.0, 2247.75, 2267.75, 1556179.0, 1, nan
0009, 2016-12-13T18:00:00.000000, 2268.0, 2272.5, 2243.0, 2252.0, 1794750.0, 1, nan
0010, 2016-12-14T18:00:00.000000, 2251.75, 2267.75, 2247.25, 2258.375, 1703803.0, 1, 0.0
0011, 2016-12-15T18:00:00.000000, 2259.5, 2264.0, 2249.5, 2255.25, 1378336.0, 1, 1.0
0012, 2016-12-18T18:00:00.000000, 2256.0, 2263.75, 2254.0, 2260.125, 920006.0, 1, 1.0
0013, 2016-12-19T18:00:00.000000, 2260.25, 2269.5, 2260.0, 2266.5, 806149.0, 1, 1.0
0014, 2016-12-20T18:00:00.000000, 2267.25, 2268.0, 2258.0, 2260.375, 648460.0, 1, 1.0
0015, 2016-12-21T18:00:00.000000, 2258.25, 2261.0, 2251.5, 2258.75, 831630.0, 1, 0.0
0016, 2016-12-22T18:00:00.000000, 2257.5, 2261.0, 2254.25, 2260.0, 391441.0, 1, 0.0
0017, 2016-12-26T18:00:00.000000, 2258.75, 2269.5, 2257.75, 2260.875, 489710.0, 1, 0.0
0018, 2016-12-27T18:00:00.000000, 2261.25, 2267.5, 2243.5, 2245.25, 974692.0, 1, 1.0
0019, 2016-12-28T18:00:00.000000, 2245.5, 2250.0, 2239.5, 2245.125, 882916.0, 1, 0.0
0020, 2016-12-29T18:00:00.000000, 2246.25, 2252.75, 2228.0, 2236.25, 1251896.0, 1, 1.0
0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 0.0
0022, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 1, 1.0
0023, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 1, 1.0
0024, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 1, 0.0
0025, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 1, 0.0
0026, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 1, 0.0
0027, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 1, 0.0
0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0

I kind of have tried various combinations of replay and different timeframes and rtbar settings, not able to find the right one.

the strategy idea is to get bar on close, then calculate a bunch of indicators and then use that "forecast" next close, based on the forecast place an market order that should fill at open and rinse and repeat every day flipping between long/short

def runstrategy():

    cerebro = bt.Cerebro()

    storekwargs = dict(
        host='127.0.0.1', port=4003,
        clientId=110,
        timeoffset=not None,
        reconnect=3,
        timeout=3.0,
        notifyall=None,
        _debug=None
    )

    ibstore = bt.stores.IBStore(**storekwargs)
    broker = ibstore.getbroker()
    cerebro.setbroker(broker)

    timeframe = bt.TimeFrame.Days
    datacomp = 1

    IBDataFactory = ibstore.getdata

    datakwargs = dict(
        timeframe=timeframe,
        compression=datacomp,
        historical=None,
        fromdate=datetime.datetime(2016, 12, 1),
        rtbar=timeframe,
        qcheck=0.5,
        what=None,
        backfill_start=not None,
        backfill=not None,
        latethrough=None,
        tz=None
    )

    rekwargs = dict(
        timeframe=timeframe,
        compression=datacomp,
        bar2edge=not None,
        adjbartime=not None,
        rightedge=not None,
        takelate=not None,
    )

    cerebro.resampledata(dataname=data0, **rekwargs)

    cerebro.addstrategy(LongShortStrategyESDaily,
                        smaperiod=10,
                        trade=True,
                        exectype=bt.bt.Order.Market,
                        stopafter=None,
                        valid=valid,
                        cancel=0,
                        donotsell=None
                       )

    cerebro.addsizer(FixedReverser)

    cerebro.run()

backtrader

It seems your problem is:

it returns a bar as of now and does not wait till eod to give a complete bar

Where I guess with as of now you mean, right after delivering the last historical data bar.

But there isn't enough information to really understand where the problem is. See for example:

IBDataFactory = ibstore.getdata

But there is no single istance of IBDataFactory in your sample above. The only reference of something being done with the data:

cerebro.resampledata(dataname=data0, **rekwargs)

Where data0 has magically appeared ...

Addtionally from your log:

0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0

The last historical data is the day 2017-01-11 and the 1st day returned live seems to be 2017-01-12 which is 1 day later. It is unclear what your timezone is but this is apparently being printed out at:

You could add details, like your own timezone, the timezone of the asset, what the localtime is when the 1st live data bar is being printed out.

jaikumarm

thanks for the quick response. Sorry in my over-eagerness to clear our some comments in the code when I was composing the posting I deleted the IBDataFactory line.

 data0 = IBDataFactory(dataname='ES-201703-GLOBEX', **datakwargs)

So to answer your queries, Am running this on a desktop in PST, and the asset is ES so its set to EST and I ran the script at 4:09AM PST on 2017-01-13.

An you are right the last historical bar was 2017-01-11 and first live bar was 2017-01-12, but this day was not complete yet, as in the market is still open, so what I got is a valid bar at that point of time with the OHLCV calculated up that point in time. my expectation or goal is to make get the live bar return only when the day is closed.

Here is the whole script
https://gist.github.com/jaikumarm/b9b8124d880d683104971e25a4ab72c2

backtrader

If your own zone is PST then you are 3 hours behind EST (please correct this if wrong, although a quick search seems to confirm it)

See the log

0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0

You get a LIVE notification at 2017-01-13 04:09:06 PST, because that timestamp is being printed in your local time. From the code in your gist at line 77:

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, datetime.datetime.now(), 'DATA NOTIF:', data._getstatusname(status), *args)
        if status == data.LIVE:
            self.counttostop = self.p.stopafter
            self.datastatus = 1

Translated to EST the actual timestamp was: 2017-01-13 07:09:06 PST. That means that the EST has been trading already some hours into Jan 13th (over 7 hours at least)

Data point 0029 has a timestamp with date 2017-01-12, i.e.: from the previous day, which theoretically is already over.

The next complete 1-day bar (to be delivered when the resampling is over) should be that with date 2017-01-13.

Unless the reasoning is very wrong, you are not (yet) getting the las bar which is still being resampled.

jaikumarm

I tried this again today and I see an issue which might explain what is going on.

Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenPositionSize, Forecast
***** 2017-01-16 01:08:41.980178 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** 2017-01-16 01:08:41.980252 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ilhmds>
***** 2017-01-16 01:08:41.980302 DATA NOTIF: DELAYED
***** 2017-01-16 01:08:42.480597 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
0001, 2016-12-01T18:00:00.000000, 2185.5, 2191.75, 2179.25, 2186.75, 9369.0, 1, nan
0002, 2016-12-04T18:00:00.000000, 2179.0, 2203.75, 2174.25, 2198.875, 23728.0, 1, nan
0003, 2016-12-05T18:00:00.000000, 2199.75, 2206.5, 2195.0, 2204.75, 39177.0, 1, nan
0004, 2016-12-06T18:00:00.000000, 2205.75, 2235.75, 2201.5, 2231.375, 111509.0, 1, nan
0005, 2016-12-07T18:00:00.000000, 2231.25, 2246.0, 2227.75, 2242.375, 392866.0, 1, nan
0006, 2016-12-08T18:00:00.000000, 2241.5, 2255.75, 2239.75, 2254.75, 1013273.0, 1, nan
0007, 2016-12-11T18:00:00.000000, 2255.25, 2264.75, 2246.75, 2250.375, 1287827.0, 1, nan
0008, 2016-12-12T18:00:00.000000, 2250.25, 2273.0, 2247.75, 2267.75, 1556179.0, 1, nan
0009, 2016-12-13T18:00:00.000000, 2268.0, 2272.5, 2243.0, 2252.0, 1794750.0, 1, nan
0010, 2016-12-14T18:00:00.000000, 2251.75, 2267.75, 2247.25, 2258.375, 1703803.0, 1, 0.0
0011, 2016-12-15T18:00:00.000000, 2259.5, 2264.0, 2249.5, 2255.25, 1378336.0, 1, 1.0
0012, 2016-12-18T18:00:00.000000, 2256.0, 2263.75, 2254.0, 2260.125, 920006.0, 1, 1.0
0013, 2016-12-19T18:00:00.000000, 2260.25, 2269.5, 2260.0, 2266.5, 806149.0, 1, 1.0
0014, 2016-12-20T18:00:00.000000, 2267.25, 2268.0, 2258.0, 2260.375, 648460.0, 1, 1.0
0015, 2016-12-21T18:00:00.000000, 2258.25, 2261.0, 2251.5, 2258.75, 831630.0, 1, 0.0
0016, 2016-12-22T18:00:00.000000, 2257.5, 2261.0, 2254.25, 2260.0, 391441.0, 1, 1.0
0017, 2016-12-26T18:00:00.000000, 2258.75, 2269.5, 2257.75, 2260.875, 489710.0, 1, 1.0
0018, 2016-12-27T18:00:00.000000, 2261.25, 2267.5, 2243.5, 2245.25, 974692.0, 1, 1.0
0019, 2016-12-28T18:00:00.000000, 2245.5, 2250.0, 2239.5, 2245.125, 882916.0, 1, 0.0
0020, 2016-12-29T18:00:00.000000, 2246.25, 2252.75, 2228.0, 2236.25, 1251896.0, 1, 1.0
0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0
0022, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 1, 0.0
0023, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 1, 1.0
0024, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 1, 1.0
0025, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 1, 0.0
0026, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 1, 0.0
0027, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 1, 0.0
0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 1, 1.0
0029, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 1, 1.0
***** 2017-01-16 01:08:43.937119 DATA NOTIF: LIVE
0030, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2266.25, 42215.0, 1, 1.0

first clue of the issue is the following bar, its dated 2017-01-02 that's the CME/GLOBEX holiday
http://www.cmegroup.com/tools-information/holiday-calendar/files/2016-holiday-calendars.zip

0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0

And from the pricing and volume numbers it looks like it actually the data from 2017-01-03, and then each date seems a day behind. which seems to make sense also the last bar 0029 as well as that was for 2017-01-13 which was last Friday.
if not for this logic we are essentially missing data for the all the Friday's but have data for Sunday's.

I got this from the IB gateway logs and it looks like IB data dumps are dated correctly.

20161230-2246.25-2252.75-2228.00-2236.25-1251896-2237.80-false-124726-20170103-2240.75-2259.50-2239.50-2252.625-1786475-2248.3125-false-157661-20170104-2252.75-2267.25-2251.00-2264.25-1383745-2262.2875-false-130558-20170105-2264.50-2266.00-2254.00-2264.25-1307080-2260.825-false-131039-20170106-2264.25-2277.00-2258.25-2271.50-1541103-2269.425-false-133538-20170109-2271.25-2275.25-2263.50-2265.00-1019553-2266.80-false-103071-20170110-2264.50-2274.00-2259.50-2263.75-1299832-2266.1375-false-120978-20170111-2263.75-2271.75-2255.00-2270.625-1727931-2264.00-false-173838-20170112-2270.50-2270.50-2248.50-2263.375-1745819-2259.0375-false-166470-20170113-2264.50-2273.50-2262.75-2272.50-1182909-2269.725-false-112790-20170116-2270.25-2270.50-2265.00-2266.25-42215-2267.25-false-7503-

So it looks like there is two issues, the one day roll-back for data for this instrument, and my original issue for bar 0030 2017-01-15 which is today's data which is not close yet, as you mentioned the market has been open for the last few hours but has not closed yet, so the daily bar should not be delivered yet.

hope you can shed some light on this, also let me know if you need me to add any further debug logs or print statements that might help.

jaikumarm

wasn't sure if I made some errors with my modifications to the ibtest.py sample. here is the run from ibtest.py unmodified from master branch

python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01
Server Version: 76
TWS Time at connection:20170116 10:12:19 UTC
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
***** DATA NOTIF: DELAYED
Data0, 0001, 736332.0, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
Data0, 0002, 736333.0, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
Data0, 0003, 736334.0, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
Data0, 0004, 736335.0, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
Data0, 0005, 736338.0, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
Data0, 0006, 736339.0, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
Data0, 0007, 736340.0, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
Data0, 0008, 736341.0, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
Data0, 0009, 736342.0, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
***** DATA NOTIF: LIVE
Data0, 0010, 736345.0, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2267.0, 49079.0, 0, 2267.45

One additional data point am running the IB gateway on a docker container which is set to UTC by default.

am going to build a new container which is set to EST and run everything inside that and see if any of these TZ differences are making an impact. thanks for your help so far in troubleshooting this.

jaikumarm

Okay looks like I can't read instructions :(.. I tried setting the timezone in the ibtest.py script to PST and voila! all the problems solved. all the datetime stamps are aligned and the last daily bar is waiting for the day to complete, I hope!

python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01 --timezone PST
Server Version: 76
TWS Time at connection:20170116 10:34:35 UTC
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
***** DATA NOTIF: DELAYED
Data0, 0001, 736332.0, 2017-01-03T00:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
Data0, 0002, 736333.0, 2017-01-04T00:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
Data0, 0003, 736334.0, 2017-01-05T00:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
Data0, 0004, 736335.0, 2017-01-06T00:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
Data0, 0005, 736338.0, 2017-01-09T00:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
Data0, 0006, 736339.0, 2017-01-10T00:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
Data0, 0007, 736340.0, 2017-01-11T00:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
Data0, 0008, 736341.0, 2017-01-12T00:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
Data0, 0009, 736342.0, 2017-01-13T00:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
***** DATA NOTIF: LIVE

PS: I fixed it in my modified script and it works there as well. thanks Daniel for your help and for backtrader!!