@backtrader I tried restarting bt after running into this issue and that seems to make it work, so am using this as a workaround.
However I think we should add some logic into backtrader to automatically detect and recover from this, as this effectively removes the ability to run any algo for more than a day against IB.
thanks for posting this, will try the PR's over the weekend and let you know my experience.
@RandyT yeah looks like we are on a similar path. I have been looking into akapur's pyiqfeed repo as well. I have a script that pull's IQFeed historical data into a postgres db, it loosely based on this. https://www.quantstart.com/articles/Downloading-Historical-Intraday-US-Equities-From-DTN-IQFeed-with-Python
For now I have different tables for various assets and timeframes, so its getting quite unwieldy, would love to standardize on 1min data and re-sample to higher timeframes on the fly, just havent come around to do it yet.
So if you have something that does it for InfluxDB would love to see it!
maybe a trades chart with up and down arrows for when the trades were entered and exited, I use it to generally eyeball and do a sanity check on when the trades occurred and whether I as a human agree with it :)
for me, the
#1 killer feature that would move bt from test/dev/play-around to production/live is supporting a high quality data feed provider, like iqfeed in US. As IB datafeed is not very accurate or reliable.
#2 is supporting some kind of persistent storage (postgres, influxdb, mysql etc) for both ohlc data and trade/backtest data.
Also it would be remiss if dont mention the #1 killer feature bt has right now which is ease of use, I was able to get an algo up and running with csv in less than 10mins of discovering the package and so far all the issues I have run into are either timezone specifc or something silly with IB :(
I don't use IB data for back tests either, I use iqfeed data stored in postgres db and i pull it as pandas dataframe and use it for back tests with bt, it would be nice if we can get direct integration with iqfeed, that way we dont have to use IB data for live trading just use ib as broker.
Okay looks like I can't read instructions :(.. I tried setting the timezone in the ibtest.py script to PST and voila! all the problems solved. all the datetime stamps are aligned and the last daily bar is waiting for the day to complete, I hope!
python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01 --timezone PST
Server Version: 76
TWS Time at connection:20170116 10:34:35 UTC
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
***** DATA NOTIF: DELAYED
Data0, 0001, 736332.0, 2017-01-03T00:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
Data0, 0002, 736333.0, 2017-01-04T00:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
Data0, 0003, 736334.0, 2017-01-05T00:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
Data0, 0004, 736335.0, 2017-01-06T00:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
Data0, 0005, 736338.0, 2017-01-09T00:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
Data0, 0006, 736339.0, 2017-01-10T00:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
Data0, 0007, 736340.0, 2017-01-11T00:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
Data0, 0008, 736341.0, 2017-01-12T00:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
Data0, 0009, 736342.0, 2017-01-13T00:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
***** DATA NOTIF: LIVE
PS: I fixed it in my modified script and it works there as well. thanks Daniel for your help and for backtrader!!
wasn't sure if I made some errors with my modifications to the ibtest.py sample. here is the run from ibtest.py unmodified from master branch
python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01
Server Version: 76
TWS Time at connection:20170116 10:12:19 UTC
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
***** DATA NOTIF: DELAYED
Data0, 0001, 736332.0, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
Data0, 0002, 736333.0, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
Data0, 0003, 736334.0, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
Data0, 0004, 736335.0, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
Data0, 0005, 736338.0, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
Data0, 0006, 736339.0, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
Data0, 0007, 736340.0, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
Data0, 0008, 736341.0, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
Data0, 0009, 736342.0, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
***** DATA NOTIF: LIVE
Data0, 0010, 736345.0, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2267.0, 49079.0, 0, 2267.45
One additional data point am running the IB gateway on a docker container which is set to UTC by default.
am going to build a new container which is set to EST and run everything inside that and see if any of these TZ differences are making an impact. thanks for your help so far in troubleshooting this.
I tried this again today and I see an issue which might explain what is going on.
Timezone from ContractDetails: America/Belize
Datetime, Open, High, Low, Close, Volume, OpenPositionSize, Forecast
***** 2017-01-16 01:08:41.980178 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** 2017-01-16 01:08:41.980252 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ilhmds>
***** 2017-01-16 01:08:41.980302 DATA NOTIF: DELAYED
***** 2017-01-16 01:08:42.480597 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
0001, 2016-12-01T18:00:00.000000, 2185.5, 2191.75, 2179.25, 2186.75, 9369.0, 1, nan
0002, 2016-12-04T18:00:00.000000, 2179.0, 2203.75, 2174.25, 2198.875, 23728.0, 1, nan
0003, 2016-12-05T18:00:00.000000, 2199.75, 2206.5, 2195.0, 2204.75, 39177.0, 1, nan
0004, 2016-12-06T18:00:00.000000, 2205.75, 2235.75, 2201.5, 2231.375, 111509.0, 1, nan
0005, 2016-12-07T18:00:00.000000, 2231.25, 2246.0, 2227.75, 2242.375, 392866.0, 1, nan
0006, 2016-12-08T18:00:00.000000, 2241.5, 2255.75, 2239.75, 2254.75, 1013273.0, 1, nan
0007, 2016-12-11T18:00:00.000000, 2255.25, 2264.75, 2246.75, 2250.375, 1287827.0, 1, nan
0008, 2016-12-12T18:00:00.000000, 2250.25, 2273.0, 2247.75, 2267.75, 1556179.0, 1, nan
0009, 2016-12-13T18:00:00.000000, 2268.0, 2272.5, 2243.0, 2252.0, 1794750.0, 1, nan
0010, 2016-12-14T18:00:00.000000, 2251.75, 2267.75, 2247.25, 2258.375, 1703803.0, 1, 0.0
0011, 2016-12-15T18:00:00.000000, 2259.5, 2264.0, 2249.5, 2255.25, 1378336.0, 1, 1.0
0012, 2016-12-18T18:00:00.000000, 2256.0, 2263.75, 2254.0, 2260.125, 920006.0, 1, 1.0
0013, 2016-12-19T18:00:00.000000, 2260.25, 2269.5, 2260.0, 2266.5, 806149.0, 1, 1.0
0014, 2016-12-20T18:00:00.000000, 2267.25, 2268.0, 2258.0, 2260.375, 648460.0, 1, 1.0
0015, 2016-12-21T18:00:00.000000, 2258.25, 2261.0, 2251.5, 2258.75, 831630.0, 1, 0.0
0016, 2016-12-22T18:00:00.000000, 2257.5, 2261.0, 2254.25, 2260.0, 391441.0, 1, 1.0
0017, 2016-12-26T18:00:00.000000, 2258.75, 2269.5, 2257.75, 2260.875, 489710.0, 1, 1.0
0018, 2016-12-27T18:00:00.000000, 2261.25, 2267.5, 2243.5, 2245.25, 974692.0, 1, 1.0
0019, 2016-12-28T18:00:00.000000, 2245.5, 2250.0, 2239.5, 2245.125, 882916.0, 1, 0.0
0020, 2016-12-29T18:00:00.000000, 2246.25, 2252.75, 2228.0, 2236.25, 1251896.0, 1, 1.0
0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0
0022, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 1, 0.0
0023, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 1, 1.0
0024, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 1, 1.0
0025, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 1, 0.0
0026, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 1, 0.0
0027, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 1, 0.0
0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 1, 1.0
0029, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 1, 1.0
***** 2017-01-16 01:08:43.937119 DATA NOTIF: LIVE
0030, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2266.25, 42215.0, 1, 1.0
first clue of the issue is the following bar, its dated 2017-01-02 that's the CME/GLOBEX holiday
http://www.cmegroup.com/tools-information/holiday-calendar/files/2016-holiday-calendars.zip
0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0
And from the pricing and volume numbers it looks like it actually the data from 2017-01-03, and then each date seems a day behind. which seems to make sense also the last bar 0029 as well as that was for 2017-01-13 which was last Friday.
if not for this logic we are essentially missing data for the all the Friday's but have data for Sunday's.
I got this from the IB gateway logs and it looks like IB data dumps are dated correctly.
20161230-2246.25-2252.75-2228.00-2236.25-1251896-2237.80-false-124726-20170103-2240.75-2259.50-2239.50-2252.625-1786475-2248.3125-false-157661-20170104-2252.75-2267.25-2251.00-2264.25-1383745-2262.2875-false-130558-20170105-2264.50-2266.00-2254.00-2264.25-1307080-2260.825-false-131039-20170106-2264.25-2277.00-2258.25-2271.50-1541103-2269.425-false-133538-20170109-2271.25-2275.25-2263.50-2265.00-1019553-2266.80-false-103071-20170110-2264.50-2274.00-2259.50-2263.75-1299832-2266.1375-false-120978-20170111-2263.75-2271.75-2255.00-2270.625-1727931-2264.00-false-173838-20170112-2270.50-2270.50-2248.50-2263.375-1745819-2259.0375-false-166470-20170113-2264.50-2273.50-2262.75-2272.50-1182909-2269.725-false-112790-20170116-2270.25-2270.50-2265.00-2266.25-42215-2267.25-false-7503-
So it looks like there is two issues, the one day roll-back for data for this instrument, and my original issue for bar 0030 2017-01-15 which is today's data which is not close yet, as you mentioned the market has been open for the last few hours but has not closed yet, so the daily bar should not be delivered yet.
hope you can shed some light on this, also let me know if you need me to add any further debug logs or print statements that might help.
thanks for the quick response. Sorry in my over-eagerness to clear our some comments in the code when I was composing the posting I deleted the IBDataFactory line.
data0 = IBDataFactory(dataname='ES-201703-GLOBEX', **datakwargs)
So to answer your queries, Am running this on a desktop in PST, and the asset is ES so its set to EST and I ran the script at 4:09AM PST on 2017-01-13.
An you are right the last historical bar was 2017-01-11 and first live bar was 2017-01-12, but this day was not complete yet, as in the market is still open, so what I got is a valid bar at that point of time with the OHLCV calculated up that point in time. my expectation or goal is to make get the live bar return only when the day is closed.
Here is the whole script
https://gist.github.com/jaikumarm/b9b8124d880d683104971e25a4ab72c2