For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Unable to get closing daily values for SPY



  • @backtrader I switched to master branch when updating when I saw your commits last night. So I was running master branch today.

    I am in US/Mountain time 'MST7MDT'.

    I guess I am still confused as to why I would see a bar on daily timeframe here before it is complete. My goal is to come up with something that could run 24x7 without restarts. Comparing on date and time suggest to me that I would need to restart on daily basis to initialize the date.

    Feels like I am overthinking the concept of getting the close bar values for the day. What am I missing?



  • Just to confirm that I am running the master branch, I reupdated from the master branch repo and am running again. Load goes up to 100% as soon as the data feeds hit LIVE status.

    0_1485219936934_upload-12baabf6-a4c1-42f0-927d-fdb3db24cbb0



  • @backtrader This appears to be a python3 issue. Specifically, I am seeing it on python 3.4.5.
    python 2.7 behaves just fine.


  • administrators

    Something that was being considered after seeing your screenshot with the process names. But a quick code inspection doesn't really show where a difference between Python 2.7 and Python 3.x could actually be.


  • administrators

    Testing repeated with python 3.5.2, one data feed resampling to Days/1 and the other to Minutes/1. Both have hit the LIVE status (they run almost 24x7, because they are EUR.USD and EUR.JPY, and are busy tickers) and at least under Windows 10 the CPU usage percentage is 0% (for sure under 1%) and under Ubuntu 16.04 is consistently ticking between 0.7% and 1.3%



  • Could someone confirm that you can actually get a daily timeframe close on daily data?

    I'm able to capture OHLC at any point in time on minute timeframe data, but I cannot seem so solve this puzzle of getting the value of the closing bar for the RTH on SPY.


  • administrators

    @randy, the entire resampling code is being looked into to achieve this feat. As commented before, the initial use cases for data feeds didn't involve such timeframe synchronizations and with off-line data was a lot simpler to achieve.

    Too many corner cases were initially added to cover resampling even without the presence of the notion of session times and/or calendars.

    The latest release has also added some regression problems in the IBData data feed. These have already been caught and the current work is pointing at making sure that the Days timeframe properly synchronizes in resampling with minor timeframes.

    Originally, and even more since data feeds in backtesting can be paused, the resampled bar would be delivered at the beginning of the next session. Ok, unless like in your case, perfectly valid, in which you need the daily bar promptly delivered to make a decision on another feed which keeps on running.

    Estimations are always there to be broken (see Telas's record of failed-to-meet milestones) but give it an estimate of 48 hours to have it ready.



  • Thanks @backtrader
    I'm very familiar with the open source process so I won't even consider holding you to any schedule. I was mainly wanting to make sure that there is in fact a problem meeting this use case and not simply an issue where I have not figured out how to make the system do it.

    I am waiting patiently and will spend my time on some other things that the group might find of value.


  • administrators

    The major problem is that there were too many goals and ambitious ideas when putting resample in play. Having kept it simpler and constrained to mainstream use cases would probably help today.

    In any case, it does already synchronize ... but there is still something to consider: you are synchronizing assets from different timezones. Because the smaller timeframe (Minutes/1) will push the larger timeframe (Days/1) to output itself at the sessionend edge (rather than having the chance to wait for itself, what would be optimal), the timezone synchronization plays a role.

    Seems doable even within the Tesla-like milestone.


  • administrators

    The code seems ready. Here the output of two test cases.

    Data0: EUR.JPY is resampled to Minutes/1 and Data1: EUR.USD is resampled to Days/1.

    Same Timezone: EST

    Server Version: 76
    TWS Time at connection:20170127 23:52:32 CET
    --------------------------------------------------
    Strategy Created
    --------------------------------------------------
    Timezone from ContractDetails: EST5EDT
    Trading Hours: 20170127:0005-2358;20170128:0005-2358
    Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
    ***** DATA NOTIF: DELAYED
    ***** DATA NOTIF: DELAYED
    -- 0001
    Data0, 0000, 736356.208333, 2017-01-27T00:00:00.000000, 122.585, 122.63, 122.58, 122.62, -1.0, 0.0, nan
    Data1, 0001, 736351.954861, 2017-01-22T17:55:00.000000, 1.0533, 1.0733, 1.044, 1.045, -1.0, 0.0, nan
    -- 0002
    Data0, 0000, 736356.208333, 2017-01-27T00:00:00.000000, 122.585, 122.63, 122.58, 122.62, -1.0, 0.0, nan
    Data1, 0002, 736352.954861, 2017-01-23T17:55:00.000000, 1.0445, 1.0773, 1.0346, 1.0705, -1.0, 0.0, nan
    ...
    -- 1078
    Data0, 1073, 736356.952778, 2017-01-27T17:52:00.000000, 123.325, 123.325, 123.32, 123.325, -1.0, 0.0, 123.293
    Data1, 0005, 736355.954861, 2017-01-26T17:55:00.000000, 1.0562, 1.0592, 1.0347, 1.0423, -1.0, 0.0, nan
    ***** DATA NOTIF: LIVE
    -- 1079
    Data0, 1074, 736356.953472, 2017-01-27T17:53:00.000000, 119.761, 119.761, 119.699, 119.699, 0.0, 0.0, 122.5908
    Data1, 0005, 736355.954861, 2017-01-26T17:55:00.000000, 1.0562, 1.0592, 1.0347, 1.0423, -1.0, 0.0, nan
    -- 1080
    Data0, 1075, 736356.954167, 2017-01-27T17:54:00.000000, 119.7, 119.751, 119.658, 119.751, 0.0, 0.0, 121.89
    Data1, 0005, 736355.954861, 2017-01-26T17:55:00.000000, 1.0562, 1.0592, 1.0347, 1.0423, -1.0, 0.0, nan
    -- 1081
    Data0, 1076, 736356.954861, 2017-01-27T17:55:00.000000, 119.749, 119.752, 119.723, 119.748, 0.0, 0.0, 121.1696
    Data1, 0006, 736356.954861, 2017-01-27T17:55:00.000000, 1.0426, 1.0803, 1.0346, 1.07537, -1.0, 0.0, nan
    -- 1082
    Data0, 1077, 736356.955556, 2017-01-27T17:56:00.000000, 119.724, 119.732, 119.702, 119.702, 0.0, 0.0, 120.445
    Data1, 0006, 736356.954861, 2017-01-27T17:55:00.000000, 1.0426, 1.0803, 1.0346, 1.07537, -1.0, 0.0, nan
    -- 1083
    Data0, 1078, 736356.95625, 2017-01-27T17:57:00.000000, 119.69, 119.72, 119.689, 119.695, 0.0, 0.0, 119.719
    Data1, 0006, 736356.954861, 2017-01-27T17:55:00.000000, 1.0426, 1.0803, 1.0346, 1.07537, -1.0, 0.0, nan
    

    The end of session time has been set to 17:55 EST. Before that time is reached on 2017-01-27, Data1 (daily resampling) remains on the end of session time of the previous day, which is 2017-01-26:17:55 (Point 1080). Data0 is still at 2017-01-27:17:54:00, which indicates that the end of session is one minute away.

    At point 1081 the minute stream reaches 17:55 and so does the daily stream, switching to 2017-01-27:17:55:00

    From that point on, Data0 keeps moving 1 minute at a time and the daily stream, Data1, remains set on the end of session.

    Different Timezones: CST and EST

    The test changes Data0 (minute resampling) to be in the CST timezone and the end of sessions are set to 17:08 in CST and 18:08 in EST

    ...
    -- 1092
    Data0, 1087, 736356.9625, 2017-01-27T17:06:00.000000, 123.425, 123.43, 123.425, 123.43, -1.0, 0.0, 123.417
    Data1, 0005, 736355.963889, 2017-01-26T18:08:00.000000, 1.0562, 1.0592, 1.0347, 1.0423, -1.0, 0.0, nan
    ***** DATA NOTIF: LIVE
    -- 1093
    Data0, 1088, 736356.963194, 2017-01-27T17:07:00.000000, 119.58, 119.593, 119.563, 119.585, 0.0, 0.0, 122.65
    Data1, 0005, 736355.963889, 2017-01-26T18:08:00.000000, 1.0562, 1.0592, 1.0347, 1.0423, -1.0, 0.0, nan
    -- 1094
    Data0, 1089, 736356.963889, 2017-01-27T17:08:00.000000, 119.586, 119.693, 119.586, 119.683, 0.0, 0.0, 121.8996
    Data1, 0006, 736356.963889, 2017-01-27T18:08:00.000000, 1.0426, 1.0803, 1.0346, 1.07307, -1.0, 0.0, nan
    -- 1095
    Data0, 1090, 736356.964583, 2017-01-27T17:09:00.000000, 119.681, 119.716, 119.681, 119.716, 0.0, 0.0, 121.1678
    Data1, 0006, 736356.963889, 2017-01-27T18:08:00.000000, 1.0426, 1.0803, 1.0346, 1.07307, -1.0, 0.0, nan
    

    And without repeating the description from above the synchronized change from 2017-01-26 to 2017-01-27 for the daily resampling happens right at the end of session at xx:08

    The changes will go now into the development branch.

    In any case resampling/replay would probably greately benefit from a rewrite to remove the many artifacts introduced trying to support all things at the same time.



  • @backtrader Awesome work. I'll give it a try over the weekend.



  • This post is deleted!


  • This post is deleted!


  • This post is deleted!


  • This all appears to be working well on the demo feed today. Will give it a proper test on Monday.

    Another related question, I am under the impression that in checking for LIVE data status in strategy.notify_data(), that it is possible to have one of the several feeds I am using return live, which thereby sets the variable to True in my code. I've created the following bit of code. Could someone please confirm or deny my conclusion and would you please offer a more elegant way to do this check in python? My python skills are still growing.

    Thanks

        def notify_data(self, data, status, *args, **kwargs):
            livestatus = 0
    
            print('*' * 3, '%s DATA NOTIF: %s' % (timestamp, data._getstatusname(status)))
    
            for feed in iter(self.datas):
                if feed._laststatus == feed.LIVE:
                    livestatus += 1
    
            if livestatus == len(self.datas):
                self.datastatus = True
    

  • administrators

    Data Feeds will signal the LIVE status when the historical/backfilling buffer has been consumed and the last piece of accumulated live data is going to be delivered.

    If the data feeds are resampled to (very) different timeframes, the signaling will happen at different stages. With 1-day and 1-minute.

    The 1-minute bars are (more or less) from the current day and before they even get delivered the system will deliver the daily bars which make up the entire last year. In this simplified scenario, the 1-minute bars will be delivered after 1-day has signaled LIVE. The last daily bar is that of today is not fully resampled, but it is LIVE because data is getting added to it. At the same time and because the 1-minute historical bars are from the current day, they are delivered until exhausted. At this point in time, the 1-minute stream will also signal LIVE



  • In live market today, I did not get the print for the closing bar. I'm adding a few other debug prints in strategy.next() and will see what I get tomorrow.

    One potentially interesting thing I do see is that IB is giving indication that the data form connection is inactive. Which would suggest to me that we are not requesting data. And not sure what happens that would trigger a request unless it is simply the sessionend time which is currently set to dt.time(16, 0).

    Server Version: 76
    TWS Time at connection:20170130 13:12:23 MST
    Timezone from ContractDetails: ES: CST SPY: EST5EDT
    *** 2017-01-30T13:12:24 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
    *** 2017-01-30T13:12:24 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
    *** 2017-01-30T13:12:24 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
    *** 2017-01-30T13:12:24 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ushmds>
    *** 2017-01-30T13:12:24 DATA NOTIF: ES DELAYED
    *** 2017-01-30T13:12:33 DATA NOTIF: SPY DELAYED
    *** 2017-01-30T13:12:33 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
    *** 2017-01-30T13:12:34 DATA NOTIF: SPY LIVE
    *** 2017-01-30T13:12:37 DATA NOTIF: ES LIVE
    *** 2017-01-30T13:26:18 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ilhmds>
    *** 2017-01-30T13:26:37 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ushmds>
    

    @backtrader, with latest changes, it is still not possible to .replaydata() ? Is that correct?

    Adding some other info, when I killed the IB instance that the system was connected to, there was no message on screen indicating any loss of connection. As if it has timed out or is no longer using the IB data feed.


  • administrators

    This was the regression mentioned above which was corrected with this commit:

    https://github.com/mementum/backtrader/commit/0c3cdae4f209e9fc0d37b646f25dd123f5f8a680

    Since you said you were testing quite early, you may have not yet synchronized to that one.

    Right now the synchronization was attempted with 2 feeds that close at 17:00 EST, namely: EUR.JPY and EUR.USD both from a live data (no demo)

    Server Version: 76
    TWS Time at connection:20170130 22:48:23 CET
    --------------------------------------------------
    Strategy Created
    --------------------------------------------------
    Timezone from ContractDetails: EST5EDT
    Trading Hours: 20170130:1715-1700;20170131:1715-1700
    Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.us>
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
    ***** DATA NOTIF: DELAYED
    ***** DATA NOTIF: DELAYED
    -- 0001
    Data0, 0000, 736358.9270833334, 2017-01-29T17:15:00.000000, 122.9, 123.09, 122.9, 123.06, -1.0, 0.0, nan
    Data1, 0001, 735995.9166666666, 2016-02-01T17:00:00.000000, 1.0831, 1.09125, 1.08145, 1.08875, -1.0, 0.0, nan
    ...
    -- 0259
    Data0, 0000, 736358.9270833334, 2017-01-29T17:15:00.000000, 122.9, 123.09, 122.9, 123.06, -1.0, 0.0, nan
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    ***** DATA NOTIF: LIVE
    -- 0260
    Data0, 0001, 736358.9270833334, 2017-01-29T17:15:00.000000, 122.9, 123.09, 122.9, 123.06, -1.0, 0.0, nan
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    ...
    Data0, 1413, 736359.9076388889, 2017-01-30T16:47:00.000000, 121.685, 121.7, 121.68, 121.68, -1.0, 0.0, 121.699
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    -- 1673
    Data0, 1414, 736359.9083333333, 2017-01-30T16:48:00.000000, 121.68, 121.68, 121.66, 121.665, -1.0, 0.0, 121.68800000000002
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    ***** DATA NOTIF: LIVE
    ***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usfarm.us>
    ***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usfarm.us>
    ***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usfarm>
    ***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usfarm>
    -- 1674
    Data0, 1415, 736359.9090277777, 2017-01-30T16:49:00.000000, 121.665, 121.665, 121.66, 121.66, 0.0, 0.0, 121.67699999999999
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    ...
    -- 1684
    Data0, 1425, 736359.9159722222, 2017-01-30T16:59:00.000000, 121.71, 121.715, 121.69, 121.705, 0.0, 0.0, 121.702
    Data1, 0259, 736356.9166666666, 2017-01-27T17:00:00.000000, 1.0678, 1.0725, 1.0658, 1.06935, -1.0, 0.0, nan
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.us>
    -- 1685
    Data0, 1426, 736359.9166666666, 2017-01-30T17:00:00.000000, 121.71, 121.71, 121.655, 121.675, 0.0, 0.0, 121.69500000000001
    Data1, 0260, 736359.9166666666, 2017-01-30T17:00:00.000000, 1.07115, 1.074, 1.062, 1.06925, -1.0, 0.0, nan
    
    • The informative 210x codes from TWS are normal and happen. The log has traces of 2108 and 2104. In the trace in your post and given it seems the regression correction was missing, it could mean that the 2107 indicates a lack of client request. But 2108 indicates the same and the stream was active. In the new docs IB emphasizes the need not to worry about these codes (they could stop producing them in the 1st place and not make the actual status text based in the second place)

    • Both feeds have output the synchronized resampled bar at 17:00 EST. There was roughly a 0.5 human-measured delay between TWS graying out the elements in the GUI and the client printing out the resampled bars at 17:00

    • The final final values shown some seconds later in TWS are one pip off when compared to the resampled bar. This may be some kind of internal closing auction at the ECN side inside Interactive Brokers, because that price wasn't on the screen the second the values were grayed out.

    • In the sample, the actual end of session (17:00) has been passed to the feeds. This has no real impact on the 1-minute feed, but forces the daily feed to produce the bar when it sees the 1-minute bar (or the real-time clock) ticking at 17:00.

    A few minutes ago there was some extra code pushed for resampling, but this is intended to further enhance the support for the end of session trick with Minute based timeframes, trying to smartly jump over bank holidays and weekends. It has been tested with a 1-Minute with a wrongly configured end of session and jumping from Friday to Monday.

    If you may sync to the latest development branch, all the changes will be with you (pending the order_target_xxx revision, this should be it for the next release)



  • @backtrader thanks again for all the hard work.

    It is really just the 2107 errors that I found interesting. They others are a waste of bits...

    Just an FYI, I am seeing the high python loads again and have replicated it both with Python3 and Python2. Load comes on right as one of the feeds goes to LIVE status. In my case right now, the ES is getting live data, but the SPY is not.

    I tried to attach to the running python process the other day when I was seeing this but did not have much success getting the python backtrace. Since I can now replicate this with Python2, I will try the same to see if I get further. Wanted to give you a heads up on this.



  • @backtrader sorry to report that this is still not providing an end of day closing bar for SPY.

    A couple of observations:

    1. I discovered that the code I wrote above to check for livestatus was preventing me from getting to the time check. This appears to be because all data feeds are not reporting LIVE and specifically, it seems that the SPY datafeed is not LIVE.
    def notify_data(self, data, status, *args, **kwargs):
            livestatus = 0
    
            print('*' * 3, '%s DATA NOTIF: %s' % (timestamp, data._getstatusname(status)))
    
            for feed in iter(self.datas):
                if feed._laststatus == feed.LIVE:
                    livestatus += 1
    
            if livestatus == len(self.datas):
                self.datastatus = True
    
    1. After changing the code to just check if status == data.LIVE in notify_data(), I saw every tick for ES (not just the minute bars) coming through the following time check. This should not have been a surprise, but I guess I also need to somehow ignore the ES data so

    2. Since I don't care about the ES ticks until there is a trade, I added a bit more code to return if the ES ticks are not at the same time I am looking for SPY closing bar. This did prevent looping through evaluation on these ES ticks, but unfortunately, never saw the closing bar for SPY daily.

                if (self.data_spy.datetime.time(tz=EST) < dt.time(16, 00) or
                    self.data_es.datetime.time(tz=CST) < dt.time(15, 00)):
                    return
    

    I am at a bit of a loss as to what to do next. I am going to add some prints to the code to print every timestamp seen through the day tomorrow to see if I can make better sense of this.
    The questions I now have are:

    1. Why am I not seeing a SPY tick in next()?
    2. What do I have to do to ignore ES ticks in next()?

    Any recommendations would be very welcome.

    To prevent you having to look or ask for this, I am including the configuration for the data below:

        # Parse static SPY data file
        bfdata1 = btfeed.GenericCSVData(dataname=bfdata1path,
                                        dtformat=('%Y-%m-%d'), datetime=0, high=1, low=2,
                                        open=3, close=4, volume=5, openinterest=6)
    
            # ES Futures Live data timeframe resampled to 1 Minute
            data0 = ibstore.getdata(dataname=args.live_es,
                                    sessionstart=dt.time(8, 30), sessionend=dt.time(15, 15),
                                    timeframe=bt.TimeFrame.Minutes, compression=1)
            cerebro.resampledata(data0, name='ES', timeframe=bt.TimeFrame.Minutes, compression=1)
    
            # SPY Live data timeframe resampled to 1 Day
            data1 = ibstore.getdata(dataname=args.live_spy, backfill_from=bfdata1,
                                    timeframe=bt.TimeFrame.Days, compression=1,
                                    sessionstart=dt.time(9, 30), sessionend=dt.time(16, 0))
            cerebro.resampledata(data1, name='SPY', timeframe=bt.TimeFrame.Days, compression=1)
    
            cerebro.resampledata(data0, name='ES-daily', timeframe=bt.TimeFrame.Days, compression=1)
    

Log in to reply
 

});