For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Unable to get closing daily values for SPY



  • While running today's test, one observation I am making that may be significant is that for the system that is using backfilled SPY daily data, it is reporting the last bar seen from 2-2 and not the truly last bar in the file from 2-3...

    -- 3165 2017-02-06 15:54:00
    Data0,ES,1015,2017-02-06 09:54:00,2288.25
    Data1,SPY,2150,2017-02-02 16:00:00,227.77
    Data2,ES-daily,0001,2017-02-06 09:15:00,2289.25
    


  • @backtrader
    After running the test for the three configurations I have described, only the system version with both backfill_from= and the resample of ES-minutes to ES-daily produced any bars in strategy.next().

    Here is the output. By the end of the day, it still had not produced a bar for SPY newer that the 2-2 date.

    -- 3470 2017-02-06 20:59:00
    Data0,ES,1320,2017-02-06 14:59:00,2287.5
    Data1,SPY,2150,2017-02-02 16:00:00,227.77
    Data2,ES-daily,0001,2017-02-06 09:15:00,2289.25
    -- 3471 2017-02-06 21:00:00
    Data0,ES,1321,2017-02-06 15:00:00,2288.25
    Data1,SPY,2150,2017-02-02 16:00:00,227.77
    Data2,ES-daily,0001,2017-02-06 09:15:00,2289.25
    -- 3472 2017-02-06 21:01:00
    Data0,ES,1322,2017-02-06 15:01:00,2287.75
    Data1,SPY,2150,2017-02-02 16:00:00,227.77
    Data2,ES-daily,0001,2017-02-06 09:15:00,2289.25
    

    Another possibly interesting observation: I also ran the basic test script that you provided with a slight change to start the strategy with cerebro.run(runonce=False, tradehistory=True, exactbars=0) in case that might be having some impact.

    The output drops one of the bars for ES as LIVE status becomes active. Not sure if that could have some impact here. Note below that the bar for 10:46 is missing. Not sure if that is just a result of the print, or if there is a real problem.

    -- 1315 2017-02-06 16:44:00
    Data0,ES,1065,2017-02-06 10:44:00,2289.0 
    Data1,SPY,0251,2017-02-03 16:00:00,229.41
    -- 1316 2017-02-06 16:45:00
    Data0,ES,1066,2017-02-06 10:45:00,2288.75
    Data1,SPY,0251,2017-02-03 16:00:00,229.41
    ***** DATA NOTIF: ES LIVE
    -- 1317 2017-02-06 16:47:00
    Data0,ES,1067,2017-02-06 10:47:00,2289.25
    Data1,SPY,0251,2017-02-03 16:00:00,229.41
    -- 1318 2017-02-06 16:48:00
    Data0,ES,1068,2017-02-06 10:48:00,2288.75
    Data1,SPY,0251,2017-02-03 16:00:00,229.41
    

    The other two versions of this same script never printed a bar from the code in strategy.next(). These were the versions that did not backfill_from and the other that did not backfill_from and did not resample the data0 ES feed to Daily. Both versions of the script did indicate that they had LIVE data for both SPY and ES.



  • @backtrader Reran this today and am happy to say that it appears I have captured the closing SPY daily bar today.
    I have the following issues I still need to resolve:

    • Need to come up with a clever way to exclude non-signaling datas from passing through the next() logic. I found recently that matching on data._name was not working for some reason. Will investigate that more.
    • It is not clear that I am getting the value I expect on certain indicators despite the fact that I am only passing the SPY daily bars as input to the indicators. They appear to somehow be using finer timeframe to calculate. Must verify.

    One more observations from this learning process:

    • This system I am porting to BT requires 3 years of daily historical data. It appears that without a backfill source, I am not receiving that much historical data from IB when run without backfill_from=. While I get LIVE status for both instruments from IB, I never see a bar print in .next() if I am not using backfill_from. I could use some help to track that further upstream but curious if there should be some other indication of when the indicators are "warmed up" and "LIVE".

  • administrators

    It seems that Interactive Brokers has lately relaxed the limitations on historical data downloads (according to the footnote in the new documentation on GitHub) and it may therefore make sense to increase the limits in the IBStore, which make reference to the limits previously documented by Interactive Brokers

    For daily data the maximum duration was set to 1 Y (*1 year), but you should be able to get up to 5 years.

    Following your Pull Requests you seem to use indicators with very long periods (252 + 1 in DV2). If the trading year which is getting back to you from Interactive Brokers with the standard fill doesn't have 253 bars, the warm-up period is for sure not yet achieved.



  • Actually, this system that I have been working to port to BT needs 3 years of historical daily data. That, coupled with the fact that I don't trust the quality of the IB data, the backfill_from= feature of BT is very much a requirement and solves this issue of waiting for things to "warm up" when restarting the system.

    Anxious to get this longer timeframe system working so I can start working on some shorter term systems. :-)



  • I don't use IB data for back tests either, I use iqfeed data stored in postgres db and i pull it as pandas dataframe and use it for back tests with bt, it would be nice if we can get direct integration with iqfeed, that way we dont have to use IB data for live trading just use ib as broker.



  • @jaikumarm I agree regarding the need for IQFeed integration. I may take a stab at this in the near future with some help from @backtrader.

    FWIW, I am storing historical data from IQ in an InfluxDB. Cool thing about InfluxDB, beyond its speed, is the ability to request the stored minute data in any timeframe. With the help of @backtrader, I have a working feed plugin to connect to InfluxDB for backfill and backtesting data. Check it out here: https://community.backtrader.com/topic/180/json-data-sources/14

    warning: probably still in early stages.



  • @backtrader

    Noticing today that the system is showing only daily data up through 2/8/2017 for SPY, even though the backfill_from data provides the bar for 2/9/2017.

    Not the first time I have seen this. It was correct yesterday. This is consistent in any instances of the same system that I start today...

    -- 3444 2017-02-10 20:31:00
    Data0,ES,1292,2017-02-10 14:31:00,2313.0
    Data1,SPY,2152,2017-02-08 16:00:00,229.24
    Data2,ES-daily,0001,2017-02-10 09:15:00,2309.25
    


  • Possibly related to this, one of the other mysteries I am now working on is that the indicator values at end of day do not agree with what is output when running a backtest. The indicator values in backtest, running from the static data I am backfilling from, are correct. But when running live, they are not correct. I'm suspicious that we are dropping some bars in this calculation as I discovered when using .resampledata() with the InflxuDB feed. When using .adddata() sources, the values are correct.

    I've put some prints in one of my indicators and will be watching that on Monday unless I can figure out something using demo data.


  • administrators

    As mentioned in the InfluxDB ... a couple of lines to fully adjust the eos (if none is given) have been added.

    Checking backwards in this thread:

        # Parse CSI SPY data file
        staticdata0path = os.path.join(modpath, '../../datas/SPY-1D.csv')
        staticdata0 = btfeed.GenericCSVData(dataname=staticdata0path,
                                            dtformat=('%Y-%m-%d'),
                                            datetime=0,
                                            high=1,
                                            low=2,
                                            open=3,
                                            close=4,
                                            volume=5,
                                            openinterest=6)
    

    It seems that no sessionend is being set for the backfill_from data source. You probably want to set it. It might also be the reason to only sometimes see the last of the bar of the backfilling.



  • Thanks Dro, I will give this a try today.


Log in to reply
 

});