YahooFinance Data Feeds


  • administrators

    Release 1.9.50.117 tries to improve the new non-API (is it an API?) YahooFinance data.

    1. By skipping lines with null values

    2. By properly handling what seems to be the new format for adjusted prices.

    3. By updating the data samples and any code samples which needed it

    Topic 2 needs some reference data and a explanation. The repository contained for example these first lines for the data sample orcl-1995-2014.txt

    Date,Open,High,Low,Close,Volume,Adj Close
    1995-01-03,44.12,44.38,42.87,42.87,36301200,2.00
    1995-01-04,43.00,43.50,42.37,43.25,46051600,2.01
    

    Where 2.00 is the adjusted close and the other values are the recorded historical values at the time of trading. One would usually prefer to work with adjusted prices using the adjusted close as a reference for the scaling (including the volume)

    The new downloaded data (no processing)

    Date,Open,High,Low,Close,Adj Close,Volume
    1995-01-03,2.179012,2.191358,2.117284,42.875004,2.117284,36301200
    1995-01-04,2.123457,2.148148,2.092592,43.250004,2.135803,46051600
    

    And suddenly the prices offered are the adjusted prices. but with errors. Adjustedclose and historical close are actually swapped. The offered volume is the same as above, so one can wonder if the volume was right before and unadjusted or is right now and adjusted.

    In any case the code takes the new conditions in account (v7) and reverts to the original column ordering and behavior if version='' (empty), if old files have to be read.

    The Quickstart samples should now work as expected.


  • administrators

    This post in Quant Stack Exchange, https://quant.stackexchange.com/questions/942/any-known-bugs-with-yahoo-finance-adjusted-close-data, may indicate that the new format may be more accurate, because in the original Yahoo was providing adjusted prices which had been rounded down to two decimals (2.00 in the first line of the previous samples), with no such rounding now.



  • Now only adjusted prices are provided? No original prices?


  • administrators

    The feed seems to have/has changed but the functionality hasn't. If the parameter adjclose=True (the default), the adjusted prices are provided.

    • If it's a file with the original format, version='' (something which isn't v7), the adjusted prices are calculated using the provided adjusted close

    • If it's a file/download with the new format, version='v7' (the current default), the adjusted prices are provided directly from the feed.

    If you change to adjclose=False, the non-adjusted prices are provided directly from the feed (old format) or calculated backwards (new format)

    As a reference the note in the docs: Yahoo data feed: Docs - Yahoo Data Feed Notes



  • @backtrader said in YahooFinance Data Feeds:

    1.9.50.117
    By skipping lines with null values
    By properly handling what seems to be the new format for adjusted prices.

    By skipping lines with null values [..] to properly handling adjusted price.
    Crashes when Volume is always == null. Could be draw anyway.




  • administrators

    @cgi1 said in YahooFinance Data Feeds:

    @backtrader said in YahooFinance Data Feeds:

    1.9.50.117
    By skipping lines with null values
    By properly handling what seems to be the new format for adjusted prices.

    By skipping lines with null values [..] to properly handling adjusted price.
    Crashes when Volume is always == null. Could be draw anyway.

    This means that some lines (a sample line would help) contains prices which can be parse, but the string null that where the volume should be, or is it something else?


  • administrators

    After further review of the pull-request which is done also against code which is no longer there (due to the changes introduced for the v7 format), a sample of what's exactly meant with null is really needed.

    • Is this check for the old format, the new format or for both?
    • How does this null look like in an actual data line?


  • Thanks for the reply.

    This means that some lines (a sample line would help) contains prices which can be parse, but the string null that where the volume should be, or is it something else?

    Exactly: So sometimes the datafeed provider (in this case Yahoo on 2017-06-08) does not provider any Volume for some of the symbols. E.g. EURHKD=X:

    Date,Open,High,Low,Close,Adj Close,Volume
    2003-12-01,9.347397,9.278463,9.339722,9.290833,9.290833,null
    2003-12-02,9.287244,9.278581,9.393217,9.380319,9.380319,null
    
    2017-06-06,8.783572,8.734257,8.789911,8.784152,8.784152,null
    2017-06-08,8.780444,8.732892,8.782912,8.748856,8.748856,0 <<<<< Date of download!
    

    Some symbols are including Volume==0. E.g. EURUSD=X

    Date,Open,High,Low,Close,Adj Close,Volume
    2003-12-01,1.203398,1.194401,1.204007,1.196501,1.196501,0
    2003-12-02,1.196101,1.194600,1.210903,1.208897,1.208897,0
    
    2005-09-20,null,null,null,null,null,null <<<< Some dates are does not even contain any data
    
    2017-06-06,1.127053,1.120523,1.127993,1.127142,1.127142,0
    2017-06-08,1.126507,1.127142,1.119946,1.122083,1.122083,0 <<<<< Date of download!
    

    Meanwhile on the same date a lot symbols (e.g. IFX.DE) contain Volume and works as expected.

    Date,Open,High,Low,Close,Adj Close,Volume
    2000-03-14,71.400002,80.000000,71.400002,78.900002,78.900002,9579700
    2000-03-15,76.900002,76.949997,70.800003,73.040001,73.040001,5855009
    

    Hope this helps. If you need further information please let me know.


  • administrators

    @cgi1 said in YahooFinance Data Feeds:

    Exactly: So sometimes the datafeed provider (in this case Yahoo on 2017-06-08) does not provider any Volume for some of the symbols. E.g. EURHKD=X:

    To be expected. See below

    @cgi1 said in YahooFinance Data Feeds:

    Some symbols are including Volume==0. E.g. EURUSD=X

    This is to be expected. Yahoo is giving you the conversion rate. It's not a broker/ECN, has no volume and cannot know what the volume around the world is.

    In this case also the adjusted close and the non-adjusted close are bound to always be the same: a currency pair has no splits or dividends.

    @cgi1 said in YahooFinance Data Feeds:

    2003-12-01,9.347397,9.278463,9.339722,9.290833,9.290833,null

    This requires exception catching in the conversion to float. The pull-request wouldn't have done this.


  • administrators

    Release 1.9.56.122 contains an exception for volume fields which are not convertible to float.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.