Navigation

    Backtrader Community
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    YahooFinance Data Feeds

    General Discussion
    3
    11
    6335
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      backtrader administrators last edited by

      Release 1.9.50.117 tries to improve the new non-API (is it an API?) YahooFinance data.

      1. By skipping lines with null values

      2. By properly handling what seems to be the new format for adjusted prices.

      3. By updating the data samples and any code samples which needed it

      Topic 2 needs some reference data and a explanation. The repository contained for example these first lines for the data sample orcl-1995-2014.txt

      Date,Open,High,Low,Close,Volume,Adj Close
1995-01-03,44.12,44.38,42.87,42.87,36301200,2.00
1995-01-04,43.00,43.50,42.37,43.25,46051600,2.01

      Where 2.00 is the adjusted close and the other values are the recorded historical values at the time of trading. One would usually prefer to work with adjusted prices using the adjusted close as a reference for the scaling (including the volume)

      The new downloaded data (no processing)

      Date,Open,High,Low,Close,Adj Close,Volume
1995-01-03,2.179012,2.191358,2.117284,42.875004,2.117284,36301200
1995-01-04,2.123457,2.148148,2.092592,43.250004,2.135803,46051600

      And suddenly the prices offered are the adjusted prices. but with errors. Adjustedclose and historical close are actually swapped. The offered volume is the same as above, so one can wonder if the volume was right before and unadjusted or is right now and adjusted.

      In any case the code takes the new conditions in account (v7) and reverts to the original column ordering and behavior if version='' (empty), if old files have to be read.

      The Quickstart samples should now work as expected.

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by backtrader

        This post in Quant Stack Exchange, https://quant.stackexchange.com/questions/942/any-known-bugs-with-yahoo-finance-adjusted-close-data, may indicate that the new format may be more accurate, because in the original Yahoo was providing adjusted prices which had been rounded down to two decimals (2.00 in the first line of the previous samples), with no such rounding now.

        1 Reply Last reply Reply Quote 0
        • A
          ab_trader last edited by

          Now only adjusted prices are provided? No original prices?

          • If my answer helped, hit reputation up arrow at lower right corner of the post.
          • Python Debugging With Pdb
          • New to python and bt - check this out
          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            The feed seems to have/has changed but the functionality hasn't. If the parameter adjclose=True (the default), the adjusted prices are provided.

            • If it's a file with the original format, version='' (something which isn't v7), the adjusted prices are calculated using the provided adjusted close

            • If it's a file/download with the new format, version='v7' (the current default), the adjusted prices are provided directly from the feed.

            If you change to adjclose=False, the non-adjusted prices are provided directly from the feed (old format) or calculated backwards (new format)

            As a reference the note in the docs: Yahoo data feed: Docs - Yahoo Data Feed Notes

            1 Reply Last reply Reply Quote 0
            • C
              cgi1 last edited by

              @backtrader said in YahooFinance Data Feeds:

              1.9.50.117
              By skipping lines with null values
              By properly handling what seems to be the new format for adjusted prices.

              By skipping lines with null values [..] to properly handling adjusted price.
              Crashes when Volume is always == null. Could be draw anyway.

              B 1 Reply Last reply Reply Quote 0
              • C
                cgi1 last edited by

                Fixed in Pull Request https://github.com/mementum/backtrader/pull/325

                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators @cgi1 last edited by backtrader

                  @cgi1 said in YahooFinance Data Feeds:

                  @backtrader said in YahooFinance Data Feeds:

                  1.9.50.117
                  By skipping lines with null values
                  By properly handling what seems to be the new format for adjusted prices.

                  By skipping lines with null values [..] to properly handling adjusted price.
                  Crashes when Volume is always == null. Could be draw anyway.

                  This means that some lines (a sample line would help) contains prices which can be parse, but the string null that where the volume should be, or is it something else?

                  1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators last edited by

                    After further review of the pull-request which is done also against code which is no longer there (due to the changes introduced for the v7 format), a sample of what's exactly meant with null is really needed.

                    • Is this check for the old format, the new format or for both?
                    • How does this null look like in an actual data line?
                    1 Reply Last reply Reply Quote 0
                    • C
                      cgi1 last edited by cgi1

                      Thanks for the reply.

                      This means that some lines (a sample line would help) contains prices which can be parse, but the string null that where the volume should be, or is it something else?

                      Exactly: So sometimes the datafeed provider (in this case Yahoo on 2017-06-08) does not provider any Volume for some of the symbols. E.g. EURHKD=X:

                      Date,Open,High,Low,Close,Adj Close,Volume
2003-12-01,9.347397,9.278463,9.339722,9.290833,9.290833,null
2003-12-02,9.287244,9.278581,9.393217,9.380319,9.380319,null

2017-06-06,8.783572,8.734257,8.789911,8.784152,8.784152,null
2017-06-08,8.780444,8.732892,8.782912,8.748856,8.748856,0 <<<<< Date of download!

                      Some symbols are including Volume==0. E.g. EURUSD=X

                      Date,Open,High,Low,Close,Adj Close,Volume
2003-12-01,1.203398,1.194401,1.204007,1.196501,1.196501,0
2003-12-02,1.196101,1.194600,1.210903,1.208897,1.208897,0

2005-09-20,null,null,null,null,null,null <<<< Some dates are does not even contain any data

2017-06-06,1.127053,1.120523,1.127993,1.127142,1.127142,0
2017-06-08,1.126507,1.127142,1.119946,1.122083,1.122083,0 <<<<< Date of download!

                      Meanwhile on the same date a lot symbols (e.g. IFX.DE) contain Volume and works as expected.

                      Date,Open,High,Low,Close,Adj Close,Volume
2000-03-14,71.400002,80.000000,71.400002,78.900002,78.900002,9579700
2000-03-15,76.900002,76.949997,70.800003,73.040001,73.040001,5855009

                      Hope this helps. If you need further information please let me know.

                      B 1 Reply Last reply Reply Quote 0
                      • B
                        backtrader administrators @cgi1 last edited by

                        @cgi1 said in YahooFinance Data Feeds:

                        Exactly: So sometimes the datafeed provider (in this case Yahoo on 2017-06-08) does not provider any Volume for some of the symbols. E.g. EURHKD=X:

                        To be expected. See below

                        @cgi1 said in YahooFinance Data Feeds:

                        Some symbols are including Volume==0. E.g. EURUSD=X

                        This is to be expected. Yahoo is giving you the conversion rate. It's not a broker/ECN, has no volume and cannot know what the volume around the world is.

                        In this case also the adjusted close and the non-adjusted close are bound to always be the same: a currency pair has no splits or dividends.

                        @cgi1 said in YahooFinance Data Feeds:

                        2003-12-01,9.347397,9.278463,9.339722,9.290833,9.290833,null

                        This requires exception catching in the conversion to float. The pull-request wouldn't have done this.

                        1 Reply Last reply Reply Quote 0
                        • B
                          backtrader administrators last edited by

                          Release 1.9.56.122 contains an exception for volume fields which are not convertible to float.

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post
                          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors