Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    YahooFinanceData gets displayed invalid due to changed adjusted close

    General Code/Help
    3
    12
    4129
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      cgi1 last edited by

      e.g. for ADS.DE on 2010-05-20 the YahooFinanceData is :

      Date,Open,High,Low,Close,Adj Close,Volume
      2010-05-20,41.275002,41.320000,39.139999,39.250000,35.047554,1874117
      

      The data is getting displayed on the chart from Adj Close=35.04 till High=41.27:

      0_1498830226519_03311fc9-a4c9-43e5-9cfa-1dac974d6e29-grafik.png

      • Option adjclose (dataautoref docu) does not change anything. Backtrader version backtrader-1.9.54.122.

      • It looks like the adjclose field does not get handled correctly.

      1 Reply Last reply Reply Quote 0
      • Curtis Miller
        Curtis Miller last edited by

        backtrader expects its data feeds to be in OHLCV format, which means there should not be a column Adj Close. You could use the backtrader data feeds dedicated to Yahoo! Finance data. Otherwise, you will need to remove the Adj Close column (use it to adjust all columns: see my blog post on doing this). Perhaps load your data into a pandas DataFrame and use that as the data feed.

        Word on the street is that Yahoo! Finance data isn't what it used to be.

        Blog posts: Walk Forward Analysis Demonstration, Stock Trading Analytics and Optimization

        Books/Video Courses: Unpacking NumPy and Pandas

        1 Reply Last reply Reply Quote 0
        • B
          backtrader administrators last edited by

          This is how the feed behaves after the Yahoo fiasco: Community - YahooFinance Data Feeds

          But most people have probably abandoned Yahoo (apparently for some (many?) assets splits and/or dividends are no longer considered) and the behavior you quote was unnoticed which was probably introduced in the 2nd revision of the new feed to support the null pricing.

          To be corrected

          C 1 Reply Last reply Reply Quote 0
          • C
            cgi1 @backtrader last edited by

            @backtrader said in YahooFinanceData gets displayed invalid due to changed adjusted close:

            This is how the feed behaves after the Yahoo fiasco: Community - YahooFinance Data Feeds

            ...the behavior you quote was unnoticed which was probably introduced in the 2nd revision of the new feed to support the null pricing.

            ... had the same assumption. I extracted some of your code to build a yahoo-downloader which downloads about 200 symbols periodically (now every week via cron) to have an up-to-date copy locally (if yahoo disables their API again).

            Due to my observation the change has been implemented between 2017-06-08 - 2017-06-30.

            @Curtis-Miller said in YahooFinanceData gets displayed invalid due to changed adjusted close:

            use it to adjust all columns: see my blog post on doing this.

            I will give it a try. Thanks for the link.

            Word on the street is that Yahoo! Finance data isn't what it used to be.

            Hm.. Y.

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              Change implemented

              adjclose=True (default)

              0_1498901773601_3c340de0-0377-4534-bb31-c77baa25bf6a-image.png

              adjclose=False

              0_1498901850255_eacfaba8-c5c2-4d50-8643-91a8ce7f12f0-image.png

              Commited to the development branch

              1 Reply Last reply Reply Quote 0
              • C
                cgi1 last edited by

                Awesome. Will test it now and provide feedback afterwards.

                1 Reply Last reply Reply Quote 0
                • C
                  cgi1 last edited by cgi1

                  It looks like there is a smaller bug inside:

                  0_1498905058408_15ec4231-5984-4fa6-9656-67ff52b5dfea-grafik.png

                  More detailed view:

                  0_1498905245339_9773e9f2-21bc-4732-b1b0-6e2c49173e6f-grafik.png

                  EDIT --> Relevant data:

                  Date,Open,High,Low,Close,Adj Close,Volume
                  2006-05-29,39.000000,39.750000,39.000000,157.830002,33.445068,2680704
                  2006-05-30,39.505001,39.750000,38.395000,153.949997,32.622871,6441648
                  2006-05-31,38.125000,38.837502,37.762501,154.800003,32.802998,9789904
                  2006-06-01,38.654999,38.685001,38.119999,154.149994,32.665260,3936688
                  2006-06-02,38.900002,39.247501,38.567501,155.440002,32.938614,6460944
                  2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
                  2006-06-06,38.360001,38.790001,37.570000,37.740002,31.989273,3515592 <<<<<<<
                  2006-06-07,37.709999,37.910000,36.750000,36.959999,31.328129,2888015
                  
                  1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators last edited by

                    ...
                    2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
                    2006-06-06,38.360001,38.790001,37.570000,37.740002,31.989273,3515592 <<<<<<<
                    ...
                    

                    Yes there is a bug: in the data Yahoo is returning. And shows what to expect from the downloads after the fiasco. Some data will look like this and some data will look like that. (i.e.: sometimes the adjusted close is the adjusted closes and some other times ... it isn't)

                    If you look at it, there is no easy way to fix this:

                    ...
                    2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
                    ...
                    

                    32.777565 is far away from the 39s and 38s from open, high and close. And there isn't (please correct this statement if wrong) any way to use it in combination with 154.679993 to restore a close price which is in between the high of 39.237499 and the low of 38.527500

                    C 1 Reply Last reply Reply Quote 0
                    • C
                      cgi1 @backtrader last edited by

                      @backtrader True.. My only idea right now would be something like a santize check to filter out invalid data with the goal of not manipulating indicators and trades.. But how to proceed with that rows? What a mess.

                      1 Reply Last reply Reply Quote 0
                      • C
                        cgi1 last edited by cgi1

                        Maybe we need to split this one up a little bit:

                        1. Invalid data could be theoratically filtered out using santize checks.
                        2. Adjusted close should be used if the value is plausible.

                        Regarding 2) I just pulled the current development branch (a83d855e9c1e219dc0cd1f243ee020b13b48e3ba) and it looks like after the code changes in 24f9515b589fe0cef88573282424424f2f784d64 (Correct YahooFinanceData adjclose behavior) it is even worse than before.

                        As an example AAPL:

                        Date,Open,High,Low,Close,Adj Close,Volume
                        2012-05-18,76.279999,77.629997,74.597145,530.380005,75.768570,183073100
                        

                        now results into:

                        0_1498929326608_c9dbf9c0-3197-4ca9-b21a-072eddb150b2-grafik.png

                        where one would assume a candle with close=75.768570 instead of 530.380005.

                        1 Reply Last reply Reply Quote 0
                        • B
                          backtrader administrators last edited by

                          It's not because it is worse than before. Notice that in this case the "Close" is unadjusted, which isn't the case for example in the ADS.DE sample from above.

                          @cgi1 said in YahooFinanceData gets displayed invalid due to changed adjusted close:

                          Date,Open,High,Low,Close,Adj Close,Volume
                          2010-05-20,41.275002,41.320000,39.139999,39.250000,35.047554,1874117
                          

                          Here the 5th component price is the one which is obviously not in line with the other four.

                          In the AAPL case now, it is the 4th component price the odd one out.

                          It is obviously an issue of finding out which case is what. And that's a manual process. The only thing the feed can receive is something like a parameter with a name like swappedcloses which you may apply to each individual case.

                          With this in mind, it seems the tests for the changes in the YahooFinanceData may have been tested with answers like the one you have for AAPL now.

                          This only confirms why people are moving away from Yahoo (there are many queries here and there about alternative reliable -and free- data sources)

                          1 Reply Last reply Reply Quote 0
                          • B
                            backtrader administrators last edited by

                            After the different findings in this topic, the following will be the final commit

                            • The original behavior for the v7 feed will be retained
                            • A new parameter named swapcloses (default: False) will be added

                            If swapcloses=True the allegedly adjusted and non-adjusted closing prices will be swapped, which will allow to place them in the right column.

                            There seems to be no clear pattern to decide when Yahoo will be returning the adjusted price or the unadjusted (leaving aside that as seen above, sometimes it is impossible to reconstruct the prices with the given values)

                            1 Reply Last reply Reply Quote 1
                            • 1 / 1
                            • First post
                              Last post
                            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors