YahooFinanceData gets displayed invalid due to changed adjusted close

[cgi1]

e.g. for ADS.DE on 2010-05-20 the YahooFinanceData is :

Date,Open,High,Low,Close,Adj Close,Volume
2010-05-20,41.275002,41.320000,39.139999,39.250000,35.047554,1874117

The data is getting displayed on the chart from Adj Close=35.04 till High=41.27:

• Option adjclose (dataautoref docu) does not change anything. Backtrader version backtrader-1.9.54.122.

• It looks like the adjclose field does not get handled correctly.

[Curtis Miller]

backtrader expects its data feeds to be in OHLCV format, which means there should not be a column Adj Close. You could use the backtrader data feeds dedicated to Yahoo! Finance data. Otherwise, you will need to remove the Adj Close column (use it to adjust all columns: see my blog post on doing this). Perhaps load your data into a pandas DataFrame and use that as the data feed.

Word on the street is that Yahoo! Finance data isn't what it used to be.

[backtrader]

This is how the feed behaves after the Yahoo fiasco: Community - YahooFinance Data Feeds

But most people have probably abandoned Yahoo (apparently for some (many?) assets splits and/or dividends are no longer considered) and the behavior you quote was unnoticed which was probably introduced in the 2nd revision of the new feed to support the null pricing.

To be corrected

[cgi1]

@backtrader said in YahooFinanceData gets displayed invalid due to changed adjusted close:

This is how the feed behaves after the Yahoo fiasco: Community - YahooFinance Data Feeds

...the behavior you quote was unnoticed which was probably introduced in the 2nd revision of the new feed to support the null pricing.

... had the same assumption. I extracted some of your code to build a yahoo-downloader which downloads about 200 symbols periodically (now every week via cron) to have an up-to-date copy locally (if yahoo disables their API again).

Due to my observation the change has been implemented between 2017-06-08 - 2017-06-30.

@Curtis-Miller said in YahooFinanceData gets displayed invalid due to changed adjusted close:

use it to adjust all columns: see my blog post on doing this.

I will give it a try. Thanks for the link.

Word on the street is that Yahoo! Finance data isn't what it used to be.

Hm.. Y.

[backtrader]

Change implemented

adjclose=True (default)

adjclose=False

Commited to the development branch

[cgi1]

Awesome. Will test it now and provide feedback afterwards.

[cgi1]

It looks like there is a smaller bug inside:

More detailed view:

EDIT --> Relevant data:

Date,Open,High,Low,Close,Adj Close,Volume
2006-05-29,39.000000,39.750000,39.000000,157.830002,33.445068,2680704
2006-05-30,39.505001,39.750000,38.395000,153.949997,32.622871,6441648
2006-05-31,38.125000,38.837502,37.762501,154.800003,32.802998,9789904
2006-06-01,38.654999,38.685001,38.119999,154.149994,32.665260,3936688
2006-06-02,38.900002,39.247501,38.567501,155.440002,32.938614,6460944
2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
2006-06-06,38.360001,38.790001,37.570000,37.740002,31.989273,3515592 <<<<<<<
2006-06-07,37.709999,37.910000,36.750000,36.959999,31.328129,2888015

[backtrader]

...
2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
2006-06-06,38.360001,38.790001,37.570000,37.740002,31.989273,3515592 <<<<<<<
...

Yes there is a bug: in the data Yahoo is returning. And shows what to expect from the downloads after the fiasco. Some data will look like this and some data will look like that. (i.e.: sometimes the adjusted close is the adjusted closes and some other times ... it isn't)

If you look at it, there is no easy way to fix this:

...
2006-06-05,39.000000,39.237499,38.527500,154.679993,32.777565,3969136 <<<<<<<
...

32.777565 is far away from the 39s and 38s from open, high and close. And there isn't (please correct this statement if wrong) any way to use it in combination with 154.679993 to restore a close price which is in between the high of 39.237499 and the low of 38.527500

[cgi1]

@backtrader True.. My only idea right now would be something like a santize check to filter out invalid data with the goal of not manipulating indicators and trades.. But how to proceed with that rows? What a mess.

[cgi1]

Maybe we need to split this one up a little bit:

1. Invalid data could be theoratically filtered out using santize checks.
2. Adjusted close should be used if the value is plausible.

Regarding 2) I just pulled the current development branch (a83d855e9c1e219dc0cd1f243ee020b13b48e3ba) and it looks like after the code changes in 24f9515b589fe0cef88573282424424f2f784d64 (Correct YahooFinanceData adjclose behavior) it is even worse than before.

As an example AAPL:

Date,Open,High,Low,Close,Adj Close,Volume
2012-05-18,76.279999,77.629997,74.597145,530.380005,75.768570,183073100

now results into:

where one would assume a candle with close=75.768570 instead of 530.380005.

[backtrader]

It's not because it is worse than before. Notice that in this case the "Close" is unadjusted, which isn't the case for example in the ADS.DE sample from above.

@cgi1 said in YahooFinanceData gets displayed invalid due to changed adjusted close:

Date,Open,High,Low,Close,Adj Close,Volume
2010-05-20,41.275002,41.320000,39.139999,39.250000,35.047554,1874117

Here the 5th component price is the one which is obviously not in line with the other four.

In the AAPL case now, it is the 4th component price the odd one out.

It is obviously an issue of finding out which case is what. And that's a manual process. The only thing the feed can receive is something like a parameter with a name like swappedcloses which you may apply to each individual case.

With this in mind, it seems the tests for the changes in the YahooFinanceData may have been tested with answers like the one you have for AAPL now.

This only confirms why people are moving away from Yahoo (there are many queries here and there about alternative reliable -and free- data sources)

[backtrader]

After the different findings in this topic, the following will be the final commit

• The original behavior for the v7 feed will be retained
• A new parameter named swapcloses (default: False) will be added

If swapcloses=True the allegedly adjusted and non-adjusted closing prices will be swapped, which will allow to place them in the right column.

There seems to be no clear pattern to decide when Yahoo will be returning the adjusted price or the unadjusted (leaving aside that as seen above, sometimes it is impossible to reconstruct the prices with the given values)