Hi @backtrader,

Any update/ETA on this?

@backtrader, I'm not entirely sure what you mean with the first option, but if it is what I think it is, the way the Quandl API would be used would be similar to how GenericCSV works. I'm OK with this. That said, I'm also OK with option 2.

I appreciate your looking into this and look forward to a resolution of the matter. Of course, I trust your judgment regarding which option you plan to implement. Personally, I feel that option 2 would be more user-friendly. Please let us know when a solution is available.

Thanks, as usual, for your effective and quick replies!

@nomnom, I agree with what @backtrader says. If you retrieved the data from Yahoo like I did (in my case using YahooFinanceData), you will observe random inconsistencies (sometimes, everything is fine, sometimes, everything is bad up to a point and then gets better**, sometimes the whole dataset is bad). If you read the post which @backtrader referenced, you will see that I had exactly the same issues when I tried to plot the data using candlesticks.

Try seeing if you get the same problem with a different data source (e.g. Quandl).

** Try extending the timeframe for your example (make it go all the way up to yesterday). I'm willing to bet that you'll see that it "improves" after a certain date.

One more thing: I think the column with the empty value does not represent the volume. From the Quandl web UI for BMW (click on Table), we see the following columns:

Date,Open,High,Low,Close,Change,Traded Volume,Turnover,Last Price of the day,Daily Traded Units,Daily Turnover

And, correspondingly, we see that linetokens is:

['2016-07-27', '77.64', '78.8', '77.51', '78.36', '', '1971401.0', '154530852.0', '', '', '']

So we are expecting to see the volume where Change actually is.

I checked 3 different databases: WIKI, FSE and NSE. The columns returned are different in each case.

WIKI: Date,Open,High,Low,Close,Volume,Ex-Dividend,SplitRatio,Adj.Open,Adj.High,Adj.Low,Adj.Close,Adj.Volume

NSE: Date,Open,High,Low,Last,Close,TotalTradeQuantity,Turnover(Lacs)

FSE: Date,Open,High,Low,Close,Change,TradedVolume,Turnover,LastPriceoftheDay,DailyTradedUnits,DailyTurnover

So, @backtrader, I guess you will have to treat each one differently. One way to retrieve the columns programmatically is to query the database like this: https://www.quandl.com/api/v3/datasets.json?database_code=<WIKI/NSE/FSE/ETC>&per_page=100&sort_by=id&page=1&api_key=<XXXX>. Every "dataset" node contains a child called "column_names" which is a list of the columns returned.

Hope this helps.

@dasch, yeah I thought so too.

@backtrader, can this be handled internally such that a warning is thrown and the empty value is interpreted as 0/NaN/infinity/etc. so that the process may continue to execute?

@dasch, it still doesn't work even if I provide the API key. I get the same error with this line:

quandl_finance_data = bt.feeds.Quandl(dataname="BMW_X", fromdate=from_date, todate=to_date, round=False, adjclose=False, apikey="xxxx", dataset="FSE")

However, if I switch to the WIKI dataset (instead of FSE) and use a ticker like YHOO, everything works fine. So I'm guessing the problem is with the usage of the FSE dataset.

Hi again @dasch and @backtrader,

As usual, thanks for your quick replies! Instead of retrieving the data from every symbol twice (once adjusted, once not) and then taking the correct values from each, I decided to see if I could switch from Yahoo to Quandl instead. I found the BMW ticker (from the FSE dataset, which seems to be free to access) and decided to take it for a test run. Sadly, I didn't get very far. Here's the code (pretty much the same as before except the data acquisition part):

from datetime import datetime, timedelta, date
import backtrader as bt

# get Yahoo finance data
lookback_timedelta_days = timedelta(days=365)
now_datetime = datetime.now()
to_date = datetime(now_datetime.year, now_datetime.month, now_datetime.day)
from_date = to_date - lookback_timedelta_days

# OLD FEED FROM YAHOO
# yahoo_finance_data = bt.feeds.YahooFinanceData(dataname="BMW.F", fromdate=from_date, todate=to_date, adjclose=False,
#                                                swapcloses=False, round=False)

# NEW FEED FROM QUANDL
quandl_finance_data = bt.feeds.Quandl(dataname="BMW_X", fromdate=from_date, todate=to_date, dataset='FSE', round=False,
                                      adjclose=False)

# start cerebro
cerebro = bt.Cerebro()
cerebro.adddata(data=quandl_finance_data)

# run cerebro
cerebro_result = cerebro.run()

# capture results
date_series = cerebro_result[0].data.datetime.array
open_series = list(cerebro_result[0].data.open.array)
high_series = list(cerebro_result[0].data.high.array)
low_series = list(cerebro_result[0].data.low.array)
close_series = list(cerebro_result[0].data.close.array)
volume_series = list(cerebro_result[0].data.volume.array)
with open('results.txt', 'w') as file_obj:
    file_obj.write('Date,Open,High,Low,Close,Volume\n')
    for day_iter in range(len(date_series)):
        this_date = date.fromordinal(int(date_series[day_iter]))
        this_open = open_series[day_iter]
        this_high = high_series[day_iter]
        this_low = low_series[day_iter]
        this_close = close_series[day_iter]
        this_volume = volume_series[day_iter]
        file_obj.write('%s,%f,%f,%f,%f,%d\n' % (this_date, this_open, this_high, this_low, this_close, this_volume))

I get this error:

/usr/bin/python2.7 <bla>/btYahooIssue.py
Traceback (most recent call last):
  File "<bla>/btYahooIssue.py", line 22, in <module>
    cerebro_result = cerebro.run()
  File "/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.py", line 1073, in run
    runstrat = self.runstrategies(iterstrat)
  File "/usr/local/lib/python2.7/dist-packages/backtrader/cerebro.py", line 1149, in runstrategies
    data.preload()
  File "/usr/local/lib/python2.7/dist-packages/backtrader/feed.py", line 682, in preload
    while self.load():
  File "/usr/local/lib/python2.7/dist-packages/backtrader/feed.py", line 476, in load
    _loadret = self._load()
  File "/usr/local/lib/python2.7/dist-packages/backtrader/feed.py", line 704, in _load
    return self._loadline(linetokens)
  File "/usr/local/lib/python2.7/dist-packages/backtrader/feeds/quandl.py", line 111, in _loadline
    v = float(linetokens[next(i)])
ValueError: could not convert string to float: 

Process finished with exit code 1

What am I doing wrong?

Thanks, @dasch and @backtrader for your quick answers!

@dasch: I tried your suggestion, but unfortunately, the results are still inconsistent. One good tip, however, from your answer was to use round=False. Thanks for that! I guess we will never get the correct results due to the seemingly inconsistent behavior of the data returned by Yahoo. From the BT data feed reference:

If True the allegedly adjusted close and non-adjusted close will be swapped. The downloads with the new v7 API show at random times the closing prices swapped. There is no known pattern

If I use your suggestion, i.e., adjclose=False, swapcloses=True, round=False, we get the following:

From Yahoo: 
Date,Open,High,Low,Close,Adj Close,Volume
2016-07-27,76.838997,78.705002,76.838997,78.142998,75.114166,12079

Ours: 
Date,Open,High,Low,Close,Volume
2016-07-27,79.937379,81.878627,79.937379,78.142998,12079

We see that the close (and volume) values match, but nothing else does.

Now, if I use adjclose=True, swapcloses=True, round=False, we get the following:

From Yahoo: 
Date,Open,High,Low,Close,Adj Close,Volume
2016-07-27,76.838997,78.705002,76.838997,78.142998,75.114166,12079

Ours: 
Date,Open,High,Low,Close,Volume
2016-07-27,76.838997,78.705002,76.838997,75.114166,11610

We see that the everything matches except for the close and volume values (exactly the opposite of the previous case). After a certain date, they do start matching, but I can't rely on something like this.

So it seems that if I want to get all the OHLCV values correct, I have to use a combination of the two approaches above.

@backtrader: I suppose I have been living under a rock. So far, I have only required the close values and since I always used adjclose=False, I got the proper values. Now, I need the other values too and therefore have only started venturing out from under my rock just now. For the record, I never meant to imply that there is some bug with BT. Please don't take it personally. I have always maintained in all my other posts that BT is an absolutely amazing product. Unfortunately, I do not see any alternative to using YahooFinanceData because I require mostly German and Indian equities and so far, I haven't been able to find an alternative service that provides these free of charge (I'm only doing this as a hobby in my free time and would therefore be unwilling to pay for a service). If you know a better source, I am all ears. Thank you again, for your continued support!

Hello again BT community!

I have noticed an interesting problem which occurs on certain Yahoo symbols (I think - I have to admit, I haven't tested very extensively) when using the YahooFinanceData API. I notice that for the affected symbols, the OHLCV (open, high, low, close, volume) values are incorrect up to a certain point in time, after which they are correct.

To demonstrate this odd behavior, I have written a small script. Please execute the code below:

from datetime import datetime, timedelta, date
import backtrader as bt

# get Yahoo finance data
lookback_timedelta_days = timedelta(days=365)
now_datetime = datetime.now()
to_date = datetime(now_datetime.year, now_datetime.month, now_datetime.day)
from_date = to_date - lookback_timedelta_days
yahoo_finance_data = bt.feeds.YahooFinanceData(dataname="BMW.F", fromdate=from_date, todate=to_date, adjclose=False)

# start cerebro
cerebro = bt.Cerebro()
cerebro.adddata(data=yahoo_finance_data)

# run cerebro
cerebro_result = cerebro.run()

# capture results
date_series = cerebro_result[0].data.datetime.array
open_series = list(cerebro_result[0].data.open.array)
high_series = list(cerebro_result[0].data.high.array)
low_series = list(cerebro_result[0].data.low.array)
close_series = list(cerebro_result[0].data.close.array)
volume_series = list(cerebro_result[0].data.volume.array)
with open('results.txt', 'w') as file_obj:
    file_obj.write('Date,Open,High,Low,Close,Volume')
    for day_iter in range(len(date_series)):
        this_date = date.fromordinal(int(date_series[day_iter]))
        this_open = open_series[day_iter]
        this_high = high_series[day_iter]
        this_low = low_series[day_iter]
        this_close = close_series[day_iter]
        this_volume = volume_series[day_iter]
        file_obj.write('%s,%f,%f,%f,%f,%d\n' % (this_date, this_open, this_high, this_low, this_close, this_volume))

Sorry for the messy coding. It's just intended as a demo. As you see, we retrieve the historical OHLCV data for the symbol "BMW.F" going a year back and then write it to a file (results.txt) with the aim that we can compare this file against the one retrieved from Yahoo's web-interface (go here, then click "download data"). Please save the file and remove the "adjusted close" column because we don't need it (I would have attached the file, but uploads are not allowed).

If you compare the two files, you will see that there are big differences in the OHLC values up to May 11 2017. There are never any errors for the volume information, by the way. After this point, there are still differences, but they are very small (maybe precision errors?). Please see the image below to see what I mean. Up to May 11 2017, we see consistent differences on the order of ~3 Euros. After this point, the difference is on the order of ~<=0.01 Euros.

I have also created a candlestick chart using the data retrieved from the YahooFinanceData API. It confirms what we see above. Before May 11, the data looks extremely weird where not only are the values wrong, but (almost) every day, the close is lower than the open. After May 11, we see that reality is reflected more accurately.

Finally, as I said, this doesn't seem to happen for all symbols. Here are a few others where I notice this behavior with the date where bad data becomes good (ARL.DE: June 1 2017; ADJ.DE; May 4 2017; AIRA.DE; April 11 2017, etc.).

Thanks for reading this long post. I hope you can figure out what the issue is! Please let me know if I can provide any additional help.

pip show backtrader

Name: backtrader
Version: 1.9.51.121
Summary: BackTesting Engine
Home-page: https://github.com/mementum/backtrader
Author: Daniel Rodriguez
Author-email: danjrod@gmail.com
License: GPLv3+
Location: /usr/local/lib/python2.7/dist-packages
Requires: 

@backtrader, is there any news/update on this issue? Thanks in advance.