Quandl Data Feed - Futures Data

jxm262

Hi,

I'm trying to import cme futures data using the Quandl Data feed I see in the docs - https://www.backtrader.com/docu/dataautoref.html?highlight=quandl , but getting an error.

bt.feeds.Quandl(
    dataset='CME',
    fromdate=datetime.datetime(2017, 1, 1),
    todate=datetime.datetime(2017, 12, 30),
    dataname='RBH2017',
    apikey='...'
)

# error
  File "/Users/jmaat/anaconda3/envs/algo/lib/python3.5/site-packages/backtrader/feeds/quandl.py", line 108, in _loadline
    h = float(linetokens[next(i)])
IndexError: list index out of range

It appears the headers on the incoming data (cme futures) don't correspond to the accepted column names used in the source code (wiki stock data works fine).

For example, the provided example getting YHOO data under WIKI

req:  https://www.quandl.com/api/v3/datasets/WIKI/YHOO.csv
resp: headers in csv
Date | Open | High | Low | Close | Volume | ...

But data from CME futures look like this

req: https://www.quandl.com/api/v3/datasets/CME/RBH2017.csv?order=asc&start_date=2017-01-01&end_date=2017-12-30

resp: csv headers 
Date | Open | High | Low | Last | Change | Settle | Volume

Note, the the change from Close to Last between the 2 calls. I think it looks like the quandl feed is implementation specific to handle a specific csv header/column types? (ie it must match the first example).

Here's the code I'm tracing through - https://github.com/mementum/backtrader/blob/3e9c6b256e37724769e937a9a0c33ee93c4a09b4/backtrader/feeds/quandl.py#L107

I was wondering if it's possible to customize the naming/positioning of the headers coming in from the datafeed, similar to GenericCSVData

ie.  something along these lines
rb17 = bt.feeds.Quandl(
    dataset='CME',
    fromdate=datetime.datetime(2017, 1, 1),
    todate=datetime.datetime(2017, 12, 30),
    dataname='RBH2017',
    datetime=0,
    open=1,
    high=2,
    low=3,
    close=4,
    volume=7,
    openinterest=-1,
    apikey='xxx'
)

Does anyone know an easy way to import quandl futures data from CME? Alternatively, would anyone like me to make a PR to generalize this somehow so users can add some sort of custom converter? (or at least handle futures and stock data in this same class)

Side note - this library is awesome :) I'm liking it much more than zipline at the moment.

ThatBlokeDave

Great timing. I just had a similar issue with this using London Stock Exchange data.

One workaround is to use the official Quandl api to download the data first. Then import it as a pandas data feed.

ab_trader

I believe I had the same issues. And decided to download data from Quandl separately from backtrader and then use simple generic CSV datafeed. More useful for me cause I am going to create continuous futures by myself.

jxm262

Thanks for the input @ThatBlokeDave @ab_trader

For now, I just extended out the existing feeds.Quandl class and rewrote this part. Obviously doesn't handle adjclose and could probably break in the future, if anything changed on the quandl side.

# from feeds.quandl.QuandlCSV

    def _loadline(self, linetokens):
        i = itertools.count(0)

        dttxt = linetokens[next(i)]  # YYYY-MM-DD
        dt = date(int(dttxt[0:4]), int(dttxt[5:7]), int(dttxt[8:10]))
        dtnum = date2num(datetime.combine(dt, self.p.sessionend))

        self.lines.datetime[0] = dtnum
        # todo add this back in the future
        # if self.p.adjclose:
        #     for _ in range(7):
        #         next(i)  # skip ohlcv, ex-dividend, split ratio

        o = float(linetokens[1])
        h = float(linetokens[2])
        l = float(linetokens[3])
        c = float(linetokens[4])
        v = float(linetokens[7])

        # o = float(linetokens[next(i)])
        # h = float(linetokens[next(i)])
        # l = float(linetokens[next(i)])
        # c = float(linetokens[next(i)])
        # v = float(linetokens[next(i)])
        self.lines.openinterest[0] = 0.0

I'll try to make something more reusable/formal and add in a PR. Would be nice to give back to the community :)

stevetree

I used the quandl python lib, which returns a pandas DataFrame. Renaming columns and then using bt.feeds.PandasData to create a feed.

import quandl
quandl.ApiConfig.api_key = "xxxx"

df = quandl.get_table('SCF/PRICES',  quandl_code='CME_GC1_ON')
df.rename(columns={'settle': 'close','prev_day_open_interest':'openinterest'}, inplace=True)

data = bt.feeds.PandasData(dataname=df.iloc[::-1],datetime="date")