Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Quandl Data Feed - Futures Data

    General Code/Help
    4
    5
    2714
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jxm262
      jxm262 last edited by

      Hi,

      I'm trying to import cme futures data using the Quandl Data feed I see in the docs - https://www.backtrader.com/docu/dataautoref.html?highlight=quandl , but getting an error.

      bt.feeds.Quandl(
          dataset='CME',
          fromdate=datetime.datetime(2017, 1, 1),
          todate=datetime.datetime(2017, 12, 30),
          dataname='RBH2017',
          apikey='...'
      )
      
      # error
        File "/Users/jmaat/anaconda3/envs/algo/lib/python3.5/site-packages/backtrader/feeds/quandl.py", line 108, in _loadline
          h = float(linetokens[next(i)])
      IndexError: list index out of range
      

      It appears the headers on the incoming data (cme futures) don't correspond to the accepted column names used in the source code (wiki stock data works fine).

      For example, the provided example getting YHOO data under WIKI

      req:  https://www.quandl.com/api/v3/datasets/WIKI/YHOO.csv
      resp: headers in csv
      Date | Open | High | Low | Close | Volume | ...
      

      But data from CME futures look like this

      req: https://www.quandl.com/api/v3/datasets/CME/RBH2017.csv?order=asc&start_date=2017-01-01&end_date=2017-12-30
      
      resp: csv headers 
      Date | Open | High | Low | Last | Change | Settle | Volume
      

      Note, the the change from Close to Last between the 2 calls. I think it looks like the quandl feed is implementation specific to handle a specific csv header/column types? (ie it must match the first example).

      Here's the code I'm tracing through - https://github.com/mementum/backtrader/blob/3e9c6b256e37724769e937a9a0c33ee93c4a09b4/backtrader/feeds/quandl.py#L107

      I was wondering if it's possible to customize the naming/positioning of the headers coming in from the datafeed, similar to GenericCSVData

      ie.  something along these lines
      rb17 = bt.feeds.Quandl(
          dataset='CME',
          fromdate=datetime.datetime(2017, 1, 1),
          todate=datetime.datetime(2017, 12, 30),
          dataname='RBH2017',
          datetime=0,
          open=1,
          high=2,
          low=3,
          close=4,
          volume=7,
          openinterest=-1,
          apikey='xxx'
      )
      

      Does anyone know an easy way to import quandl futures data from CME? Alternatively, would anyone like me to make a PR to generalize this somehow so users can add some sort of custom converter? (or at least handle futures and stock data in this same class)

      Side note - this library is awesome :) I'm liking it much more than zipline at the moment.

      1 Reply Last reply Reply Quote 0
      • T
        ThatBlokeDave last edited by

        Great timing. I just had a similar issue with this using London Stock Exchange data.

        One workaround is to use the official Quandl api to download the data first. Then import it as a pandas data feed.

        1 Reply Last reply Reply Quote 0
        • A
          ab_trader last edited by

          I believe I had the same issues. And decided to download data from Quandl separately from backtrader and then use simple generic CSV datafeed. More useful for me cause I am going to create continuous futures by myself.

          • If my answer helped, hit reputation up arrow at lower right corner of the post.
          • Python Debugging With Pdb
          • New to python and bt - check this out
          1 Reply Last reply Reply Quote 0
          • jxm262
            jxm262 last edited by

            Thanks for the input @ThatBlokeDave @ab_trader

            For now, I just extended out the existing feeds.Quandl class and rewrote this part. Obviously doesn't handle adjclose and could probably break in the future, if anything changed on the quandl side.

            # from feeds.quandl.QuandlCSV
            
                def _loadline(self, linetokens):
                    i = itertools.count(0)
            
                    dttxt = linetokens[next(i)]  # YYYY-MM-DD
                    dt = date(int(dttxt[0:4]), int(dttxt[5:7]), int(dttxt[8:10]))
                    dtnum = date2num(datetime.combine(dt, self.p.sessionend))
            
                    self.lines.datetime[0] = dtnum
                    # todo add this back in the future
                    # if self.p.adjclose:
                    #     for _ in range(7):
                    #         next(i)  # skip ohlcv, ex-dividend, split ratio
            
                    o = float(linetokens[1])
                    h = float(linetokens[2])
                    l = float(linetokens[3])
                    c = float(linetokens[4])
                    v = float(linetokens[7])
            
                    # o = float(linetokens[next(i)])
                    # h = float(linetokens[next(i)])
                    # l = float(linetokens[next(i)])
                    # c = float(linetokens[next(i)])
                    # v = float(linetokens[next(i)])
                    self.lines.openinterest[0] = 0.0
            

            I'll try to make something more reusable/formal and add in a PR. Would be nice to give back to the community :)

            1 Reply Last reply Reply Quote 0
            • stevetree
              stevetree last edited by

              I used the quandl python lib, which returns a pandas DataFrame. Renaming columns and then using bt.feeds.PandasData to create a feed.

              import quandl
              quandl.ApiConfig.api_key = "xxxx"
              
              df = quandl.get_table('SCF/PRICES',  quandl_code='CME_GC1_ON')
              df.rename(columns={'settle': 'close','prev_day_open_interest':'openinterest'}, inplace=True)
              
              data = bt.feeds.PandasData(dataname=df.iloc[::-1],datetime="date")
              
              1 Reply Last reply Reply Quote 1
              • 1 / 1
              • First post
                Last post
              Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors