IQFeed store and feed: help needed



  • Hi @backtrader,

    I'm trying to implement IQFeed support for backtrader: iqfed branch@ my github repo

    It started to work, but I can't figure out one thing:
    For the strategy with 2 dataframes (see below) the data for the second dataframe (daily) is continuously growing.

    ...
    store = bt.stores.IQFeedStore(tickers=tickers, fields=fields, reconnect=args.reconnect,
                                      timeout=args.timeout, debug=args.debug)
        # Add data feeds
        for ticker in tickers:
            data_min = store.getdata(dataname=ticker, name="%s_min" % ticker, tz='UTC',
                                     timeframe=bt.TimeFrame.Ticks)
            data_min = cerebro.resampledata(data_min, timeframe=bt.TimeFrame.Minutes)
    
            hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 1)[0].to_pydatetime()
            data_day = store.getdata(dataname=ticker, name="%s_day" % ticker, tz='UTC', fromdate=hist_start_date,
                                     timeframe=bt.TimeFrame.Days, backfill_start=True)
            cerebro.adddata(data_day)
    ...
    

    I've added this code to the 'next' method of my strategy:

    print "len:", len(self.datas[1])) #, Series(self.data1.high.get(size=30)))
    print(">>> dates:", [bt.utils.date.num2date(date).strftime("%Y-%m-%d %H:%M:%S") for date in self.data1.datetime.get(size=len(self.data1))])
    

    Here is the output:

    len: 32                                                                                                                                                               
    dates: ['2017-08-22 00:00:00', '2017-08-23 00:00:00', '2017-08-24 00:00:00', '2017-08-25 00:00:00', '2017-08-28 00:00:00', '2017-08-29 00:00:00', '2017-08-30 00:00:00'
    , '2017-08-31 00:00:00', '2017-09-01 00:00:00', '2017-09-05 00:00:00', '2017-09-06 00:00:00', '2017-09-07 00:00:00', '2017-09-08 00:00:00', '2017-09-11 00:00:00', '2017-09-12 
    00:00:00', '2017-09-13 00:00:00', '2017-09-14 00:00:00', '2017-09-15 00:00:00', '2017-09-18 00:00:00', '2017-09-19 00:00:00', '2017-09-20 00:00:00', '2017-09-21 00:00:00', '20
    17-09-22 00:00:00', '2017-09-25 00:00:00', '2017-09-26 00:00:00', '2017-09-27 00:00:00', '2017-09-28 00:00:00', '2017-09-29 00:00:00', '2017-10-02 00:00:00', '2017-10-03 00:00
    :00', '2017-10-04 19:44:27', '2017-10-04 19:45:00'])
    

    When 'next' is called first time self.data1 contains 30 bars as expected and then for every minute(timeframe of the self.data0) new bar is added also to the self.data1.

    Can you point me out what's wrong in my implementation? I tried to debug it, but the code is quite complex and I couldn't find the reason.

    It seems that I don't fully understand backtrader design and missed some important piece in my impolementation.

    P.S. I'm going to contribute my implementation to backtrader if there is any interest in having it there.


  • administrators

    @Ed-Bartosh said in IQFeed store and feed: help needed:

    '2017-10-04 19:44:27', '2017-10-04 19:45:00']

    A long shot:

    • The historical download for the 1st 30 bars is being done in Days
    • But from there onwards, your 2 data feeds share the data source in the background, hence data1 is being fed with the ticks (where each tick has 1-minute, as influenced by resampledata for data0)


  • @backtrader thanks. Can you suggest how to do it better? I was trying to avoid getting history in ticks as it's simply not needed for the strategy. How to keep data1 in daily time frame and get it updated only once per day?


  • administrators

    That depends entirely on what your provider offers to you. Some providers offer bars already in your desired timeframe, or at least in a close enough timeframe. If not, you need the ticks.

    Alternatives:

    • Periodically collect bars from a lower resolution than 1-Day and let them be resampled to day
      or
    • Wait until your provider has generated the 1-Day bar and collect it


  • @backtrader > Periodically collect bars from a lower resolution than 1-Day and let them be resampled to day

    That's what I was hoping to achieve. IQFeed provides data in a variety of time frames from ticks to months. I thought that as I'm using the same store and specify different timeframes I can download data in one time frame (Daily in this case) and then get it resampled automatically using lower time frame data.

    Looks like it's not the case and I still don't understand how achieve this. I feel like all the pieces are here (I can have data in any time frame I want), but I can't assemble the puzzle :)



  • @backtrader Can you explain the concept of store in a couple of words? For example, why store is a singleton? This is probably useful when you want to operate in a single stream of events from data providers and share it with multiple data feeds. However, when you can have several streams in different time frames it becomes difficult to implement as all feeds share the same store object. How would you suggest to deal with this? Do I need a store at all? Can I use my feed class as a simple proxy to the stream of bars in required timeframe from the data provider without using store or it would be agains of backtrader design?


  • administrators

    The store is the interface to the broker/data feed -call it provider- (it can have both or only 1) Broker and Data Feed instances don't know anything about the actual API of the provider. They receive "events" via queues or API calls after the store has removed the specifics (or most of them) from the events sent by the provider.

    It is a singleton because it seems logical that usually only one account (i.e: connection) is available and allowed by the provider. It's not a must.

    @Ed-Bartosh said in IQFeed store and feed: help needed:

    However, when you can have several streams in different time frames it becomes difficult to implement as all feeds share the same store object. How would you suggest to deal with this?

    Each stream generates events identified by some kind of identifier. The identifier is the key to a dictionary which is used to know to which data feed the event actually belongs to.

    @Ed-Bartosh said in IQFeed store and feed: help needed:

    Do I need a store at all?

    No.

    @Ed-Bartosh said in IQFeed store and feed: help needed:

    Can I use my feed class as a simple proxy to the stream of bars in required timeframe from the data provider without using store or it would be agains of backtrader design?

    You can (and probably should/must) do what best suits you. The design of backtrader doesn't have to be everyone's cup of tea.



  • @backtrader Thank you for the help! I went further with the implementation. I ended up using singleton store and one connection and one queue per time frame as IQFeed doesn't support streaming price data for the same symbol and multiple time frames on the same connection.

    One more question. Sometimes I'm getting ValueError traceback in backtrader code:

    >>> rewind: -1
    >>> rewind: -1
    >>> rewind: -1
    >>> rewind: -1
    >>> rewind: -1
    >>> rewind: -1
    >>> rewind: -1
    >>>__len__ -1
    Traceback (most recent call last):
      File "./test.py", line 425, in <module>
        sys.exit(main(sys.argv))
      File "./test.py", line 419, in main
        cerebro.run()
      File "backtrader\cerebro.py", line 1142, in run
        runstrat = self.runstrategies(iterstrat)
      File "backtrader\cerebro.py", line 1310, in runstrategies
        self._runnext(runstrats)
      File "backtrader\cerebro.py", line 1641, in _runnext
        strat._next()
      File "backtrader\strategy.py", line 325, in _next
        super(Strategy, self)._next()
      File "backtrader\lineiterator.py", line 255, in _next
        clock_len = self._clk_update()
      File "backtrader\strategy.py", line 305, in _clk_update
        newdlens = [len(d) for d in self.datas]
      File "backtrader\lineseries.py", line 468, in __len__
        return len(self.lines)
      File "backtrader\lineseries.py", line 221, in __len__
        return len(self.lines[0])
    ValueError: __len__() should return >= 0 
    

    It breaks as LineBuffer.len returns self.lencount, which is negative due to this code in LineBuffer.rewind: self.lencount -= size.

    Can you make an educated guess what could be the reason for this?


  • administrators

    You are probably replaying data and at the same time resampling. And when the data feeds don't fully synchronize that breaks.

    It's heritage from the original design which was upgraded trying to no break the existing behavior. A clean re-implementation of that would be needed.



  • I don't have any resampling nor replaying in the code for sure. Here is how feeds are added:

    # Create a cerebro entity
        cerebro = bt.Cerebro(maxcpus=1)
    
        # Set broker
        broker = bt.brokers.IBBroker(host=args.host, port=args.port,
                                     clientId=args.clientId, reconnect=args.reconnect,
                                     timeout=args.timeout, _debug=args.debug)
        cerebro.setbroker(broker)
    
        # Create store
        store = bt.stores.IQFeedStore(reconnect=args.reconnect,
                                      timeout=args.timeout,
                                      debug=args.debug)
        # Add data feeds
        for stock in Strategy.params.stocks:
            ticker = stock.split('-')[0]
    
            data_min = bt.feeds.IQFeed(dataname=ticker, name="%s_min" % ticker, tz='US/Eastern',
                                       timeframe=bt.TimeFrame.Minutes)
            cerebro.adddata(data_min)
    
            hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime()
            data_day = bt.feeds.IQFeed(dataname=ticker, name="%s_day" % ticker, tz='UTC', fromdate=hist_start_date,
                                       timeframe=bt.TimeFrame.Days, historical=True)
            cerebro.adddata(data_day)
    
        # Print out the starting conditions
        print 'Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()
    
        # Add a strategy
        cerebro.addstrategy(Strategy)
    
        # Run over everything
        cerebro.run()
    
        # Print out the final result
        print 'Final Portfolio Value: %.2f' % cerebro.broker.getvalue()
    

    Any other ideas why it happens? It doesn't happen too often, but happens quite regularly.


  • administrators

    It was an idea. The other is that something is implemented wrongly.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.