IQFeed store and feed: help needed
-
Hi @backtrader,
I'm trying to implement IQFeed support for backtrader: iqfed branch@ my github repo
It started to work, but I can't figure out one thing:
For the strategy with 2 dataframes (see below) the data for the second dataframe (daily) is continuously growing.... store = bt.stores.IQFeedStore(tickers=tickers, fields=fields, reconnect=args.reconnect, timeout=args.timeout, debug=args.debug) # Add data feeds for ticker in tickers: data_min = store.getdata(dataname=ticker, name="%s_min" % ticker, tz='UTC', timeframe=bt.TimeFrame.Ticks) data_min = cerebro.resampledata(data_min, timeframe=bt.TimeFrame.Minutes) hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 1)[0].to_pydatetime() data_day = store.getdata(dataname=ticker, name="%s_day" % ticker, tz='UTC', fromdate=hist_start_date, timeframe=bt.TimeFrame.Days, backfill_start=True) cerebro.adddata(data_day) ...
I've added this code to the 'next' method of my strategy:
print "len:", len(self.datas[1])) #, Series(self.data1.high.get(size=30))) print(">>> dates:", [bt.utils.date.num2date(date).strftime("%Y-%m-%d %H:%M:%S") for date in self.data1.datetime.get(size=len(self.data1))])
Here is the output:
len: 32 dates: ['2017-08-22 00:00:00', '2017-08-23 00:00:00', '2017-08-24 00:00:00', '2017-08-25 00:00:00', '2017-08-28 00:00:00', '2017-08-29 00:00:00', '2017-08-30 00:00:00' , '2017-08-31 00:00:00', '2017-09-01 00:00:00', '2017-09-05 00:00:00', '2017-09-06 00:00:00', '2017-09-07 00:00:00', '2017-09-08 00:00:00', '2017-09-11 00:00:00', '2017-09-12 00:00:00', '2017-09-13 00:00:00', '2017-09-14 00:00:00', '2017-09-15 00:00:00', '2017-09-18 00:00:00', '2017-09-19 00:00:00', '2017-09-20 00:00:00', '2017-09-21 00:00:00', '20 17-09-22 00:00:00', '2017-09-25 00:00:00', '2017-09-26 00:00:00', '2017-09-27 00:00:00', '2017-09-28 00:00:00', '2017-09-29 00:00:00', '2017-10-02 00:00:00', '2017-10-03 00:00 :00', '2017-10-04 19:44:27', '2017-10-04 19:45:00'])
When 'next' is called first time self.data1 contains 30 bars as expected and then for every minute(timeframe of the self.data0) new bar is added also to the self.data1.
Can you point me out what's wrong in my implementation? I tried to debug it, but the code is quite complex and I couldn't find the reason.
It seems that I don't fully understand backtrader design and missed some important piece in my impolementation.
P.S. I'm going to contribute my implementation to backtrader if there is any interest in having it there.
-
@Ed-Bartosh said in IQFeed store and feed: help needed:
'2017-10-04 19:44:27', '2017-10-04 19:45:00']
A long shot:
- The historical download for the 1st 30 bars is being done in
Days
- But from there onwards, your 2 data feeds share the data source in the background, hence
data1
is being fed with the ticks (where each tick has 1-minute, as influenced byresampledata
fordata0
)
- The historical download for the 1st 30 bars is being done in
-
@backtrader thanks. Can you suggest how to do it better? I was trying to avoid getting history in ticks as it's simply not needed for the strategy. How to keep data1 in daily time frame and get it updated only once per day?
-
That depends entirely on what your provider offers to you. Some providers offer bars already in your desired timeframe, or at least in a close enough timeframe. If not, you need the ticks.
Alternatives:
- Periodically collect bars from a lower resolution than
1-Day
and let them be resampled to day
or - Wait until your provider has generated the
1-Day
bar and collect it
- Periodically collect bars from a lower resolution than
-
@backtrader > Periodically collect bars from a lower resolution than 1-Day and let them be resampled to day
That's what I was hoping to achieve. IQFeed provides data in a variety of time frames from ticks to months. I thought that as I'm using the same store and specify different timeframes I can download data in one time frame (Daily in this case) and then get it resampled automatically using lower time frame data.
Looks like it's not the case and I still don't understand how achieve this. I feel like all the pieces are here (I can have data in any time frame I want), but I can't assemble the puzzle :)
-
@backtrader Can you explain the concept of store in a couple of words? For example, why store is a singleton? This is probably useful when you want to operate in a single stream of events from data providers and share it with multiple data feeds. However, when you can have several streams in different time frames it becomes difficult to implement as all feeds share the same store object. How would you suggest to deal with this? Do I need a store at all? Can I use my feed class as a simple proxy to the stream of bars in required timeframe from the data provider without using store or it would be agains of backtrader design?
-
The store is the interface to the broker/data feed -call it provider- (it can have both or only 1) Broker and Data Feed instances don't know anything about the actual API of the provider. They receive "events" via queues or API calls after the store has removed the specifics (or most of them) from the events sent by the provider.
It is a singleton because it seems logical that usually only one account (i.e: connection) is available and allowed by the provider. It's not a must.
@Ed-Bartosh said in IQFeed store and feed: help needed:
However, when you can have several streams in different time frames it becomes difficult to implement as all feeds share the same store object. How would you suggest to deal with this?
Each stream generates events identified by some kind of identifier. The identifier is the key to a dictionary which is used to know to which data feed the event actually belongs to.
@Ed-Bartosh said in IQFeed store and feed: help needed:
Do I need a store at all?
No.
@Ed-Bartosh said in IQFeed store and feed: help needed:
Can I use my feed class as a simple proxy to the stream of bars in required timeframe from the data provider without using store or it would be agains of backtrader design?
You can (and probably should/must) do what best suits you. The design of backtrader doesn't have to be everyone's cup of tea.
-
@backtrader Thank you for the help! I went further with the implementation. I ended up using singleton store and one connection and one queue per time frame as IQFeed doesn't support streaming price data for the same symbol and multiple time frames on the same connection.
One more question. Sometimes I'm getting ValueError traceback in backtrader code:
>>> rewind: -1 >>> rewind: -1 >>> rewind: -1 >>> rewind: -1 >>> rewind: -1 >>> rewind: -1 >>> rewind: -1 >>>__len__ -1 Traceback (most recent call last): File "./test.py", line 425, in <module> sys.exit(main(sys.argv)) File "./test.py", line 419, in main cerebro.run() File "backtrader\cerebro.py", line 1142, in run runstrat = self.runstrategies(iterstrat) File "backtrader\cerebro.py", line 1310, in runstrategies self._runnext(runstrats) File "backtrader\cerebro.py", line 1641, in _runnext strat._next() File "backtrader\strategy.py", line 325, in _next super(Strategy, self)._next() File "backtrader\lineiterator.py", line 255, in _next clock_len = self._clk_update() File "backtrader\strategy.py", line 305, in _clk_update newdlens = [len(d) for d in self.datas] File "backtrader\lineseries.py", line 468, in __len__ return len(self.lines) File "backtrader\lineseries.py", line 221, in __len__ return len(self.lines[0]) ValueError: __len__() should return >= 0
It breaks as LineBuffer.len returns self.lencount, which is negative due to this code in LineBuffer.rewind: self.lencount -= size.
Can you make an educated guess what could be the reason for this?
-
You are probably replaying data and at the same time resampling. And when the data feeds don't fully synchronize that breaks.
It's heritage from the original design which was upgraded trying to no break the existing behavior. A clean re-implementation of that would be needed.
-
I don't have any resampling nor replaying in the code for sure. Here is how feeds are added:
# Create a cerebro entity cerebro = bt.Cerebro(maxcpus=1) # Set broker broker = bt.brokers.IBBroker(host=args.host, port=args.port, clientId=args.clientId, reconnect=args.reconnect, timeout=args.timeout, _debug=args.debug) cerebro.setbroker(broker) # Create store store = bt.stores.IQFeedStore(reconnect=args.reconnect, timeout=args.timeout, debug=args.debug) # Add data feeds for stock in Strategy.params.stocks: ticker = stock.split('-')[0] data_min = bt.feeds.IQFeed(dataname=ticker, name="%s_min" % ticker, tz='US/Eastern', timeframe=bt.TimeFrame.Minutes) cerebro.adddata(data_min) hist_start_date = bdate_range(end=datetime.datetime.now(), periods=Strategy.params.stdperiod + 2)[0].to_pydatetime() data_day = bt.feeds.IQFeed(dataname=ticker, name="%s_day" % ticker, tz='UTC', fromdate=hist_start_date, timeframe=bt.TimeFrame.Days, historical=True) cerebro.adddata(data_day) # Print out the starting conditions print 'Starting Portfolio Value: %.2f' % cerebro.broker.getvalue() # Add a strategy cerebro.addstrategy(Strategy) # Run over everything cerebro.run() # Print out the final result print 'Final Portfolio Value: %.2f' % cerebro.broker.getvalue()
Any other ideas why it happens? It doesn't happen too often, but happens quite regularly.
-
It was an idea. The other is that something is implemented wrongly.
-
Hi Ed,Daniel
I know this thread is pretty old, but probably you may remember how you've overcome the above problem.
Exactly the same error appears right now (the same stack as well) while live trading using IB with multiple datas using RTVolume prices (rtbar=False, timeframe=Seconds, compression=5, no resampling, no replay for all datas):
File "/home/vladisld_trade/run_multi_ib.py", line 144, in run cerebro.run() File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "/home/vladisld_trade/backtrader/backtrader/cerebro.py", line 1630, in _runnext strat._next() File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 347, in _next super(Strategy, self)._next() File "/home/vladisld_trade/backtrader/backtrader/lineiterator.py", line 260, in _next clock_len = self._clk_update() File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 327, in _clk_update newdlens = [len(d) for d in self.datas] File "/home/vladisld_trade/backtrader/backtrader/strategy.py", line 327, in <listcomp> newdlens = [len(d) for d in self.datas] File "/home/vladisld_trade/backtrader/backtrader/lineseries.py", line 464, in __len__ return len(self.lines) File "/home/vladisld_trade/backtrader/backtrader/lineseries.py", line 220, in __len__ return len(self.lines[0]) ValueError: __len__() should return >= 0
The following events were preceding the crash:
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds> <error id=-1, errorCode=2158, errorMsg=Sec-def data farm connection is OK:secdefeu> <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:usfarm> <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:usfarm> <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm> <marketDataType reqId=16777217, marketDataType=1> <tickString tickerId=16777217, tickType=45, value=1581108818> <tickPrice tickerId=16777217, field=4, price=6.64, canAutoExecute=0> <tickSize tickerId=16777217, field=5, size=1> <tickSize tickerId=16777217, field=5, size=1> <tickSize tickerId=16777217, field=8, size=18642> <tickPrice tickerId=16777217, field=6, price=6.66, canAutoExecute=0> <tickPrice tickerId=16777217, field=7, price=6.6, canAutoExecute=0> <tickPrice tickerId=16777217, field=9, price=6.64, canAutoExecute=0> <tickPrice tickerId=16777217, field=14, price=6.64, canAutoExecute=0> <tickPrice tickerId=16777217, field=1, price=6.64, canAutoExecute=1> <tickSize tickerId=16777217, field=0, size=316> <tickPrice tickerId=16777217, field=2, price=6.65, canAutoExecute=1> <tickSize tickerId=16777217, field=3, size=491> <tickSize tickerId=16777217, field=0, size=316> <tickSize tickerId=16777217, field=3, size=491> <tickString tickerId=16777217, tickType=32, value=AIXBQTMHJ> <tickString tickerId=16777217, tickType=33, value=ACIXBQTMHJ> <marketDataType reqId=16777219, marketDataType=1> <tickPrice tickerId=16777219, field=1, price=4.37, canAutoExecute=1> <tickSize tickerId=16777219, field=0, size=0> <tickString tickerId=16777219, tickType=45, value=1581108819> <tickPrice tickerId=16777219, field=4, price=4.39, canAutoExecute=0> <tickSize tickerId=16777219, field=5, size=1> <tickSize tickerId=16777219, field=5, size=1> <tickSize tickerId=16777219, field=8, size=978454> <tickPrice tickerId=16777219, field=6, price=4.42, canAutoExecute=0> <tickPrice tickerId=16777219, field=7, price=4.26, canAutoExecute=0> <tickPrice tickerId=16777219, field=9, price=4.2, canAutoExecute=0> <tickPrice tickerId=16777219, field=14, price=4.27, canAutoExecute=0> <tickPrice tickerId=16777219, field=1, price=4.39, canAutoExecute=1> <tickSize tickerId=16777219, field=0, size=6611> <tickPrice tickerId=16777219, field=2, price=4.4, canAutoExecute=1> <tickSize tickerId=16777219, field=3, size=8356> <tickSize tickerId=16777219, field=0, size=6611> <tickSize tickerId=16777219, field=3, size=8356> <tickString tickerId=16777219, tickType=32, value=ACIXBQTMHJ> <tickString tickerId=16777219, tickType=33, value=ACIXBQTMHJ> <marketDataType reqId=16777221, marketDataType=1> <tickString tickerId=16777221, tickType=45, value=1581108819> <tickPrice tickerId=16777221, field=4, price=56.93, canAutoExecute=0> <tickSize tickerId=16777221, field=5, size=1> <tickSize tickerId=16777221, field=5, size=1> <tickSize tickerId=16777221, field=8, size=179048> <tickPrice tickerId=16777221, field=6, price=58.0, canAutoExecute=0> <tickPrice tickerId=16777221, field=7, price=56.77, canAutoExecute=0> <tickPrice tickerId=16777221, field=9, price=58.68, canAutoExecute=0> <tickPrice tickerId=16777221, field=14, price=57.97, canAutoExecute=0> <tickPrice tickerId=16777221, field=1, price=56.93, canAutoExecute=1> <tickSize tickerId=16777221, field=0, size=6> <tickPrice tickerId=16777221, field=2, price=56.94, canAutoExecute=1> <tickSize tickerId=16777221, field=3, size=43> <tickSize tickerId=16777221, field=0, size=6> <tickSize tickerId=16777221, field=3, size=43> <tickString tickerId=16777221, tickType=32, value=E> <tickString tickerId=16777221, tickType=33, value=IBQWTEJ> <marketDataType reqId=16777223, marketDataType=1> <tickString tickerId=16777223, tickType=45, value=1581108818> <tickPrice tickerId=16777223, field=4, price=12.15, canAutoExecute=0> <tickSize tickerId=16777223, field=5, size=12> <tickSize tickerId=16777223, field=5, size=12> <tickSize tickerId=16777223, field=8, size=97877> <tickPrice tickerId=16777223, field=6, price=12.17, canAutoExecute=0> <tickPrice tickerId=16777223, field=7, price=11.98, canAutoExecute=0> <tickPrice tickerId=16777223, field=9, price=12.12, canAutoExecute=0> <tickPrice tickerId=16777223, field=14, price=12.14, canAutoExecute=0> <tickPrice tickerId=16777223, field=1, price=12.14, canAutoExecute=1> <tickSize tickerId=16777223, field=0, size=378> <tickPrice tickerId=16777223, field=2, price=12.15, canAutoExecute=1> <tickSize tickerId=16777223, field=3, size=249> <tickSize tickerId=16777223, field=0, size=378> <tickSize tickerId=16777223, field=3, size=249> <tickString tickerId=16777223, tickType=32, value=ACIXBQTMHJ> <tickString tickerId=16777223, tickType=33, value=AIXQTMHJ> <marketDataType reqId=16777225, marketDataType=1> <tickString tickerId=16777225, tickType=45, value=1581108819> <tickPrice tickerId=16777225, field=4, price=49.78, canAutoExecute=0> <tickSize tickerId=16777225, field=5, size=2> <tickSize tickerId=16777225, field=5, size=2> <tickSize tickerId=16777225, field=8, size=497234> <tickPrice tickerId=16777225, field=6, price=50.67, canAutoExecute=0> <tickPrice tickerId=16777225, field=7, price=48.63, canAutoExecute=0> <tickPrice tickerId=16777225, field=9, price=49.32, canAutoExecute=0> <tickPrice tickerId=16777225, field=14, price=48.93, canAutoExecute=0> <tickPrice tickerId=16777225, field=1, price=49.78, canAutoExecute=1> <tickSize tickerId=16777225, field=0, size=38> <tickPrice tickerId=16777225, field=2, price=49.79, canAutoExecute=1> <tickSize tickerId=16777225, field=3, size=98> <tickSize tickerId=16777225, field=0, size=38> <tickSize tickerId=16777225, field=3, size=98> <tickString tickerId=16777225, tickType=32, value=QWT> <tickString tickerId=16777225, tickType=33, value=ACIBQWTMEJ> <marketDataType reqId=16777227, marketDataType=1> <tickPrice tickerId=16777227, field=2, price=13.1, canAutoExecute=1> <tickSize tickerId=16777227, field=3, size=0> <tickString tickerId=16777227, tickType=45, value=1581108817> <tickPrice tickerId=16777227, field=4, price=13.15, canAutoExecute=0> <tickSize tickerId=16777227, field=5, size=1> <tickSize tickerId=16777227, field=5, size=1> <tickPrice tickerId=16777227, field=14, price=13.79, canAutoExecute=0> <tickSize tickerId=16777227, field=8, size=3442> <tickPrice tickerId=16777227, field=6, price=13.79, canAutoExecute=0> <tickPrice tickerId=16777227, field=7, price=13.02, canAutoExecute=0> <tickPrice tickerId=16777227, field=9, price=13.79, canAutoExecute=0> <tickPrice tickerId=16777225, field=4, price=49.79, canAutoExecute=0> <tickSize tickerId=16777225, field=5, size=2> <tickSize tickerId=16777225, field=8, size=497236> <tickPrice tickerId=16777227, field=1, price=13.13, canAutoExecute=1> <tickSize tickerId=16777227, field=0, size=5> <tickPrice tickerId=16777227, field=2, price=13.17, canAutoExecute=1> <tickSize tickerId=16777227, field=3, size=6> <tickSize tickerId=16777227, field=0, size=5> <tickSize tickerId=16777227, field=3, size=6> <tickString tickerId=16777227, tickType=32, value=XB> <tickString tickerId=16777227, tickType=33, value=XM> <tickString tickerId=16777221, tickType=48, value=56.93;4;1581108819203;179276;57.21285544;false> <tickString tickerId=16777221, tickType=48, value=;0;1581108819206;179276;57.21285544;true> <tickString tickerId=16777221, tickType=48, value=;0;1581108819207;179276;57.21285544;true> <tickPrice tickerId=16777221, field=1, price=56.92, canAutoExecute=1> <tickSize tickerId=16777221, field=0, size=69> <tickPrice tickerId=16777221, field=2, price=56.93, canAutoExecute=1> <tickSize tickerId=16777221, field=3, size=41> <tickSize tickerId=16777221, field=0, size=69> <tickSize tickerId=16777221, field=3, size=41> <tickString tickerId=16777221, tickType=32, value=IQWTE> <tickString tickerId=16777221, tickType=33, value=TE> <tickString tickerId=16777221, tickType=48, value=56.93;2;1581108819400;179278;57.21285224;false> <tickString tickerId=16777221, tickType=48, value=;0;1581108819401;179278;57.21285224;true> <tickString tickerId=16777225, tickType=48, value=49.78;37;1581108819401;497722;50.08132026;false> <tickString tickerId=16777225, tickType=48, value=;0;1581108819401;497722;50.08132026;true> <tickString tickerId=16777225, tickType=48, value=;0;1581108819403;497722;50.08132026;true> <tickSize tickerId=16777221, field=8, size=179049> <tickPrice tickerId=16777221, field=4, price=56.92, canAutoExecute=0> <tickSize tickerId=16777221, field=5, size=1> <tickSize tickerId=16777227, field=8, size=3446> <tickPrice tickerId=16777225, field=1, price=49.77, canAutoExecute=1> <tickSize tickerId=16777225, field=0, size=129> <tickPrice tickerId=16777225, field=2, price=49.78, canAutoExecute=1> <tickSize tickerId=16777225, field=3, size=84> <tickSize tickerId=16777225, field=0, size=129> <tickSize tickerId=16777225, field=3, size=84> <tickString tickerId=16777225, tickType=32, value=AIQWTE> <tickString tickerId=16777225, tickType=33, value=TE> <tickString tickerId=16777219, tickType=48, value=4.39;187;1581108819649;978667;4.35446026;false> <tickString tickerId=16777219, tickType=48, value=;0;1581108819650;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819650;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819652;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819652;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819653;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819653;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819654;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819655;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819656;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819656;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819659;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819660;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819661;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819661;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819669;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819670;978667;4.35446026;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819670;978667;4.35446026;true> <tickString tickerId=16777225, tickType=48, value=49.78;1;1581108819670;497723;50.08131965;true> <tickString tickerId=16777219, tickType=45, value=1581108820> <tickSize tickerId=16777219, field=5, size=39> <tickSize tickerId=16777219, field=8, size=978493> <tickString tickerId=16777225, tickType=45, value=1581108820> <tickPrice tickerId=16777225, field=4, price=49.78, canAutoExecute=0> <tickSize tickerId=16777225, field=5, size=1> <tickSize tickerId=16777225, field=5, size=1> <tickSize tickerId=16777225, field=8, size=497237> <tickSize tickerId=16777223, field=3, size=250> <tickString tickerId=16777219, tickType=48, value=4.39;24;1581108819900;978691;4.35446113;false> <tickString tickerId=16777219, tickType=48, value=;0;1581108819900;978691;4.35446113;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819900;978691;4.35446113;true> <tickString tickerId=16777219, tickType=48, value=;0;1581108819901;978691;4.35446113;true> <tickSize tickerId=16777219, field=0, size=6408> <tickSize tickerId=16777219, field=3, size=8511> <tickSize tickerId=16777221, field=0, size=77> <tickSize tickerId=16777221, field=3, size=10> <tickString tickerId=16777221, tickType=32, value=CIQWTMEJ> <tickString tickerId=16777221, tickType=33, value=WT> <tickSize tickerId=16777217, field=3, size=481> Traceback (most recent call last): ... <the stacktrace you saw above goes here> ...
Thanks
Vlad -
found the answer here: ERROR: len() should return >= 0 on liveFeed