ibtest.py and vix?

Rami Ramich

hello,

i tried to get VIX from ibtest - no success.. How to get VIX in realtime?

python ./ibtest.py --data0 VIX-IND-CBOE --timeframe Seconds --port 4001
Server Version: 76
TWS Time at connection:20180307 16:05:05 EET

Strategy Created

Timezone from ContractDetails: CST
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usbond>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture.us>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.us>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:hkhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** DATA NOTIF: DELAYED
***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usbond>
***** STORE NOTIF: <error id=-1, errorCode=2108, errorMsg=Market data farm connection is inactive but should be available upon demand.usbond>

backtrader

Isn't that symbol for the continuous display in TWS? If yes, that's an artifact in TWS. You cannot get continuous futures over the API.

Rami Ramich

it's index, should be no problem to get it, qtpylib getting it from same 4001 port.
looks like there some issue.

backtrader

The symbol may need to be more specific (getting the currency added for example). I don't know the complete specification for that symbol.

backtrader

Another possibility is that the symbol was not yet trading real-time ... and the sample will therefore wait until it does to find out how much it can back-fill.

Add --historical

Rami Ramich

@backtrader
***** DATA NOTIF: DISCONNECTED
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds.us>
***** STORE NOTIF: <error id=16777217, errorCode=162, errorMsg=Historical Market Data Service error message:HMDS query returned no data: VIX@CBOE Trades>

there no trades - it's index.

backtrader

Most indices (there may be many I am unaware of) trade (in the sense that they are calculated) between A -> B and don't have real-time ticks between B -> A.

If VIX-IND-CBOE trades 24 hours a day non-stop, the situation may be different.

As stated above, the symbol specification may need extra data.

Rami Ramich

this info - from the terminal

0_1520439506728_Screen Shot 2018-03-07 at 18.18.06.png

similiar to sample with GBP.USD
0_1520439614405_Screen Shot 2018-03-07 at 18.19.59.png

Rami Ramich

@backtrader, here working sample from ezibpy

0_1520440114907_Screen Shot 2018-03-07 at 18.26.05.png

when i put into ibtest.py VIX-IND - getting following error:

***** STORE NOTIF: <error id=16777216, errorCode=200, errorMsg=No security definition has been found for the request>
***** STORE NOTIF: Ambiguous contract: none/multiple answers received
***** DATA NOTIF: DISCONNECTED

Rami Ramich

@backtrader, can you please advice how to handle such index with in backtrader (i think to move into this platform).

Rami Ramich

@backtrader as I understand there some difference between handling market data and real-time data (that not traded, like indexes) - I open bug in github - the author closed it. how can this issue be resolved?

backtrader

@rami-ramich said in ibtest.py and vix?:

as I understand there some difference between handling market data and real-time data (that not traded, like indexes)

There seems to be misunderstanding from your side, there is no such difference. The only thing that was stated above was about opening and closing times. Even if an index isn't directly traded, it is only active during trading times.

@rami-ramich said in ibtest.py and vix?:

I open bug in github - the author closed it. how can this issue be resolved?

Yes, closed. Opening an issue while discussing here is not very productive.

You are confusing issue with something you are not able to achieve. The ibtest.py is a sample with complex behavior intended to test many different behaviours and situations. It is not a one stop shop: "it is going to do exactly what I had in mind".

@rami-ramich said in ibtest.py and vix?:

when i put into ibtest.py VIX-IND - getting following error:

The Interactive Brokers API doesn't specify how applications gather the needed info for a ticker. If that notation works with some applications, it may not work with backtrader and may not work with many other applications. That notation is not related to how the information is passed to the IB API.

See here for more info, since you are not resampling and the test is probably failing due to it:

https://community.backtrader.com/topic/124/ib-backfilling-gap/2

Specify --resample and you will probably get data.

And this which shows how to do it with your own code and not rely on something which is possibly not intended for your use case such as ibtest.py

https://medium.com/@danjrod/interactive-brokers-in-python-with-backtrader-23dea376b2fc

Rami Ramich

@backtrader Thank you for a quick response.
I already tested with --resample option - no index data, when the market was open.
i will try when market will open with this sample from medium.

I very hope i will get this index at real time.

Thank you.

Rami Ramich

deleted...

Rami Ramich

I played with ibtest.py from sample folder.
historical data it's able to get, but real time - unable...
it's receive index as debug output <tickPrice tickerId=16777217, field=4, price=15.78, canAutoExecute=0>

Could you please point me to any sample that access to any index ticker?

python ./ibtest.py --data0 VIX-IND-CBOE --resample --timeframe Seconds --compression 1 --port 4001 --debug

Server Version: 76
TWS Time at connection:20180313 01:26:16 EET
<managedAccounts accountsList=U20zzzz,U21zzzz,>
<nextValidId orderId=1>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture.us>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.us>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds.us>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
<error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
<currentTime time=1520897517>
<contractDetails reqId=16777216, contractDetails=<ib.ext.ContractDetails.ContractDetails object at 0x1100eff60>>
<contractDetailsEnd reqId=16777216>

Strategy Created

Timezone from ContractDetails: CST
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture.us>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.us>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds.us>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** DATA NOTIF: DELAYED
<tickString tickerId=16777217, tickType=45, value=1520885690>
<tickPrice tickerId=16777217, field=4, price=15.78, canAutoExecute=0>
<tickSize tickerId=16777217, field=5, size=0>
<tickPrice tickerId=16777217, field=6, price=16.35, canAutoExecute=0>
<tickPrice tickerId=16777217, field=7, price=15.18, canAutoExecute=0>
<tickPrice tickerId=16777217, field=9, price=14.64, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=0>
<tickPrice tickerId=16777217, field=2, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=0>

python ./ibtest.py --data0 VIX-IND-CBOE --resample --timeframe Seconds --compression 1 --port 4001 --historical
Server Version: 76
TWS Time at connection:20180313 01:36:14 EET

Strategy Created

Timezone from ContractDetails: CST
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture.us>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds.us>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** DATA NOTIF: DELAYED
Data0, 0001, 736765.8229166666, 2018-03-12T14:45:00.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, nan
Data0, 0002, 736765.8229282408, 2018-03-12T14:45:01.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, nan
Data0, 0003, 736765.8229398148, 2018-03-12T14:45:02.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, nan
Data0, 0004, 736765.8229513889, 2018-03-12T14:45:03.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, nan
Data0, 0005, 736765.8229629629, 2018-03-12T14:45:04.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0006, 736765.822974537, 2018-03-12T14:45:05.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0007, 736765.8229861112, 2018-03-12T14:45:06.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0008, 736765.8229976852, 2018-03-12T14:45:07.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0009, 736765.8230092593, 2018-03-12T14:45:08.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0010, 736765.8230208333, 2018-03-12T14:45:09.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0011, 736765.8230324074, 2018-03-12T14:45:10.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0012, 736765.8230439815, 2018-03-12T14:45:11.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0013, 736765.8230555556, 2018-03-12T14:45:12.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0014, 736765.8230671296, 2018-03-12T14:45:13.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0015, 736765.8230787037, 2018-03-12T14:45:14.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0016, 736765.8230902777, 2018-03-12T14:45:15.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
Data0, 0017, 736765.8231018519, 2018-03-12T14:45:16.000000, 15.4, 15.4, 15.4, 15.4, 0.0, 0, 15.4
.
.
.
Data0, 1792, 736765.8436458333, 2018-03-12T15:14:51.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.785999999999998
Data0, 1793, 736765.8436574074, 2018-03-12T15:14:52.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.784
Data0, 1794, 736765.8436689815, 2018-03-12T15:14:53.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.782
Data0, 1795, 736765.8436805556, 2018-03-12T15:14:54.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
Data0, 1796, 736765.8436921296, 2018-03-12T15:14:55.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
Data0, 1797, 736765.8437037037, 2018-03-12T15:14:56.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
Data0, 1798, 736765.8437152777, 2018-03-12T15:14:57.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
Data0, 1799, 736765.8437268519, 2018-03-12T15:14:58.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
Data0, 1800, 736765.8437384259, 2018-03-12T15:14:59.000000, 15.78, 15.78, 15.78, 15.78, 0.0, 0, 15.779999999999998
***** DATA NOTIF: DISCONNECTED

I reviewed code a bit, i am not good in python yet, this piece of code looks responsible for this
<tickPrice tickerId=16777217, field=1, price=-1.0, canAutoExecute=1>

def tickPrice(self, msg):
    '''Cash Markets have no notion of "last_price"/"last_size" and the
    tracking of the price is done (industry de-facto standard at least with
    the IB API) following the BID price
    A RTVolume which will only contain a price is put into the client's
    queue to have a consistent cross-market interface
    '''
    # Used for "CASH" markets
    # The price field has been seen to be missing in some instances even if
    # "field" is 1
    tickerId = msg.tickerId
    if self.iscash[tickerId]:
        if msg.field == 1:  # Bid Price
            try:
                if msg.price == -1.0:
                    # seems to indicate the stream is halted for example in
                    # between 23:00 - 23:15 CET for FOREX
                    return

Rami Ramich

@backtrader, could you please help me with the issue?

backtrader

That only says that if the price is -1.0 the message has to be ignored. That's also for CASH (or CFD) markets (for example: Forex pairs like USD/JPY)

For an IND the subscription is for tickstring messages and not tickprice messages. And for real-time the code is waiting for a tickstring with price information and the message apparently ... doesn't arrive.

Rami Ramich

@backtrader thank you for quick answer.
Could you please advice how to fix or workaround it?
I need this IND for the algo.

I coded this - could you please tell if this OK?

def tickPrice(self, msg):
    '''Cash Markets have no notion of "last_price"/"last_size" and the
    tracking of the price is done (industry de-facto standard at least with
    the IB API) following the BID price

    A RTVolume which will only contain a price is put into the client's
    queue to have a consistent cross-market interface
    '''
    # Used for "CASH" markets
    # The price field has been seen to be missing in some instances even if
    # "field" is 1
    tickerId = msg.tickerId
    if self.iscash[tickerId]:
        if msg.field == 1 :  # Bid Price
            try:
                if msg.price == -1.0:
                    # seems to indicate the stream is halted for example in
                    # between 23:00 - 23:15 CET for FOREX
                    return
            except AttributeError:
                pass

            try:
                rtvol = RTVolume(price=msg.price, tmoffset=self.tmoffset)
                # print('rtvol with datetime:', rtvol.datetime)
            except ValueError:  # price not in message ...
                pass
            else:
                self.qs[tickerId].put(rtvol)
    elif msg.field == 4:
        try:
            if msg.price == -1.0:
                    # seems to indicate the stream is halted for example in
                    # between 23:00 - 23:15 CET for FOREX
                return
        except AttributeError:
            pass

        try:
            rtvol = RTVolume(price=msg.price, tmoffset=self.tmoffset)
                # print('rtvol with datetime:', rtvol.datetime)
        except ValueError:  # price not in message ...
             pass
        else:
            self.qs[tickerId].put(rtvol)

python ./ibtest.py --data0 VIX-IND-CBOE --data1 QQQ-STK --resample --timeframe Minutes --port 4001 --no-backfill

strange, it's different time.... on 2 items

Data0, 0361, 736768.8034722222, 2018-03-15T14:17:00.000000, 16.6, 16.63, 16.6, 16.62, 0.0, 0, 16.610000000000003
Data1, 0675, 736768.8034722222, 2018-03-15T15:17:00.000000, 171.48, 171.5, 171.44, 171.46, 399.0, 0, nan
Data0, 0362, 736768.8041666667, 2018-03-15T14:18:00.000000, 16.65, 16.79, 16.65, 16.79, 0.0, 0, 16.656
Data1, 0676, 736768.8041666667, 2018-03-15T15:18:00.000000, 171.45, 171.45, 171.34, 171.37, 703.0, 0, nan
Data0, 0363, 736768.8048611111, 2018-03-15T14:19:00.000000, 16.75, 16.87, 16.75, 16.87, 0.0, 0, 16.696
Data1, 0677, 736768.8048611111, 2018-03-15T15:19:00.000000, 171.37, 171.37, 171.26, 171.28, 451.0, 0, nan
Data0, 0364, 736768.8055555555, 2018-03-15T14:20:00.000000, 16.84, 16.84, 16.8, 16.8, 0.0, 0, 16.736
Data1, 0678, 736768.8055555555, 2018-03-15T15:20:00.000000, 171.27, 171.34, 171.26, 171.34, 162.0, 0, nan
Data0, 0365, 736768.80625, 2018-03-15T14:21:00.000000, 16.76, 16.81, 16.71, 16.71, 0.0, 0, 16.758000000000003
Data1, 0679, 736768.80625, 2018-03-15T15:21:00.000000, 171.36, 171.43, 171.35, 171.41, 410.0, 0, nan

and get error... without --resample

Data1, 0676, 736768.8041666667, 2018-03-15T15:18:00.000000, 171.37, 171.37, 171.26, 171.28, 428, 0, nan
Traceback (most recent call last):
File "./ibtest.py", line 558, in <module>
runstrategy()
File "./ibtest.py", line 336, in runstrategy
cerebro.run(exactbars=args.exactbars)
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/cerebro.py", line 1295, in runstrategies
self._runnext(runstrats)
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/cerebro.py", line 1626, in _runnext
strat._next()
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/strategy.py", line 325, in _next
super(Strategy, self)._next()
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/lineiterator.py", line 258, in _next
indicator._next()
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/lineiterator.py", line 258, in _next
indicator._next()
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/lineiterator.py", line 275, in _next
self.next()
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/indicators/basicops.py", line 356, in next
math.fsum(self.data.get(size=self.p.period)) / self.p.period
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/lineseries.py", line 241, in get
return self.lines[line].get(ago, size=size)
File "/Users/roman/anaconda3/envs/newenv/lib/python3.6/site-packages/backtrader/linebuffer.py", line 182, in get
return list(islice(self.array, start, end))
ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys.maxsize.

backtrader

@rami-ramich said in ibtest.py and vix?:

Could you please advice how to fix or workaround it?

VIX is not an index which has ever been accessed over here.

@rami-ramich said in ibtest.py and vix?:

and get error... without --resample

It's only a sample to test different behaviors. It's not a bulletproof script to test strategies.

@rami-ramich said in ibtest.py and vix?:

I coded this - could you please tell if this OK?
def tickPrice(self, msg):

As expressed above. No access to your VIX.

Rami Ramich

@backtrader my code modification is working, i have access to VIX-IND-CBOE data, but from your point of view modifications are OK? can you merge into main branch?

the crash stack i provided happen inside of backtrader framework.

ibtest.py and vix?

python ./ibtest.py --data0 VIX-IND-CBOE --timeframe Seconds --port 4001 Server Version: 76 TWS Time at connection:20180307 16:05:05 EET

Strategy Created

Strategy Created

python ./ibtest.py --data0 VIX-IND-CBOE --resample --timeframe Seconds --compression 1 --port 4001 --historical Server Version: 76 TWS Time at connection:20180313 01:36:14 EET

Strategy Created

python ./ibtest.py --data0 VIX-IND-CBOE --timeframe Seconds --port 4001
Server Version: 76
TWS Time at connection:20180307 16:05:05 EET

python ./ibtest.py --data0 VIX-IND-CBOE --resample --timeframe Seconds --compression 1 --port 4001 --historical
Server Version: 76
TWS Time at connection:20180313 01:36:14 EET