How to process data on CFD (Live IB)

[ealvarpe]

I´m trying to process data from IB on CFD products, but I can´t get it done.
Basically, the same code if I try to process "SPY-STK-SMART-USD" works correctly. But when I set the security to "IBUS500-CFD-SMART" doesn´t get any data processed.
Any ideas? (I have the last version of backtrader - 1.9.40.116 - that I think has the support for CFD´s)

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime
import backtrader as bt
from http.cookiejar import _debug


# Create a Stratey
class IBFwkLive(bt.Strategy):

    def __init__(self):
        self.datastatus = 0
        self.orderid = list()
        
    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
        if status == data.LIVE:
            self.datastatus = 1

    def notify_store(self, msg, *args, **kwargs):
        print('*' * 5, 'STORE NOTIF:', msg)

    def notify_order(self, order):
        if order.status in [order.Completed, order.Cancelled, order.Rejected]:
            self.order = None

        print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
        print(order)
        print('-' * 50, 'ORDER END')

    def notify_trade(self, trade):
        print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
        print(trade)
        print('-' * 50, 'TRADE END')

    def prenext(self):
        self.next(frompre=True)               
        
        
    def next(self, frompre=False):
        
        txt = list()
        txt.append('Data0')
        txt.append('%04d' % len(self.data0))
        dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
        txt.append('{}'.format(self.data.datetime[0]))
        txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
        txt.append('{}'.format(self.data.open[0]))
        txt.append('{}'.format(self.data.high[0]))
        txt.append('{}'.format(self.data.low[0]))
        txt.append('{}'.format(self.data.close[0]))
        print(', '.join(txt))

        if len(self.datas) > 1 and len(self.data1):
            txt = list()
            txt.append('Data1')
            txt.append('%04d' % len(self.data1))
            dtfmt = '%Y-%m-%dT%H:%M:%S.%f'
            txt.append('{}'.format(self.data1.datetime[0]))
            txt.append('%s' % self.data1.datetime.datetime(0).strftime(dtfmt))
            txt.append('{}'.format(self.data1.open[0]))
            txt.append('{}'.format(self.data1.high[0]))
            txt.append('{}'.format(self.data1.low[0]))
            txt.append('{}'.format(self.data1.close[0]))
            print(', '.join(txt)) 

    def start(self):
        if self.data0.contractdetails is not None:
            print('Timezone from ContractDetails: {}'.format(
                  self.data0.contractdetails.m_timeZoneId))

        header = ['Datetime', 'Open', 'High', 'Low', 'Close']
        print(', '.join(header))

        self.done = False
        
if __name__ == '__main__':

    cerebro = bt.Cerebro()
    
    storekwargs = dict(
        host='127.0.0.1', port=7496,
        clientId=None, reconnect=100, timeout=3.0, _debug=False)
    ibstore = bt.stores.IBStore(**storekwargs)
    broker = ibstore.getbroker()
    cerebro.setbroker(broker)
    
    
    datakwargs = dict(
        timeframe=bt.TimeFrame.Ticks, compression=1,
        rtbar=False,
        what='BID')
    #data0 = ibstore.getdata(dataname='SPY-STK-SMART-USD', **datakwargs)
    data0 = ibstore.getdata(dataname='IBUS500-CFD-SMART', **datakwargs)
    cerebro.replaydata(data0, timeframe=bt.TimeFrame.Days, compression=1)

    cerebro.addstrategy(IBFwkLive)
    cerebro.run(exactbars=1)

[backtrader]

The code contains notify_store and notify_data, both of which produce a print out. It might be helpful to understand why that data isn't coming through.

Giving the same symbol to the ibtestsample in the source can also help. For example:

$ ./ibtest.py --data0 IBUS500-CFD-SMART --resample --timeframe Seconds --compression 5 --debug

[ealvarpe]

I´ve tried to do a debug an the only thing I receive is

***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** DATA NOTIF: DELAYED
<tickPrice tickerId=16777217, field=9, price=2357.0, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=0>
<tickPrice tickerId=16777217, field=2, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=0>

but the data never get processed. Is like if the next() method never gets called. Also it seems like with CFD I cannot backfill data, is it like this?
Instead, when I try to use the same code for SPY it works and receive backfill data and gets Live.

[backtrader]

You may still run the sample as pointed out above and post the output from the beginning, (obviously you would want to blank your account number) where things like the actually retrieved contract (if any) is shown in the debug messages.

[ealvarpe]

I´ve run the example with this output

Server Version: 76
TWS Time at connection:20170411 01:17:04 Greenwich Mean Time
<managedAccounts accountsList=XXXXX>
<nextValidId orderId=1>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
<currentTime time=1491869824>
<contractDetails reqId=16711216, contractDetails=<ib.ext.ContractDetails.ContractDetails object at 0x000001CA8F005630>>
<contractDetailsEnd reqId=16711216>
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: GB
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** DATA NOTIF: DELAYED
<error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:eufarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
<tickPrice tickerId=16777217, field=9, price=2357.0, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=0>
<tickPrice tickerId=16777217, field=2, price=-1.0, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=0>

But as the market is now close for the CFD´s, not sure if I can expect something else. I´ll try again tomorrow during the open market.

[ealvarpe]

I can see the tick prices are coming, but next() is never called within the strategy
Nether the autofilling seems to work
But when trying with, for example "SPY-STK-SMART-USD" works perfectly...

Server Version: 76
TWS Time at connection:20170411 19:30:55 Greenwich Mean Time
<managedAccounts accountsList=XXXXXX>
<nextValidId orderId=1>
<currentTime time=1491935455>
<contractDetails reqId=16777216, contractDetails=<ib.ext.ContractDetails.ContractDetails object at 0x00000280E3ED55F8>>
<contractDetailsEnd reqId=16777216>
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: GB
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** DATA NOTIF: DELAYED
<error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:eufarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
<tickPrice tickerId=16777217, field=9, price=2357.13, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=2350.23, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=300>
<tickPrice tickerId=16777217, field=2, price=2350.48, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=300>
<tickSize tickerId=16777217, field=0, size=300>
<tickSize tickerId=16777217, field=3, size=300>
<tickPrice tickerId=16777217, field=1, price=2349.98, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=300>
<tickPrice tickerId=16777217, field=2, price=2350.23, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=300>
<tickPrice tickerId=16777217, field=1, price=2350.23, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=300>
<tickPrice tickerId=16777217, field=2, price=2350.48, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=300>
<tickPrice tickerId=16777217, field=1, price=2349.98, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=300>
<tickPrice tickerId=16777217, field=2, price=2350.23, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=300>

[backtrader]

STK and CFD are two different type of assets. How to handle requests for STK is well known with several examples and samples.

CFD is in comparison a recent addition to the Interactive Brokers product line and there is no indication (at least not one that has been found) as to how to manage the assets (No, IB API Reference doesn't help either).

@skytrading54 reported (Community - Anyone using Backtrader for live trading) the lack of support and some code was added to 1st parse the contract specification and 2nd to try to request the prices, which according to the input in the thread had to be handled as a CASH asset.

There were no further reports about it not working. But further inspection of the code could have the CFD processing missing in one point.

In any case and because the OandaBroker also needed a patch, release 1.9.41.116 contains a commit to also managed CFD as cash during rqmMktData.

You can install this release and give it a try.

[ealvarpe]

Unfortunately no luck with the last version. Same behaviour: it receive the ticks, but next() is never called.
I´ll update the thread if I found the reason.

[ealvarpe]

ok, it works. Seems that I was not working with the last version as I thought.
The problem I see with CFD's is that for Index CFD's (like IBUS500) works as IB is providing market data.

But with stock CFD's (like SPY-CFD-SMART) doesn't. The IB system defers to the underlying (SPY-STK-SMART-USD).
One tricky way to make it work is to add the datafeed from the underlying and only before buying or selling any amount, change the contract by the CFD one. With this, all your algorithm can relay on the STK and have the leverage power of CFD's when really accessing the market.

[backtrader]

Good to know. The Visual Chart data feed meant for trading has the following parameter

      - ``tradename`` (default: ``None``)
        Continous futures cannot be traded but are ideal for data tracking. If
        this parameter is supplied it will be the name of the current future
        which will be the trading asset. Example:

        - 001ES -> ES-Mini continuous supplied as ``dataname``

        - ESU16 -> ES-Mini 2016-09. If this is supplied in ``tradename`` it
          will be the trading asset.

A similar approach could be thought of for the CFD case described above. Let's confirm the use case:

• dataname would be SPY-STK-SMART-USD, which is used for data tracking

• tradename would be SPY-CFD-SMART, which is used for trading operations

Hopefully this is what you described above

[ealvarpe]

yes, it´s exactly that.

[backtrader]

Available with this commit in the development branch: https://github.com/mementum/backtrader/commit/aba90a3ff4e115e3373ea15389410adf10f2b4cf

The regular IB functionality doesn't seem broken by these changes.