Rollover multiple contracts [help needed]

[chewbacca]

Hey guys, I have some VIX futures data I need to backtest, it is separated by individual contracts.
I need to have 2 feeds: VX1 and VX2

Here is an example for 2010:
The test should start at 21 December 2019 (the day VXF10 becomes VX1 and VXG10 - VX2)

VXF10 - 2009-06-18 09:59:00 # Initially at year(test) start - VX1: VXF10 | VX2: VXG10
- 2010-01-19 16:15:00 LT 19th | EXP 20th January at 08:15 VXG10 becomes VX1, VXH10 becomes VX2

VXG10 - 2009-07-23 10:07:00 # VX1: VXG10 | VX2: VXH10
- 2010-02-16 16:15:00 LT 16th | EXP 17th February at 08:15 VXH10 becomes VX1, VXJ10 becomes VX2

VXH10 - 2009-08-20 15:58:00 # VX1: VXH10 | VX2: VXJ10
- 2010-03-16 16:15:00 LT 16th | EXP 17 March at 08:15 VXJ10 becomes VX1, VXK10 becomes VX2

VXJ10 - 2009-09-16 10:21:00 # VX1: VXJ10 | VX2: VXK10
- 2010-04-20 16:15:00 LT 20th | EXP 21 April at 08:15 VXK10 becomes VX1, VXM10 becomes VX2

VXK10 - 2009-09-25 13:54:00 # VX1: VXK10 | VX2: VXM10
- 2010-05-18 16:15:00 LT 18th | EXP 19 May at 08:15 VXM10 becomes VX1, VXN10 becomes VX2

VXM10 - 2009-10-23 14:03:00 # VX1: VXM10 | VX2: VXN10
- 2010-06-15 16:15:00 LT 15th | EXP 16 June at 08:15 VXN10 becomes VX1, VXQ10 becomes VX2

VXN10 - 2009-11-24 10:41:00 # VX1: VXN10 | VX2: VXQ10
- 2010-07-20 16:15:00 LT 20th | EXP 21 July at 08:15 VXQ10 becomes VX1, VXU10 becomes VX2

VXQ10 - 2009-12-22 10:08:00 # VX1: VXQ10 | VX2: VXU10
- 2010-08-17 16:15:00 LT 17th | EXP 18 August at 08:15 VXU10 becomes VX1, VXV10 becomes VX2

VXU10 - 2010-01-26 10:32:00 # VX1: VXU10 | VX2: VXV10
- 2010-09-14 16:15:00 LT 14th | EXP 15 September at 08:15 VXV10 becomes VX1, VXX10 becomes VX2

VXV10 - 2010-02-23 10:38:00 # VX1: VXV10 | VX2: VXX10
- 2010-10-19 16:15:00 LT 19th | EXP 20 October at 08:15 VXX10 becomes VX1, VXZ10 becomes VX2

VXX10 - 2010-03-23 15:57:00 # VX1: VXX10 | VX2: VXZ10
- 2010-11-16 16:15:00 LT 16th | EXP 17 November at 08:15 VXZ10 becomes VX1, VXF11 becomes VX2

VXZ10 - 2010-04-27 10:11:00 # VX1: VXZ10 | VX2: VXF11
- 2010-12-21 16:15:00 LT 21th | EXP 22 December at 08:15 VXF11 becomes VX1, VXG11 becomes VX2

2011
VXF11 - 2010-05-25 10:04:00 # VX1: VXF11 | VX2: VXG11
- 2011-01-18 16:15:00 LT 18th | EXP 19 January 2011

How do I use rollover with checkdate on these, each contract has its own expiration date, at which point VX1 and VX2 should change? Has someone done anything like this? How do I specify the expiry date of each contract for checkdate ?

I've been hitting my head against the wall on this for a few days. Hopefully someone's gonna help ...

Regards,
C

[chewbacca]

There's a typo in the previous post, but for some reason the forum doesn't have an "edit" function.

The test should start at 21 December 2009 (the day VXF10 becomes VX1 and VXG10 - VX2).

As far as documentation on rollover goes - there's no such example (with different expire dates for each contract)

[chewbacca]

Anyone gonna point me in the right direction? How to approach doing this?

[run-out]

Try using continuous contracts.

[chewbacca]

@run-out said in Rollover multiple contracts [help needed]:

Try using continuous contracts.

What do you mean by "continuous contracts"? The broker offers the contracts in the exact same way with each contract having its own expiry date and unique symbol (depending on the month & year).

So, how in your opinion do I backtest these contracts, and actually live-trade on them with the broker, if I cannot "rotate" them properly?

[chewbacca]

Any other suggestions? No way to specify expiry-date-per-contract for rollover?

[run-out]

@chewbacca Have you ever looked at a one year es minute chart? They use continuous contracts for this. Just look it up. You can download the continuous contract for backtesting. In live trading you use the near term contract and just swap to the next one at maturity (or before).

[chewbacca]

@run-out Has nobody done auto-swap for live trading futures ?

[ab_trader]

@chewbacca said in Rollover multiple contracts [help needed]:

How do I use rollover with checkdate on these, each contract has its own expiration date, at which point VX1 and VX2 should change? Has someone done anything like this? How do I specify the expiry date of each contract for checkdate ?

Docs - Data Feeds - Rollover has a good example of how to use checkdate. Based on what you shown above it can be 15-16th day of the month.

Another way, as @run-out mentioned, is to build separate continuous future contracts, and use it for backtests. With IB broker you can use these contracts to backfill live contract - backfill_from parameter.

I personally prefer second way since bt rollover approach doesn't adjust prices.

[chewbacca]

@ab_trader using only a single date won't be as accurate as using separate date for each of the contracts.
There's no way of doing that?

[ab_trader]

@chewbacca said in Rollover multiple contracts [help needed]:

using only a single date won't be as accurate as using separate date for each of the contracts.

i am not sure what accuracy do you mean. unless you have some special strategy, moving from contract to contract is just maintaining open position, but not exiting/entering position by rules. therefore it doesn't matter at what price you switch, main thing is to decrease slippage.

typically (from my experience with american futures) traders move from contract to contract not on the exact date (for sure not on the expiration date), but at the moment when it is enough volume in the new contract and still good volume on the current contract. So you sell and buy with the small slippage. i don't think that selling on last day is good.

@chewbacca said in Rollover multiple contracts [help needed]:

There's no way of doing that?

you may try another option - checkcondition (see my link above). two neighboring contracts are available for this callable, so you can try to get dates and do the roll over.

[chewbacca]

@ab_trader said in Rollover multiple contracts [help needed]:

@chewbacca said in Rollover multiple contracts [help needed]:

using only a single date won't be as accurate as using separate date for each of the contracts.

i am not sure what accuracy do you mean. unless you have some special strategy, moving from contract to contract is just maintaining open position, but not exiting/entering position by rules. therefore it doesn't matter at what price you switch, main thing is to decrease slippage.

typically (from my experience with american futures) traders move from contract to contract not on the exact date (for sure not on the expiration date), but at the moment when it is enough volume in the new contract and still good volume on the current contract. So you sell and buy with the small slippage. i don't think that selling on last day is good.

@chewbacca said in Rollover multiple contracts [help needed]:

There's no way of doing that?

you may try another option - checkcondition (see my link above). two neighboring contracts are available for this callable, so you can try to get dates and do the roll over.

So.. I've had some progress rolling over, however, the fillers don't seem to work properly, there seem to be miss-alignments, and fill values are taken from I don't know where? Not properly at all:

VX1 January (VXF10)
2009-07-22 14:54:00,28.15,28.15,28.15,28.15,13,10
2009-07-23 10:21:00,28.10,28.10,28.10,28.10,1,1
2009-07-23 10:44:00,28.00,28.00,28.00,28.00,11,10
2009-07-23 13:12:00,27.90,27.90,27.90,27.90,12,1
2009-07-23 13:24:00,27.90,27.90,27.90,27.90,13,1

VX2 February (VXG10)
2009-07-23 10:07:00,28.20,28.20,28.20,28.20,1,1
2009-07-23 10:22:00,28.15,28.15,28.15,28.15,2,1
2009-07-23 10:34:00,28.15,28.15,28.15,28.15,3,1
2009-07-23 11:06:00,28.10,28.10,28.10,28.10,4,1
2009-07-23 11:09:00,28.05,28.05,28.05,28.05,5,1
2009-07-23 12:13:00,27.95,27.95,27.95,27.95,6,1

Filler:
VX1, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:32:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:32:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:33:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:34:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:33:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:34:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:35:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:36:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:35:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:36:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:37:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:38:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:37:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:38:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:39:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:40:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:39:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:40:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:41:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:42:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:41:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:42:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:43:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:44:00,28.20,28.20,28.20,28.20,0.00

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import ast
import datetime as dtm
import time

import itertools
import os
import sys
import argparse
from enum import Enum, IntEnum
from sys import argv
import backtrader as bt
import pandas_market_calendars as mcal
import pytz
import csv


def rollover_test():
    cerebro = bt.Cerebro()
    cerebro.broker = bt.brokers.BackBroker()
    cerebro.broker.setcash(100000.0)
    
    def get_data_feed(data_filename):
        data = bt.feeds.GenericCSVData(
            dataname=data_filename,
            headers=False,
            reverse=True,
            dtformat='%Y-%m-%d %H:%M:%S',
            datetime=0,
            open=1,
            high=2,
            low=3,
            close=4,
            volume=6,        # volume traded at this bar
            openinterest=5,  # total volume since market open of the day
            compression=1,
            timeframe=bt.TimeFrame.Minutes,
            sessionstart=dtm.time(8, 30),
            sessionend=dtm.time(15, 15),
        )
        return data

    def checkdate(dt, d):
        dates = "20060117,20060213,20060320,20060413,20060515,20060619,20060717,20060814,20060918,20061016,20061113,20061218," \
                "20070115,20070212,20070319,20070416,20070514,20070618,20070716,20070820,20070919,20071015,20071119,20071217," \
                "20080114,20080214,20080317,20080414,20080519,20080616,20080714,20080818,20080915,20081020,20081117,20081215," \
                "20090119,20090216,20090316,20090413,20090518,20090615,20090720,20090818,20090914,20091019,20091116,20091214," \
                "20100118,20100215,20100315,20100419,20100517,20100614,20100719,20100816,20100913,20101018,20101115,20101220," \
                "20110117,20110214,20110314,20110418,20110516,20110613,20110718,20110815,20110919,20111017,20111114,20111220," \
                "20120116,20120213,20120319,20120416,20120514,20120618,20120716,20120820,20120917,20121015,20121119,20121217," \
                "20130114,20130211,20130318,20130415,20130520,20130617,20130715,20130819,20130916,20131014,20131118,20131216," \
                "20140122,20140217,20140316,20140414,20140519,20140616,20140714,20140818,20140915,20141020,20141118,20141216," \
                "20150120,20150217,20150317,20150414,20150519,20150616,20150721,20150818,20150915,20151020,20151117,20151215," \
                "20160119,20160216,20160315,20160419,20160517,20160614,20160719,20160816,20160920,20161018,20161115,20161220," \
                "20170117,20170214,20170321,20170418,20170516,20170620,20170718,20170815,20170919,20171017,20171114,20171219," \
                "20180116,20180213,20180320,20180417,20180515,20180619,20180717,20180821,20180918,20181016,20181120,20181218," \
                "20190115,20190212,20190318,20190416,20190521,20190618,20190716,20190820,20190917,20191015,20191119,20191217," \
                "20200121,20200218,20200317,20200414,20200519,20200616,20200721,20200818,20200915,20201020,20201117,20201215"
        dates_list = dates.split(",")

        check_date = dt.date()
        name = d._name
        for date in dates_list:
            check_against = dtm.datetime.strptime(date, '%Y%m%d').date()
            if check_date == check_against:
                return True

        return False

    #              JAN    FEB    MAR    APR    MAY    JUN    JUL    AUG    SEP    OCT    NOV    DEC
    #contracts = ['VXF', 'VXG', 'VXH', 'VXJ', 'VXK', 'VXM', 'VXN', 'VXQ', 'VXU', 'VXV', 'VXX', 'VXZ']
    contracts = ['VXF', 'VXG', 'VXH', 'VXJ'] # DEBUG
    rollfeeds = []
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))

    for year in range(10, 11):  # DEBUG 2010 - 4 contracts
        for contract in contracts:
            item = 'FuturesData/contracts/{}{}.txt'.format(contract, year) # Path
            feed = get_data_feed(os.path.join(modpath, item)) # Datafeed
            feed.addfilter(bt.filters.SessionFiller, fill_price=None, skip_first_fill=False, fill_vol=0)
            feed.addfilter(bt.filters.SessionFilter(feed))
            rollfeeds.append(feed)

    # VX1, Initially VXF
    rollkwargs_vx1 = dict()
    rollkwargs_vx1['checkdate'] = checkdate
    rollkwargs_vx1['skip_first'] = False # True For VX2
    cerebro.rolloverdata(name='VX1', *rollfeeds, **rollkwargs_vx1)

    # VX2, Initially VXG
    rollkwargs_vx2 = dict()
    rollkwargs_vx2['checkdate'] = checkdate
    rollkwargs_vx2['skip_first'] = True # When rolling over for VX2, it needs to start 1 contract after VX1
    cerebro.rolloverdata(name='VX2', *rollfeeds, **rollkwargs_vx2)

    ## Add a strategy
    cerebro.addstrategy(BacktestLOG)

    # Start cerebro
    results = cerebro.run(stdstats=False, maxcpus=40)


class BacktestLOG(bt.Strategy):
    params = (
        ('data_feeds', 2)
    )

    def logdata(self):
        txt = ['{}'.format(self.data0._name), ' {}'.format(self.data0.datetime.datetime(0)), '{:.2f}'.format(self.data0.open[0]),
           '{:.2f}'.format(self.data0.high[0]), '{:.2f}'.format(self.data0.low[0]),
           '{:.2f}'.format(self.data0.close[0]), '{:.2f}'.format(self.data0.volume[0])]
        print(','.join(txt))
        if self.p.data_feeds > 1:
            txt1 = ['{}'.format(self.data1._name), ' {}'.format(self.data1.datetime.datetime(0)), '{:.2f}'.format(self.data1.open[0]),
               '{:.2f}'.format(self.data1.high[0]), '{:.2f}'.format(self.data1.low[0]),
               '{:.2f}'.format(self.data1.close[0]), '{:.2f}'.format(self.data1.volume[0])]
            print(','.join(txt1))

    def __init__(self):
        print("STRATEGY STARTED")

    def next(self):
        self.logdata()

    def stop(self):
        print("Finished!")



if __name__ == '__main__':
    rollover_test()

[chewbacca]

Also, how does rollover work on live data (IB) ?
I've set a few contracts, waiting for Monday to see what's gonna come up.

[chewbacca]

Has anyone ever used rolloverdata() on IB live feeds ?

For some reason the "data" objects returned by rolloverdata() don't have the necessary inner bits.
Example - a call to self.position.size or self.broker.getposition(data=self.data0) results in:

File "/home/user/PycharmProjects/backtrader/Strategies/VixLive.py", line 615, in notify_timer
self.initial_position = self.position.size
File "/home/user/PycharmProjects/backtrader/venv/lib/python3.8/site-packages/backtrader/lineseries.py", line 461, in _ _ getattr _ _
return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_LineIterator_DataAccessor_Strateg' object has no attribute 'position'

Also, notify_data() in the strategy is not being called at all.

Here's how I'm generating the feeds:

def get_data_feed(connection, name, local_symbol, compression):
    liveData = connection.getdata(
        dataname=local_symbol,
        name=name,
        local_symbol=local_symbol,	# I've reworked contract generation in ib to support local symbol (easier for futures)
        sectype='FUT',
        exchange='CFE',
        currency='USD',
        #mult='1000',
        tz=pytz.timezone('US/Central'),
        #useRTH=True,
        #rtBAR=True,
        sessionstart=dtm.time(8, 31),
        sessionend=dtm.time(15, 15),
        timeframe=bt.TimeFrame.Minutes,
        qcheck=0.5,
        historical=False,
        latethrough=True,
        compression=compression,
        backfill=False,
        backfill_start=False,
        logger=log,
    )
    return liveData

def checkdate(dt, d):
    dates = "20200121,20200218,20200317,20200414,20200519,20200616,20200721,20200818,20200915,20201020,20201117,20201215," \
            "20210119,20210216,20210316,20210420,20210518,20210615,20210720,20210817,20210914,20211019,20211116,20211221"
    dates_list = dates.split(",")

    check_date = dt.date()
    name = d._name
    localName = d._localname
    for date in dates_list:
        check_against = dtm.datetime.strptime(date, '%Y%m%d').date()
        if check_date == check_against:
            return True
    return False

contracts_vx1 = ['VXX0', 'VXZ0'] # Remaining for 2020
contracts_vx2 = ['VXZ0', 'VXF1'] # Remaining for 2020 + 1 month of 2021
rollfeeds_vx2_minute = []   # data0
rollfeeds_vx2 = []          # data1
rollfeeds_vx1 = []          # data2

for contract in contracts_vx2:
    vx2_feed_1min = get_data_feed(ibstore_vx2, "VX2-1min", contract, 1)
    vx2_feed_1min.addfilter(bt.filters.SessionFilter(vx2_feed_1min))
    rollfeeds_vx2_minute.append(vx2_feed_1min)
    #
    vx2_feed_30min = get_data_feed(ibstore_vx2, "VX2-30min", contract, 30)
    vx2_feed_30min.addfilter(bt.filters.SessionFilter(vx2_feed_30min))
    rollfeeds_vx2.append(vx2_feed_30min)

for contract in contracts_vx1:
    vx1_feed_30min = get_data_feed(ibstore_vx1, "VX1-30min", contract, 30)
    vx1_feed_30min.addfilter(bt.filters.SessionFilter(vx1_feed_30min))
    rollfeeds_vx1.append(vx1_feed_30min)

rollkwargs_vx2_minute = dict()
rollkwargs_vx2_minute['checkdate'] = checkdate
vx2_1_min = cerebro.rolloverdata(*rollfeeds_vx2_minute, **rollkwargs_vx2_minute)   # data0 - Trading on this one

rollkwargs_vx2 = dict()
rollkwargs_vx2['checkdate'] = checkdate
rollkwargs_vx2['skip_first'] = False # True For VX2
vx2_30_min = cerebro.rolloverdata(name='VX2-30min-ROLL', *rollfeeds_vx2, **rollkwargs_vx2)   # data1 - using for calculations

rollkwargs_vx1 = dict()
rollkwargs_vx1['checkdate'] = checkdate
rollkwargs_vx1['skip_first'] = False # True For VX2
vx1_30min = cerebro.rolloverdata(name='VX1-30min-ROLL', *rollfeeds_vx1, **rollkwargs_vx1)    # data2 - using for calculations

In the strategy I'm checking the feeds during _ _ init _ _():

    feeds_enum = list(enumerate(self.getdatanames()))
    for index in feeds_enum:
        print('#DATA INDEX: {}'.format(index))

Result:
#DATA INDEX: (0, 'VXZ0')
#DATA INDEX: (1, 'VX2-30min-ROLL')
#DATA INDEX: (2, 'VX1-30min-ROLL')

[vladisld]

@chewbacca said in Rollover multiple contracts [help needed]:

File "/home/user/PycharmProjects/backtrader/Strategies/VixLive.py", line 615, in notify_timer
self.initial_position = self.position.size
File "/home/user/PycharmProjects/backtrader/venv/lib/python3.8/site-packages/backtrader/lineseries.py", line 461, in _ _ getattr _ _
return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_LineIterator_DataAccessor_Strateg' object has no attribute 'position'

The error backtrace above seems strange. Accessing the position property in the strategy should call the bt.Strategy's getposition method:

class Strategy:
...
    def getposition(self, data=None, broker=None):
        data = data if data is not None else self.datas[0]
        broker = broker or self.broker
        return broker.getposition(data)

    position = property(getposition)

This should be reflected in the error backtrace. It looks like the self.position was called from the indicator and not from the strategy.

It could be useful to if you could share the whole VixLive.py code?

[chewbacca]

@vladisld said in Rollover multiple contracts [help needed]:

@chewbacca said in Rollover multiple contracts [help needed]:

File "/home/user/PycharmProjects/backtrader/Strategies/VixLive.py", line 615, in notify_timer
self.initial_position = self.position.size
File "/home/user/PycharmProjects/backtrader/venv/lib/python3.8/site-packages/backtrader/lineseries.py", line 461, in _ _ getattr _ _
return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_LineIterator_DataAccessor_Strateg' object has no attribute 'position'

The error backtrace above seems strange. Accessing the position property in the strategy should call the bt.Strategy's getposition method:

class Strategy:
...
    def getposition(self, data=None, broker=None):
        data = data if data is not None else self.datas[0]
        broker = broker or self.broker
        return broker.getposition(data)

    position = property(getposition)

This should be reflected in the error backtrace. It looks like the self.position was called from the indicator and not from the strategy.

It could be useful to if you could share the whole VixLive.py code?

By default data0 is used as the "primary", correct?
vx2_1_min = cerebro.rolloverdata(*rollfeeds_vx2_minute, **rollkwargs_vx2_minute) -- is data0 since it's called & added first.

Correct me if I'm wrong..