Nevermind, I wasn't placing the expiry date correctly:
dataname='VIX-FUT-CFE-USD-20201118-1000'

All good :)

Anyone ?!

I also thought I'd need someone to help teach me, however, after reading the source code of bt a few times (and learning python in the mean time), I started to do what I need myself.

Well, I tried, I'll provide everything here and its up to the community to use/develop it or not.

@baykids4 Also, the samples are very helpful, once you read them .. you'll start getting it.

@vladisld said in Backtrader's Future:

I've created a discord server. Here the invite: https://discord.gg/EwpKMUbd

Hope it will be useful for the community.

"The invite is invalid or is expired"

@vladisld / @ab_trader can someone notify the forum admin -- check the bottom of the page:
$(document).ready(function () { app.coldLoad(); }); }
A script that is outside of <script> </script> tags... breaks the forum.

Also, since there are no private messages on the forum... @run-out check your PMs @ Discord.

@vladisld / @ab_trader can someone notify the forum admin -- check the bottom of the page:
$(document).ready(function () { app.coldLoad(); }); }
A script that is outside of <script> </script> tags... breaks the forum.

For some reason - it started working only after enabling rtbar=True.
The VIX Futures however seem to work without it.

Has anyone have any idea on this?

Has anyone here been using any sort of INDEX in BT?

I have live data subscription for the VIX Index, I see realtime data in TWS, however, I've tried a 100 different ways to load it in BT - all failed.

I've tried:
dataname='VIX-IND-CBOE-USD'
dataname='VIX-IND-CBOE-USD-1000'

I've also tried by requesting it a different way (the same way as described in the IBApi):
symbol="VIX",
sectype="IND",
exchange="CBOE",
currency='USD',

All I get is:
makecontract() called by init, symbol: VIX, sectype: IND, exchange: CBOE, currency: USD, expiry: None, strike: None, right: None, mult: , localSymbol: None (mult is ''), also tried with 1 and 1000, no difference

getContractDetails() returns:
ContractID: 13455763 | Category: Volatility Index | Contract Month: None | Industry: Indices | Long name: CBOE Volatility Index
Market Name: None | Currency: USD | Exchange: CBOE | Multiplier: None | Expiry: None
Local symbol: VIX | Sec type: IND | Symbol: VIX | Trading class: None
Liquid Hours: 20201215:0215-20201215:0815;20201215:0830-20201215:1515;20201216:0215-20201216:0815;20201216:0830-20201216:1515;20201217:0215-20201217:0815;20201217:0830-20201217:1515;20201218:0215-20201218:0815;20201218:0830-20201218:1515
Trading Hours: 20201215:0215-20201215:0815;20201215:0830-20201215:1515;20201216:0215-20201216:0815;20201216:0830-20201216:1515;20201217:0215-20201217:0815;20201217:0830-20201217:1515;20201218:0215-20201218:0815;20201218:0830-20201218:1515
Valid Exchanges: CBOE

So - it returns a "valid" contract, but - no live data? I've not set session filters.

@vladisld said in Rollover multiple contracts [help needed]:

@chewbacca said in Rollover multiple contracts [help needed]:

File "/home/user/PycharmProjects/backtrader/Strategies/VixLive.py", line 615, in notify_timer
self.initial_position = self.position.size
File "/home/user/PycharmProjects/backtrader/venv/lib/python3.8/site-packages/backtrader/lineseries.py", line 461, in _ _ getattr _ _
return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_LineIterator_DataAccessor_Strateg' object has no attribute 'position'

The error backtrace above seems strange. Accessing the position property in the strategy should call the bt.Strategy's getposition method:

class Strategy:
...
    def getposition(self, data=None, broker=None):
        data = data if data is not None else self.datas[0]
        broker = broker or self.broker
        return broker.getposition(data)

    position = property(getposition)

This should be reflected in the error backtrace. It looks like the self.position was called from the indicator and not from the strategy.

It could be useful to if you could share the whole VixLive.py code?

By default data0 is used as the "primary", correct?
vx2_1_min = cerebro.rolloverdata(*rollfeeds_vx2_minute, **rollkwargs_vx2_minute) -- is data0 since it's called & added first.

Correct me if I'm wrong..

Has anyone ever used rolloverdata() on IB live feeds ?

For some reason the "data" objects returned by rolloverdata() don't have the necessary inner bits.
Example - a call to self.position.size or self.broker.getposition(data=self.data0) results in:

File "/home/user/PycharmProjects/backtrader/Strategies/VixLive.py", line 615, in notify_timer
self.initial_position = self.position.size
File "/home/user/PycharmProjects/backtrader/venv/lib/python3.8/site-packages/backtrader/lineseries.py", line 461, in _ _ getattr _ _
return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_LineIterator_DataAccessor_Strateg' object has no attribute 'position'

Also, notify_data() in the strategy is not being called at all.

Here's how I'm generating the feeds:

def get_data_feed(connection, name, local_symbol, compression):
    liveData = connection.getdata(
        dataname=local_symbol,
        name=name,
        local_symbol=local_symbol,	# I've reworked contract generation in ib to support local symbol (easier for futures)
        sectype='FUT',
        exchange='CFE',
        currency='USD',
        #mult='1000',
        tz=pytz.timezone('US/Central'),
        #useRTH=True,
        #rtBAR=True,
        sessionstart=dtm.time(8, 31),
        sessionend=dtm.time(15, 15),
        timeframe=bt.TimeFrame.Minutes,
        qcheck=0.5,
        historical=False,
        latethrough=True,
        compression=compression,
        backfill=False,
        backfill_start=False,
        logger=log,
    )
    return liveData

def checkdate(dt, d):
    dates = "20200121,20200218,20200317,20200414,20200519,20200616,20200721,20200818,20200915,20201020,20201117,20201215," \
            "20210119,20210216,20210316,20210420,20210518,20210615,20210720,20210817,20210914,20211019,20211116,20211221"
    dates_list = dates.split(",")

    check_date = dt.date()
    name = d._name
    localName = d._localname
    for date in dates_list:
        check_against = dtm.datetime.strptime(date, '%Y%m%d').date()
        if check_date == check_against:
            return True
    return False

contracts_vx1 = ['VXX0', 'VXZ0'] # Remaining for 2020
contracts_vx2 = ['VXZ0', 'VXF1'] # Remaining for 2020 + 1 month of 2021
rollfeeds_vx2_minute = []   # data0
rollfeeds_vx2 = []          # data1
rollfeeds_vx1 = []          # data2

for contract in contracts_vx2:
    vx2_feed_1min = get_data_feed(ibstore_vx2, "VX2-1min", contract, 1)
    vx2_feed_1min.addfilter(bt.filters.SessionFilter(vx2_feed_1min))
    rollfeeds_vx2_minute.append(vx2_feed_1min)
    #
    vx2_feed_30min = get_data_feed(ibstore_vx2, "VX2-30min", contract, 30)
    vx2_feed_30min.addfilter(bt.filters.SessionFilter(vx2_feed_30min))
    rollfeeds_vx2.append(vx2_feed_30min)

for contract in contracts_vx1:
    vx1_feed_30min = get_data_feed(ibstore_vx1, "VX1-30min", contract, 30)
    vx1_feed_30min.addfilter(bt.filters.SessionFilter(vx1_feed_30min))
    rollfeeds_vx1.append(vx1_feed_30min)

rollkwargs_vx2_minute = dict()
rollkwargs_vx2_minute['checkdate'] = checkdate
vx2_1_min = cerebro.rolloverdata(*rollfeeds_vx2_minute, **rollkwargs_vx2_minute)   # data0 - Trading on this one

rollkwargs_vx2 = dict()
rollkwargs_vx2['checkdate'] = checkdate
rollkwargs_vx2['skip_first'] = False # True For VX2
vx2_30_min = cerebro.rolloverdata(name='VX2-30min-ROLL', *rollfeeds_vx2, **rollkwargs_vx2)   # data1 - using for calculations

rollkwargs_vx1 = dict()
rollkwargs_vx1['checkdate'] = checkdate
rollkwargs_vx1['skip_first'] = False # True For VX2
vx1_30min = cerebro.rolloverdata(name='VX1-30min-ROLL', *rollfeeds_vx1, **rollkwargs_vx1)    # data2 - using for calculations

In the strategy I'm checking the feeds during _ _ init _ _():

    feeds_enum = list(enumerate(self.getdatanames()))
    for index in feeds_enum:
        print('#DATA INDEX: {}'.format(index))

Result:
#DATA INDEX: (0, 'VXZ0')
#DATA INDEX: (1, 'VX2-30min-ROLL')
#DATA INDEX: (2, 'VX1-30min-ROLL')

@vladisld said in Backtrader's Future:

I've created a discord server. Here the invite: https://discord.gg/EwpKMUbd

Hope it will be useful for the community.

"The invite is invalid or is expired"

@Atul-Patel said in Backtrader's Future:

discord

If there is a discord group, it SHOULD be created by @vladisld since he has taken the responsibility of maintaining the backtrader2/backtrader github repository.

@vladisld please, create a discord group and place the invite link in a good place on the forum and github aswell.

Also, how does rollover work on live data (IB) ?
I've set a few contracts, waiting for Monday to see what's gonna come up.

@ab_trader said in Rollover multiple contracts [help needed]:

@chewbacca said in Rollover multiple contracts [help needed]:

using only a single date won't be as accurate as using separate date for each of the contracts.

i am not sure what accuracy do you mean. unless you have some special strategy, moving from contract to contract is just maintaining open position, but not exiting/entering position by rules. therefore it doesn't matter at what price you switch, main thing is to decrease slippage.

typically (from my experience with american futures) traders move from contract to contract not on the exact date (for sure not on the expiration date), but at the moment when it is enough volume in the new contract and still good volume on the current contract. So you sell and buy with the small slippage. i don't think that selling on last day is good.

@chewbacca said in Rollover multiple contracts [help needed]:

There's no way of doing that?

you may try another option - checkcondition (see my link above). two neighboring contracts are available for this callable, so you can try to get dates and do the roll over.

So.. I've had some progress rolling over, however, the fillers don't seem to work properly, there seem to be miss-alignments, and fill values are taken from I don't know where? Not properly at all:

VX1 January (VXF10)
2009-07-22 14:54:00,28.15,28.15,28.15,28.15,13,10
2009-07-23 10:21:00,28.10,28.10,28.10,28.10,1,1
2009-07-23 10:44:00,28.00,28.00,28.00,28.00,11,10
2009-07-23 13:12:00,27.90,27.90,27.90,27.90,12,1
2009-07-23 13:24:00,27.90,27.90,27.90,27.90,13,1

VX2 February (VXG10)
2009-07-23 10:07:00,28.20,28.20,28.20,28.20,1,1
2009-07-23 10:22:00,28.15,28.15,28.15,28.15,2,1
2009-07-23 10:34:00,28.15,28.15,28.15,28.15,3,1
2009-07-23 11:06:00,28.10,28.10,28.10,28.10,4,1
2009-07-23 11:09:00,28.05,28.05,28.05,28.05,5,1
2009-07-23 12:13:00,27.95,27.95,27.95,27.95,6,1

Filler:
VX1, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:32:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:31:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:32:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:33:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:34:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:33:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:34:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:35:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:36:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:35:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:36:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:37:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:38:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:37:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:38:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:39:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:40:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:39:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:40:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:41:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:42:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:41:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:42:00,28.20,28.20,28.20,28.20,0.00
VX2, 2009-07-23 08:43:00,28.20,28.20,28.20,28.20,0.00
VX1, 2009-07-23 08:44:00,28.20,28.20,28.20,28.20,0.00

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import ast
import datetime as dtm
import time

import itertools
import os
import sys
import argparse
from enum import Enum, IntEnum
from sys import argv
import backtrader as bt
import pandas_market_calendars as mcal
import pytz
import csv


def rollover_test():
    cerebro = bt.Cerebro()
    cerebro.broker = bt.brokers.BackBroker()
    cerebro.broker.setcash(100000.0)
    
    def get_data_feed(data_filename):
        data = bt.feeds.GenericCSVData(
            dataname=data_filename,
            headers=False,
            reverse=True,
            dtformat='%Y-%m-%d %H:%M:%S',
            datetime=0,
            open=1,
            high=2,
            low=3,
            close=4,
            volume=6,        # volume traded at this bar
            openinterest=5,  # total volume since market open of the day
            compression=1,
            timeframe=bt.TimeFrame.Minutes,
            sessionstart=dtm.time(8, 30),
            sessionend=dtm.time(15, 15),
        )
        return data

    def checkdate(dt, d):
        dates = "20060117,20060213,20060320,20060413,20060515,20060619,20060717,20060814,20060918,20061016,20061113,20061218," \
                "20070115,20070212,20070319,20070416,20070514,20070618,20070716,20070820,20070919,20071015,20071119,20071217," \
                "20080114,20080214,20080317,20080414,20080519,20080616,20080714,20080818,20080915,20081020,20081117,20081215," \
                "20090119,20090216,20090316,20090413,20090518,20090615,20090720,20090818,20090914,20091019,20091116,20091214," \
                "20100118,20100215,20100315,20100419,20100517,20100614,20100719,20100816,20100913,20101018,20101115,20101220," \
                "20110117,20110214,20110314,20110418,20110516,20110613,20110718,20110815,20110919,20111017,20111114,20111220," \
                "20120116,20120213,20120319,20120416,20120514,20120618,20120716,20120820,20120917,20121015,20121119,20121217," \
                "20130114,20130211,20130318,20130415,20130520,20130617,20130715,20130819,20130916,20131014,20131118,20131216," \
                "20140122,20140217,20140316,20140414,20140519,20140616,20140714,20140818,20140915,20141020,20141118,20141216," \
                "20150120,20150217,20150317,20150414,20150519,20150616,20150721,20150818,20150915,20151020,20151117,20151215," \
                "20160119,20160216,20160315,20160419,20160517,20160614,20160719,20160816,20160920,20161018,20161115,20161220," \
                "20170117,20170214,20170321,20170418,20170516,20170620,20170718,20170815,20170919,20171017,20171114,20171219," \
                "20180116,20180213,20180320,20180417,20180515,20180619,20180717,20180821,20180918,20181016,20181120,20181218," \
                "20190115,20190212,20190318,20190416,20190521,20190618,20190716,20190820,20190917,20191015,20191119,20191217," \
                "20200121,20200218,20200317,20200414,20200519,20200616,20200721,20200818,20200915,20201020,20201117,20201215"
        dates_list = dates.split(",")

        check_date = dt.date()
        name = d._name
        for date in dates_list:
            check_against = dtm.datetime.strptime(date, '%Y%m%d').date()
            if check_date == check_against:
                return True

        return False

    #              JAN    FEB    MAR    APR    MAY    JUN    JUL    AUG    SEP    OCT    NOV    DEC
    #contracts = ['VXF', 'VXG', 'VXH', 'VXJ', 'VXK', 'VXM', 'VXN', 'VXQ', 'VXU', 'VXV', 'VXX', 'VXZ']
    contracts = ['VXF', 'VXG', 'VXH', 'VXJ'] # DEBUG
    rollfeeds = []
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))

    for year in range(10, 11):  # DEBUG 2010 - 4 contracts
        for contract in contracts:
            item = 'FuturesData/contracts/{}{}.txt'.format(contract, year) # Path
            feed = get_data_feed(os.path.join(modpath, item)) # Datafeed
            feed.addfilter(bt.filters.SessionFiller, fill_price=None, skip_first_fill=False, fill_vol=0)
            feed.addfilter(bt.filters.SessionFilter(feed))
            rollfeeds.append(feed)

    # VX1, Initially VXF
    rollkwargs_vx1 = dict()
    rollkwargs_vx1['checkdate'] = checkdate
    rollkwargs_vx1['skip_first'] = False # True For VX2
    cerebro.rolloverdata(name='VX1', *rollfeeds, **rollkwargs_vx1)

    # VX2, Initially VXG
    rollkwargs_vx2 = dict()
    rollkwargs_vx2['checkdate'] = checkdate
    rollkwargs_vx2['skip_first'] = True # When rolling over for VX2, it needs to start 1 contract after VX1
    cerebro.rolloverdata(name='VX2', *rollfeeds, **rollkwargs_vx2)

    ## Add a strategy
    cerebro.addstrategy(BacktestLOG)

    # Start cerebro
    results = cerebro.run(stdstats=False, maxcpus=40)


class BacktestLOG(bt.Strategy):
    params = (
        ('data_feeds', 2)
    )

    def logdata(self):
        txt = ['{}'.format(self.data0._name), ' {}'.format(self.data0.datetime.datetime(0)), '{:.2f}'.format(self.data0.open[0]),
           '{:.2f}'.format(self.data0.high[0]), '{:.2f}'.format(self.data0.low[0]),
           '{:.2f}'.format(self.data0.close[0]), '{:.2f}'.format(self.data0.volume[0])]
        print(','.join(txt))
        if self.p.data_feeds > 1:
            txt1 = ['{}'.format(self.data1._name), ' {}'.format(self.data1.datetime.datetime(0)), '{:.2f}'.format(self.data1.open[0]),
               '{:.2f}'.format(self.data1.high[0]), '{:.2f}'.format(self.data1.low[0]),
               '{:.2f}'.format(self.data1.close[0]), '{:.2f}'.format(self.data1.volume[0])]
            print(','.join(txt1))

    def __init__(self):
        print("STRATEGY STARTED")

    def next(self):
        self.logdata()

    def stop(self):
        print("Finished!")



if __name__ == '__main__':
    rollover_test()