Furtures data with missing days - how to handle

[j45p41]

Hi,

I am trying to read in futures data in the following fomat:

DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
19900531,104687.5000,104850.0000,104387.5000,104525.0000,18592,32108,0,0,0.0000
19900601,104750.0000,105512.5000,104612.5000,104762.5000,26697,27205,0,0,0.0000
19900604,104250.0000,104562.5000,103937.5000,104437.5000,15315,25228,0,0,0.0000
19900605,104575.0000,105250.0000,104425.0000,105137.5000,18054,25452,0,0,0.0000
19900606,105187.5000,105437.5000,105112.5000,105300.0000,10104,24872,0,0,0.0000
19900607,105250.0000,105450.0000,105137.5000,105425.0000,9931,24237,199009,199009,0.0000
19900608,103562.5000,103562.5000,103125.0000,103437.5000,12791,9134,0,0,-1600.0000

I intend to trim the data so that it does not include the last three columns.

However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

I suspect there is a setting somewhere which will allow this data to be handled.

I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

but can't figure out how to handle this data - any suggestions pls?

[Laurent Michelizza]

This sounds suspicious. I use futures data with missing days but don't have this issue.
Do you load only one future when running your tests?

Also, you should include your code.

[backtrader]

@j45p41 said in Furtures data with missing days - how to handle:

I intend to trim the data so that it does not include the last three columns.

There is no need to trim anything. Extra columns will be ignored (if using GenericCSVData for example)

@j45p41 said in Furtures data with missing days - how to handle:

However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

Bogus statement. Please provide proof that trades are not processed correctly.

@j45p41 said in Furtures data with missing days - how to handle:

I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

but can't figure out how to handle this data - any suggestions pls?

The data shown above is completely normal. You show no code and the statements are not backed by any kind of evidence of anything. How can any suggestion be made?

[j45p41]

My apologies - i thought there was a problem due to discontinues dates.

here is my code in which I have disabled my analysers as they end up giving me a keyerror. When disabled I get a graph that does not execute trades.

import datetime
import time
import os.path
import sys
import backtrader as bt
import backtrader.feeds as btfeed
import ccxt
import csv
import io
import pandas as pd
from collections import OrderedDict
from multiprocessing import Pool, cpu_count
import math
import os



# DECLARE MODE FOR PROGRAM - OPTOMISATION OR STRATEGY
opt_mode = False
DEBUG = False

# LOG OUTPUT TO FILE
class Logger(object):
    def __init__(self, filename="Default.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)

    def flush(self):
        pass

# CSV INPUT FILE FORMAT CONFIGURATION
class dataFeed(btfeed.GenericCSVData):
    params = (
        ('dtformat', '%Y%m%d'),
        ('datetime', 0),
        ('open', 1),
        ('high', 2),
        ('low', 3),
        ('close', 4),
        ('volume', 5),
        ('openinterest', 6)
    )
class OrderObserver(bt.observer.Observer):
    lines = ('created', 'expired',)

    plotinfo = dict(plot=True, subplot=True, plotlinelabels=True)

    plotlines = dict(
        created=dict(marker='*', markersize=8.0, color='lime', fillstyle='full'),
        expired=dict(marker='s', markersize=8.0, color='red', fillstyle='full')
    )

    def next(self):
        for order in self._owner._orderspending:
            if order.data is not self.data:
                continue

            if not order.isbuy():
                continue

            # Only interested in "buy" orders, because the sell orders
            # in the strategy are Market orders and will be immediately
            # executed

            if order.status in [bt.Order.Accepted, bt.Order.Submitted]:
                self.lines.created[0] = order.created.price

            elif order.status in [bt.Order.Expired]:
                self.lines.expired[0] = order.created.price

# MAIN STRATEGY DEFINITION - DEFINE VALUES HERE FOR NON-OPTOMISATION MODE
class firstStrategy(bt.Strategy):
    params = (
        ("period", 28),
        ("rsi_low", 30),
        ("rsi_high", 70),
    )

    #TRADE LOGGING FUNCTION
    def log(self, txt, dt=None):
        ''' Logging function fot this strategy'''
        if not opt_mode:
            dt = dt or self.datas[0].datetime.datetime(0)
            print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self):
        self.startcash = self.broker.getvalue()
        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)


    #TRADE LOGGING FUNCTION
    def notify_trade(self, trade):
        if not trade.isclosed and not opt_mode:
            return

        self.log('TRADE INFO, PRICE  %.2f, GROSS %.2f, NET %.2f' %
                 (trade.price, trade.pnl, trade.pnlcomm))

    def next(self):
        if not self.position:
            if self.rsi < self.params.rsi_low:
                self.buy(size=10)
        else:
            if self.rsi > self.params.rsi_high:
                self.sell(size=10)
        if opt_mode and DEBUG:
            print('period: {}, rsi low: {}, rsi high {}'.format(self.params.period,self.params.rsi_low,self.params.rsi_high))

# if opt_mode == False:
#     def printTradeAnalysis(analyzer):
#         '''
#         Function to print the Technical Analysis results in a nice format.
#         '''
#         #Get the results we are interested in
#         total_open = analyzer.total.open
#         total_closed = analyzer.total.closed
#         total_won = analyzer.won.total
#         total_lost = analyzer.lost.total
#         win_streak = analyzer.streak.won.longest
#         lose_streak = analyzer.streak.lost.longest
#         pnl_net = round(analyzer.pnl.net.total,2)
#         strike_rate = round((total_won / total_closed) * 100,2)
#         #Designate the rows
#         h1 = ['Total Open', 'Total Closed', 'Total Won', 'Total Lost']
#         h2 = ['Strike Rate','Win Streak', 'Losing Streak', 'PnL Net']
#         r1 = [total_open, total_closed,total_won,total_lost]
#         r2 = [strike_rate, win_streak, lose_streak, pnl_net]
#         #Check which set of headers is the longest.
#         if len(h1) > len(h2):
#             header_length = len(h1)
#         else:
#             header_length = len(h2)
#         #Print the rows
#         print_list = [h1,r1,h2,r2]
#         row_format ="{:<15}" * (header_length + 1)
#         print("Trade Analysis Results:")
#         for row in print_list:
#             print(row_format.format('',*row))
#
#     def printSQN(analyzer):
#         sqn = round(analyzer.sqn,2)
#         print('SQN: {}'.format(sqn))

# INPUT CONDITIONS TO FEED INTO CEREBRO IS ADDED HERE
if __name__ == '__main__':
    sys.stdout = Logger("strat_evaluator.log")


    #periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12])
    periods = pd.DataFrame(columns=['FROM','TO'],index=[1])
    periods.loc[1] = ('1990-1-01','2017-12-31')
    #periods.loc[2] = ('2017-02-01','2017-03-01')
    #periods.loc[3] = ('2017-03-01','2017-04-01')
    #periods.loc[4] = ('2017-04-01','2017-05-01')
    #periods.loc[5] = ('2017-05-01','2017-06-01')
    #periods.loc[6] = ('2017-06-01','2017-07-01')
    #periods.loc[7] = ('2017-07-01','2017-08-01')
    #periods.loc[8] = ('2017-08-01','2017-09-01')
    #periods.loc[9] = ('2017-09-01','2017-10-01')
    #periods.loc[10] = ('2017-10-01','2017-11-01')
    #periods.loc[11] = ('2017-11-01','2017-12-01')
    #periods.loc[12] = ('2017-12-01','2017-12-31')

    for index, row in periods.iterrows():


        # Variable for our starting cash
        startcash = 10000
        # Create an instance of cerebro
        cerebro = bt.Cerebro(optreturn=False)

        # Timing the whole operation
        time_at_start = time.time()

        if opt_mode:
            # ADD STRATEGY OPTIMISATION
            #cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))
            cerebro.optstrategy(firstStrategy, period=range(13, 14), rsi_low=range(20, 21), rsi_high=range(40, 41))
            cerebro.addanalyzer(bt.analyzers.SharpeRatio)
        else:
            #ADD STRATEGY
            cerebro.addstrategy(firstStrategy)

        #format dates for datafeed object
        fy,fm,fd = periods['FROM'][index].split('-')
        ty,tm,td = periods['TO'][index].split('-')


#READ DATA FROM CSV FILE

        filename = 'tickerData/F_BO.txt'
        modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
        datapath = os.path.join(modpath,str(filename))
        data = dataFeed(dataname=datapath, timeframe=bt.TimeFrame.Days,
                        fromdate=datetime.datetime(int(fy),int(fm),int(fd)),
                        todate=datetime.datetime(int(ty),int(tm),int(td)),)

        # Add the data to Cerebro
        cerebro.adddata(data)

        # Set our desired cash start
        cerebro.broker.setcash(startcash)

        if not opt_mode:
            # Add the analyzers we are interested in
            cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name="ta")
            cerebro.addanalyzer(bt.analyzers.SQN, _name="sqn")


        #WRITER TEST
        #cerebro.addwriter(bt.WriterFile, csv=True, rounding=2)

        # RUN STRATEGY THROUGH CEREBRO USING INPUT DATA
        # Timing the operation
        time_at_end = time.time()
        time_elapsed = round(time_at_end - time_at_start,2)
        print('Time elapsed: {} seconds'.format(time_elapsed))
        print ('Running Cerebro')
        if not opt_mode: opt_runs = cerebro.run(tradehistory=False)
        else: opt_runs = cerebro.run(tradehistory=False, runonce = False, Exactbars=True)

        firstStrat = opt_runs[0]

        if opt_mode:
            # CREATE A LIST VARIABLE THAT CONTAINS RESULTS
            final_results_list = []
            for run in opt_runs:
                for strategy in run:
                    value = round(strategy.broker.get_value(), 2)
                    PnL = round(value - startcash, 2)
                    period = strategy.params.period
                    rsi_low = strategy.params.rsi_low
                    rsi_high = strategy.params.rsi_high
                    final_results_list.append([period, rsi_low, rsi_high, PnL])

            # Sort Results List
            by_period = sorted(final_results_list, key=lambda x: x[0])
            by_PnL = sorted(final_results_list, key=lambda x: x[3], reverse=True)

            # PRINT RESULTS IN OPTIMISATION AND FILTER TOP 3
            result_number = 0
            print('Results: Ordered by Profit:')
            for result in by_PnL:
                if result_number < 3:
                    print('Asset: {} Start: {}, End: {}, Period: {}, rsi_low: {}, rsi_high: {}, PnL: {}'.format(filename, periods['FROM'][index], periods['TO'][index], result[0], result[1], result[2], result[3]))
                    result_number = result_number + 1

        # Timing the operation
        time_at_end = time.time()
        time_elapsed = round(time_at_end - time_at_start,2)


        print('Time elapsed: {} seconds'.format(time_elapsed))
        if opt_mode == False:
            # print the analyzers
            # printTradeAnalysis(firstStrat.analyzers.ta.get_analysis())
            # printSQN(firstStrat.analyzers.sqn.get_analysis())

            #Get final portfolio Value
            portvalue = cerebro.broker.getvalue()

            #Print out the final result
            print('Final Portfolio Value: ${}'.format(portvalue))
            cerebro.plot(style='candlestick')

        #cerebro.plot(style='candlestick')

for completeness here is the data that I am using

DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
19900102,11520.0000,11520.0000,11316.0000,11322.0000,19779,38563,0,0,0.0000
19900103,11322.0000,11460.0000,11298.0000,11442.0000,16270,39377,0,0,0.0000
19900104,11472.0000,11520.0000,11340.0000,11346.0000,12743,39220,0,0,0.0000
19900105,11298.0000,11466.0000,11286.0000,11418.0000,11808,39932,0,0,0.0000
19900108,11448.0000,11808.0000,11442.0000,11784.0000,20844,40091,0,0,0.0000
19900109,11730.0000,11820.0000,11682.0000,11778.0000,19457,39502,0,0,0.0000
19900110,11760.0000,11778.0000,11598.0000,11724.0000,16266,39703,0,0,0.0000
19900111,11712.0000,11790.0000,11706.0000,11754.0000,13699,39704,0,0,0.0000
19900112,11826.0000,11844.0000,11634.0000,11640.0000,12677,38444,0,0,0.0000
19900115,11598.0000,11688.0000,11580.0000,11652.0000,6067,38307,0,0,0.0000
19900116,11610.0000,11640.0000,11490.0000,11496.0000,13691,37511,0,0,0.0000
19900117,11484.0000,11616.0000,11484.0000,11502.0000,12378,38223,0,0,0.0000
19900118,11508.0000,11580.0000,11496.0000,11538.0000,8570,37584,0,0,0.0000
19900119,11562.0000,11568.0000,11400.0000,11406.0000,14732,36865,0,0,0.0000
19900122,11400.0000,11448.0000,11316.0000,11358.0000,15163,36837,0,0,0.0000
19900123,11382.0000,11508.0000,11376.0000,11466.0000,13141,36918,0,0,0.0000
19900124,11472.0000,11514.0000,11436.0000,11448.0000,10550,36551,0,0,0.0000
19900125,11472.0000,11478.0000,11340.0000,11340.0000,14579,35655,0,0,0.0000
19900126,11340.0000,11430.0000,11328.0000,11418.0000,7547,34389,0,0,0.0000
19900129,11430.0000,11544.0000,11370.0000,11538.0000,11285,34198,0,0,0.0000
19900130,11544.0000,11574.0000,11442.0000,11460.0000,12167,33318,0,0,0.0000
19900131,11460.0000,11544.0000,11430.0000,11502.0000,10708,32328,0,0,0.0000
19900201,11520.0000,11700.0000,11502.0000,11532.0000,17957,32443,0,0,0.0000
19900202,11544.0000,11580.0000,11508.0000,11568.0000,14151,31419,0,0,0.0000
19900205,11568.0000,11634.0000,11460.0000,11478.0000,14241,30695,0,0,0.0000

[j45p41]

@j45p41 Hi guys, with the additional info provided can you help me understand why my code is not processing trades? - thanks.