For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Furtures data with missing days - how to handle



  • Hi,

    I am trying to read in futures data in the following fomat:

    DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
    19900531,104687.5000,104850.0000,104387.5000,104525.0000,18592,32108,0,0,0.0000
    19900601,104750.0000,105512.5000,104612.5000,104762.5000,26697,27205,0,0,0.0000
    19900604,104250.0000,104562.5000,103937.5000,104437.5000,15315,25228,0,0,0.0000
    19900605,104575.0000,105250.0000,104425.0000,105137.5000,18054,25452,0,0,0.0000
    19900606,105187.5000,105437.5000,105112.5000,105300.0000,10104,24872,0,0,0.0000
    19900607,105250.0000,105450.0000,105137.5000,105425.0000,9931,24237,199009,199009,0.0000
    19900608,103562.5000,103562.5000,103125.0000,103437.5000,12791,9134,0,0,-1600.0000
    

    I intend to trim the data so that it does not include the last three columns.

    However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

    I suspect there is a setting somewhere which will allow this data to be handled.

    I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

    but can't figure out how to handle this data - any suggestions pls?



  • This sounds suspicious. I use futures data with missing days but don't have this issue.
    Do you load only one future when running your tests?

    Also, you should include your code.


  • administrators

    @j45p41 said in Furtures data with missing days - how to handle:

    I intend to trim the data so that it does not include the last three columns.

    There is no need to trim anything. Extra columns will be ignored (if using GenericCSVData for example)

    @j45p41 said in Furtures data with missing days - how to handle:

    However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

    Bogus statement. Please provide proof that trades are not processed correctly.

    @j45p41 said in Furtures data with missing days - how to handle:

    I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

    but can't figure out how to handle this data - any suggestions pls?

    The data shown above is completely normal. You show no code and the statements are not backed by any kind of evidence of anything. How can any suggestion be made?



  • My apologies - i thought there was a problem due to discontinues dates.

    here is my code in which I have disabled my analysers as they end up giving me a keyerror. When disabled I get a graph that does not execute trades.

    import datetime
    import time
    import os.path
    import sys
    import backtrader as bt
    import backtrader.feeds as btfeed
    import ccxt
    import csv
    import io
    import pandas as pd
    from collections import OrderedDict
    from multiprocessing import Pool, cpu_count
    import math
    import os
    
    
    
    # DECLARE MODE FOR PROGRAM - OPTOMISATION OR STRATEGY
    opt_mode = False
    DEBUG = False
    
    # LOG OUTPUT TO FILE
    class Logger(object):
        def __init__(self, filename="Default.log"):
            self.terminal = sys.stdout
            self.log = open(filename, "a")
    
        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)
    
        def flush(self):
            pass
    
    # CSV INPUT FILE FORMAT CONFIGURATION
    class dataFeed(btfeed.GenericCSVData):
        params = (
            ('dtformat', '%Y%m%d'),
            ('datetime', 0),
            ('open', 1),
            ('high', 2),
            ('low', 3),
            ('close', 4),
            ('volume', 5),
            ('openinterest', 6)
        )
    class OrderObserver(bt.observer.Observer):
        lines = ('created', 'expired',)
    
        plotinfo = dict(plot=True, subplot=True, plotlinelabels=True)
    
        plotlines = dict(
            created=dict(marker='*', markersize=8.0, color='lime', fillstyle='full'),
            expired=dict(marker='s', markersize=8.0, color='red', fillstyle='full')
        )
    
        def next(self):
            for order in self._owner._orderspending:
                if order.data is not self.data:
                    continue
    
                if not order.isbuy():
                    continue
    
                # Only interested in "buy" orders, because the sell orders
                # in the strategy are Market orders and will be immediately
                # executed
    
                if order.status in [bt.Order.Accepted, bt.Order.Submitted]:
                    self.lines.created[0] = order.created.price
    
                elif order.status in [bt.Order.Expired]:
                    self.lines.expired[0] = order.created.price
    
    # MAIN STRATEGY DEFINITION - DEFINE VALUES HERE FOR NON-OPTOMISATION MODE
    class firstStrategy(bt.Strategy):
        params = (
            ("period", 28),
            ("rsi_low", 30),
            ("rsi_high", 70),
        )
    
        #TRADE LOGGING FUNCTION
        def log(self, txt, dt=None):
            ''' Logging function fot this strategy'''
            if not opt_mode:
                dt = dt or self.datas[0].datetime.datetime(0)
                print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
            self.startcash = self.broker.getvalue()
            self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)
    
    
        #TRADE LOGGING FUNCTION
        def notify_trade(self, trade):
            if not trade.isclosed and not opt_mode:
                return
    
            self.log('TRADE INFO, PRICE  %.2f, GROSS %.2f, NET %.2f' %
                     (trade.price, trade.pnl, trade.pnlcomm))
    
        def next(self):
            if not self.position:
                if self.rsi < self.params.rsi_low:
                    self.buy(size=10)
            else:
                if self.rsi > self.params.rsi_high:
                    self.sell(size=10)
            if opt_mode and DEBUG:
                print('period: {}, rsi low: {}, rsi high {}'.format(self.params.period,self.params.rsi_low,self.params.rsi_high))
    
    # if opt_mode == False:
    #     def printTradeAnalysis(analyzer):
    #         '''
    #         Function to print the Technical Analysis results in a nice format.
    #         '''
    #         #Get the results we are interested in
    #         total_open = analyzer.total.open
    #         total_closed = analyzer.total.closed
    #         total_won = analyzer.won.total
    #         total_lost = analyzer.lost.total
    #         win_streak = analyzer.streak.won.longest
    #         lose_streak = analyzer.streak.lost.longest
    #         pnl_net = round(analyzer.pnl.net.total,2)
    #         strike_rate = round((total_won / total_closed) * 100,2)
    #         #Designate the rows
    #         h1 = ['Total Open', 'Total Closed', 'Total Won', 'Total Lost']
    #         h2 = ['Strike Rate','Win Streak', 'Losing Streak', 'PnL Net']
    #         r1 = [total_open, total_closed,total_won,total_lost]
    #         r2 = [strike_rate, win_streak, lose_streak, pnl_net]
    #         #Check which set of headers is the longest.
    #         if len(h1) > len(h2):
    #             header_length = len(h1)
    #         else:
    #             header_length = len(h2)
    #         #Print the rows
    #         print_list = [h1,r1,h2,r2]
    #         row_format ="{:<15}" * (header_length + 1)
    #         print("Trade Analysis Results:")
    #         for row in print_list:
    #             print(row_format.format('',*row))
    #
    #     def printSQN(analyzer):
    #         sqn = round(analyzer.sqn,2)
    #         print('SQN: {}'.format(sqn))
    
    # INPUT CONDITIONS TO FEED INTO CEREBRO IS ADDED HERE
    if __name__ == '__main__':
        sys.stdout = Logger("strat_evaluator.log")
    
    
        #periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12])
        periods = pd.DataFrame(columns=['FROM','TO'],index=[1])
        periods.loc[1] = ('1990-1-01','2017-12-31')
        #periods.loc[2] = ('2017-02-01','2017-03-01')
        #periods.loc[3] = ('2017-03-01','2017-04-01')
        #periods.loc[4] = ('2017-04-01','2017-05-01')
        #periods.loc[5] = ('2017-05-01','2017-06-01')
        #periods.loc[6] = ('2017-06-01','2017-07-01')
        #periods.loc[7] = ('2017-07-01','2017-08-01')
        #periods.loc[8] = ('2017-08-01','2017-09-01')
        #periods.loc[9] = ('2017-09-01','2017-10-01')
        #periods.loc[10] = ('2017-10-01','2017-11-01')
        #periods.loc[11] = ('2017-11-01','2017-12-01')
        #periods.loc[12] = ('2017-12-01','2017-12-31')
    
        for index, row in periods.iterrows():
    
    
            # Variable for our starting cash
            startcash = 10000
            # Create an instance of cerebro
            cerebro = bt.Cerebro(optreturn=False)
    
            # Timing the whole operation
            time_at_start = time.time()
    
            if opt_mode:
                # ADD STRATEGY OPTIMISATION
                #cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))
                cerebro.optstrategy(firstStrategy, period=range(13, 14), rsi_low=range(20, 21), rsi_high=range(40, 41))
                cerebro.addanalyzer(bt.analyzers.SharpeRatio)
            else:
                #ADD STRATEGY
                cerebro.addstrategy(firstStrategy)
    
            #format dates for datafeed object
            fy,fm,fd = periods['FROM'][index].split('-')
            ty,tm,td = periods['TO'][index].split('-')
    
    
    #READ DATA FROM CSV FILE
    
            filename = 'tickerData/F_BO.txt'
            modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
            datapath = os.path.join(modpath,str(filename))
            data = dataFeed(dataname=datapath, timeframe=bt.TimeFrame.Days,
                            fromdate=datetime.datetime(int(fy),int(fm),int(fd)),
                            todate=datetime.datetime(int(ty),int(tm),int(td)),)
    
            # Add the data to Cerebro
            cerebro.adddata(data)
    
            # Set our desired cash start
            cerebro.broker.setcash(startcash)
    
            if not opt_mode:
                # Add the analyzers we are interested in
                cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name="ta")
                cerebro.addanalyzer(bt.analyzers.SQN, _name="sqn")
    
    
            #WRITER TEST
            #cerebro.addwriter(bt.WriterFile, csv=True, rounding=2)
    
            # RUN STRATEGY THROUGH CEREBRO USING INPUT DATA
            # Timing the operation
            time_at_end = time.time()
            time_elapsed = round(time_at_end - time_at_start,2)
            print('Time elapsed: {} seconds'.format(time_elapsed))
            print ('Running Cerebro')
            if not opt_mode: opt_runs = cerebro.run(tradehistory=False)
            else: opt_runs = cerebro.run(tradehistory=False, runonce = False, Exactbars=True)
    
            firstStrat = opt_runs[0]
    
            if opt_mode:
                # CREATE A LIST VARIABLE THAT CONTAINS RESULTS
                final_results_list = []
                for run in opt_runs:
                    for strategy in run:
                        value = round(strategy.broker.get_value(), 2)
                        PnL = round(value - startcash, 2)
                        period = strategy.params.period
                        rsi_low = strategy.params.rsi_low
                        rsi_high = strategy.params.rsi_high
                        final_results_list.append([period, rsi_low, rsi_high, PnL])
    
                # Sort Results List
                by_period = sorted(final_results_list, key=lambda x: x[0])
                by_PnL = sorted(final_results_list, key=lambda x: x[3], reverse=True)
    
                # PRINT RESULTS IN OPTIMISATION AND FILTER TOP 3
                result_number = 0
                print('Results: Ordered by Profit:')
                for result in by_PnL:
                    if result_number < 3:
                        print('Asset: {} Start: {}, End: {}, Period: {}, rsi_low: {}, rsi_high: {}, PnL: {}'.format(filename, periods['FROM'][index], periods['TO'][index], result[0], result[1], result[2], result[3]))
                        result_number = result_number + 1
    
            # Timing the operation
            time_at_end = time.time()
            time_elapsed = round(time_at_end - time_at_start,2)
    
    
            print('Time elapsed: {} seconds'.format(time_elapsed))
            if opt_mode == False:
                # print the analyzers
                # printTradeAnalysis(firstStrat.analyzers.ta.get_analysis())
                # printSQN(firstStrat.analyzers.sqn.get_analysis())
    
                #Get final portfolio Value
                portvalue = cerebro.broker.getvalue()
    
                #Print out the final result
                print('Final Portfolio Value: ${}'.format(portvalue))
                cerebro.plot(style='candlestick')
    
            #cerebro.plot(style='candlestick')
    
    

    for completeness here is the data that I am using

    DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
    19900102,11520.0000,11520.0000,11316.0000,11322.0000,19779,38563,0,0,0.0000
    19900103,11322.0000,11460.0000,11298.0000,11442.0000,16270,39377,0,0,0.0000
    19900104,11472.0000,11520.0000,11340.0000,11346.0000,12743,39220,0,0,0.0000
    19900105,11298.0000,11466.0000,11286.0000,11418.0000,11808,39932,0,0,0.0000
    19900108,11448.0000,11808.0000,11442.0000,11784.0000,20844,40091,0,0,0.0000
    19900109,11730.0000,11820.0000,11682.0000,11778.0000,19457,39502,0,0,0.0000
    19900110,11760.0000,11778.0000,11598.0000,11724.0000,16266,39703,0,0,0.0000
    19900111,11712.0000,11790.0000,11706.0000,11754.0000,13699,39704,0,0,0.0000
    19900112,11826.0000,11844.0000,11634.0000,11640.0000,12677,38444,0,0,0.0000
    19900115,11598.0000,11688.0000,11580.0000,11652.0000,6067,38307,0,0,0.0000
    19900116,11610.0000,11640.0000,11490.0000,11496.0000,13691,37511,0,0,0.0000
    19900117,11484.0000,11616.0000,11484.0000,11502.0000,12378,38223,0,0,0.0000
    19900118,11508.0000,11580.0000,11496.0000,11538.0000,8570,37584,0,0,0.0000
    19900119,11562.0000,11568.0000,11400.0000,11406.0000,14732,36865,0,0,0.0000
    19900122,11400.0000,11448.0000,11316.0000,11358.0000,15163,36837,0,0,0.0000
    19900123,11382.0000,11508.0000,11376.0000,11466.0000,13141,36918,0,0,0.0000
    19900124,11472.0000,11514.0000,11436.0000,11448.0000,10550,36551,0,0,0.0000
    19900125,11472.0000,11478.0000,11340.0000,11340.0000,14579,35655,0,0,0.0000
    19900126,11340.0000,11430.0000,11328.0000,11418.0000,7547,34389,0,0,0.0000
    19900129,11430.0000,11544.0000,11370.0000,11538.0000,11285,34198,0,0,0.0000
    19900130,11544.0000,11574.0000,11442.0000,11460.0000,12167,33318,0,0,0.0000
    19900131,11460.0000,11544.0000,11430.0000,11502.0000,10708,32328,0,0,0.0000
    19900201,11520.0000,11700.0000,11502.0000,11532.0000,17957,32443,0,0,0.0000
    19900202,11544.0000,11580.0000,11508.0000,11568.0000,14151,31419,0,0,0.0000
    19900205,11568.0000,11634.0000,11460.0000,11478.0000,14241,30695,0,0,0.0000
    


  • @j45p41 Hi guys, with the additional info provided can you help me understand why my code is not processing trades? - thanks.