Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Furtures data with missing days - how to handle

    Indicators/Strategies/Analyzers
    3
    5
    1503
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      j45p41 last edited by

      Hi,

      I am trying to read in futures data in the following fomat:

      DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
      19900531,104687.5000,104850.0000,104387.5000,104525.0000,18592,32108,0,0,0.0000
      19900601,104750.0000,105512.5000,104612.5000,104762.5000,26697,27205,0,0,0.0000
      19900604,104250.0000,104562.5000,103937.5000,104437.5000,15315,25228,0,0,0.0000
      19900605,104575.0000,105250.0000,104425.0000,105137.5000,18054,25452,0,0,0.0000
      19900606,105187.5000,105437.5000,105112.5000,105300.0000,10104,24872,0,0,0.0000
      19900607,105250.0000,105450.0000,105137.5000,105425.0000,9931,24237,199009,199009,0.0000
      19900608,103562.5000,103562.5000,103125.0000,103437.5000,12791,9134,0,0,-1600.0000
      

      I intend to trim the data so that it does not include the last three columns.

      However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

      I suspect there is a setting somewhere which will allow this data to be handled.

      I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

      but can't figure out how to handle this data - any suggestions pls?

      B 1 Reply Last reply Reply Quote 0
      • L
        Laurent Michelizza last edited by

        This sounds suspicious. I use futures data with missing days but don't have this issue.
        Do you load only one future when running your tests?

        Also, you should include your code.

        1 Reply Last reply Reply Quote 1
        • B
          backtrader administrators @j45p41 last edited by

          @j45p41 said in Furtures data with missing days - how to handle:

          I intend to trim the data so that it does not include the last three columns.

          There is no need to trim anything. Extra columns will be ignored (if using GenericCSVData for example)

          @j45p41 said in Furtures data with missing days - how to handle:

          However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.

          Bogus statement. Please provide proof that trades are not processed correctly.

          @j45p41 said in Furtures data with missing days - how to handle:

          I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html

          but can't figure out how to handle this data - any suggestions pls?

          The data shown above is completely normal. You show no code and the statements are not backed by any kind of evidence of anything. How can any suggestion be made?

          1 Reply Last reply Reply Quote 1
          • J
            j45p41 last edited by

            My apologies - i thought there was a problem due to discontinues dates.

            here is my code in which I have disabled my analysers as they end up giving me a keyerror. When disabled I get a graph that does not execute trades.

            import datetime
            import time
            import os.path
            import sys
            import backtrader as bt
            import backtrader.feeds as btfeed
            import ccxt
            import csv
            import io
            import pandas as pd
            from collections import OrderedDict
            from multiprocessing import Pool, cpu_count
            import math
            import os
            
            
            
            # DECLARE MODE FOR PROGRAM - OPTOMISATION OR STRATEGY
            opt_mode = False
            DEBUG = False
            
            # LOG OUTPUT TO FILE
            class Logger(object):
                def __init__(self, filename="Default.log"):
                    self.terminal = sys.stdout
                    self.log = open(filename, "a")
            
                def write(self, message):
                    self.terminal.write(message)
                    self.log.write(message)
            
                def flush(self):
                    pass
            
            # CSV INPUT FILE FORMAT CONFIGURATION
            class dataFeed(btfeed.GenericCSVData):
                params = (
                    ('dtformat', '%Y%m%d'),
                    ('datetime', 0),
                    ('open', 1),
                    ('high', 2),
                    ('low', 3),
                    ('close', 4),
                    ('volume', 5),
                    ('openinterest', 6)
                )
            class OrderObserver(bt.observer.Observer):
                lines = ('created', 'expired',)
            
                plotinfo = dict(plot=True, subplot=True, plotlinelabels=True)
            
                plotlines = dict(
                    created=dict(marker='*', markersize=8.0, color='lime', fillstyle='full'),
                    expired=dict(marker='s', markersize=8.0, color='red', fillstyle='full')
                )
            
                def next(self):
                    for order in self._owner._orderspending:
                        if order.data is not self.data:
                            continue
            
                        if not order.isbuy():
                            continue
            
                        # Only interested in "buy" orders, because the sell orders
                        # in the strategy are Market orders and will be immediately
                        # executed
            
                        if order.status in [bt.Order.Accepted, bt.Order.Submitted]:
                            self.lines.created[0] = order.created.price
            
                        elif order.status in [bt.Order.Expired]:
                            self.lines.expired[0] = order.created.price
            
            # MAIN STRATEGY DEFINITION - DEFINE VALUES HERE FOR NON-OPTOMISATION MODE
            class firstStrategy(bt.Strategy):
                params = (
                    ("period", 28),
                    ("rsi_low", 30),
                    ("rsi_high", 70),
                )
            
                #TRADE LOGGING FUNCTION
                def log(self, txt, dt=None):
                    ''' Logging function fot this strategy'''
                    if not opt_mode:
                        dt = dt or self.datas[0].datetime.datetime(0)
                        print('%s, %s' % (dt.isoformat(), txt))
            
                def __init__(self):
                    self.startcash = self.broker.getvalue()
                    self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True)
            
            
                #TRADE LOGGING FUNCTION
                def notify_trade(self, trade):
                    if not trade.isclosed and not opt_mode:
                        return
            
                    self.log('TRADE INFO, PRICE  %.2f, GROSS %.2f, NET %.2f' %
                             (trade.price, trade.pnl, trade.pnlcomm))
            
                def next(self):
                    if not self.position:
                        if self.rsi < self.params.rsi_low:
                            self.buy(size=10)
                    else:
                        if self.rsi > self.params.rsi_high:
                            self.sell(size=10)
                    if opt_mode and DEBUG:
                        print('period: {}, rsi low: {}, rsi high {}'.format(self.params.period,self.params.rsi_low,self.params.rsi_high))
            
            # if opt_mode == False:
            #     def printTradeAnalysis(analyzer):
            #         '''
            #         Function to print the Technical Analysis results in a nice format.
            #         '''
            #         #Get the results we are interested in
            #         total_open = analyzer.total.open
            #         total_closed = analyzer.total.closed
            #         total_won = analyzer.won.total
            #         total_lost = analyzer.lost.total
            #         win_streak = analyzer.streak.won.longest
            #         lose_streak = analyzer.streak.lost.longest
            #         pnl_net = round(analyzer.pnl.net.total,2)
            #         strike_rate = round((total_won / total_closed) * 100,2)
            #         #Designate the rows
            #         h1 = ['Total Open', 'Total Closed', 'Total Won', 'Total Lost']
            #         h2 = ['Strike Rate','Win Streak', 'Losing Streak', 'PnL Net']
            #         r1 = [total_open, total_closed,total_won,total_lost]
            #         r2 = [strike_rate, win_streak, lose_streak, pnl_net]
            #         #Check which set of headers is the longest.
            #         if len(h1) > len(h2):
            #             header_length = len(h1)
            #         else:
            #             header_length = len(h2)
            #         #Print the rows
            #         print_list = [h1,r1,h2,r2]
            #         row_format ="{:<15}" * (header_length + 1)
            #         print("Trade Analysis Results:")
            #         for row in print_list:
            #             print(row_format.format('',*row))
            #
            #     def printSQN(analyzer):
            #         sqn = round(analyzer.sqn,2)
            #         print('SQN: {}'.format(sqn))
            
            # INPUT CONDITIONS TO FEED INTO CEREBRO IS ADDED HERE
            if __name__ == '__main__':
                sys.stdout = Logger("strat_evaluator.log")
            
            
                #periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12])
                periods = pd.DataFrame(columns=['FROM','TO'],index=[1])
                periods.loc[1] = ('1990-1-01','2017-12-31')
                #periods.loc[2] = ('2017-02-01','2017-03-01')
                #periods.loc[3] = ('2017-03-01','2017-04-01')
                #periods.loc[4] = ('2017-04-01','2017-05-01')
                #periods.loc[5] = ('2017-05-01','2017-06-01')
                #periods.loc[6] = ('2017-06-01','2017-07-01')
                #periods.loc[7] = ('2017-07-01','2017-08-01')
                #periods.loc[8] = ('2017-08-01','2017-09-01')
                #periods.loc[9] = ('2017-09-01','2017-10-01')
                #periods.loc[10] = ('2017-10-01','2017-11-01')
                #periods.loc[11] = ('2017-11-01','2017-12-01')
                #periods.loc[12] = ('2017-12-01','2017-12-31')
            
                for index, row in periods.iterrows():
            
            
                    # Variable for our starting cash
                    startcash = 10000
                    # Create an instance of cerebro
                    cerebro = bt.Cerebro(optreturn=False)
            
                    # Timing the whole operation
                    time_at_start = time.time()
            
                    if opt_mode:
                        # ADD STRATEGY OPTIMISATION
                        #cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85))
                        cerebro.optstrategy(firstStrategy, period=range(13, 14), rsi_low=range(20, 21), rsi_high=range(40, 41))
                        cerebro.addanalyzer(bt.analyzers.SharpeRatio)
                    else:
                        #ADD STRATEGY
                        cerebro.addstrategy(firstStrategy)
            
                    #format dates for datafeed object
                    fy,fm,fd = periods['FROM'][index].split('-')
                    ty,tm,td = periods['TO'][index].split('-')
            
            
            #READ DATA FROM CSV FILE
            
                    filename = 'tickerData/F_BO.txt'
                    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
                    datapath = os.path.join(modpath,str(filename))
                    data = dataFeed(dataname=datapath, timeframe=bt.TimeFrame.Days,
                                    fromdate=datetime.datetime(int(fy),int(fm),int(fd)),
                                    todate=datetime.datetime(int(ty),int(tm),int(td)),)
            
                    # Add the data to Cerebro
                    cerebro.adddata(data)
            
                    # Set our desired cash start
                    cerebro.broker.setcash(startcash)
            
                    if not opt_mode:
                        # Add the analyzers we are interested in
                        cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name="ta")
                        cerebro.addanalyzer(bt.analyzers.SQN, _name="sqn")
            
            
                    #WRITER TEST
                    #cerebro.addwriter(bt.WriterFile, csv=True, rounding=2)
            
                    # RUN STRATEGY THROUGH CEREBRO USING INPUT DATA
                    # Timing the operation
                    time_at_end = time.time()
                    time_elapsed = round(time_at_end - time_at_start,2)
                    print('Time elapsed: {} seconds'.format(time_elapsed))
                    print ('Running Cerebro')
                    if not opt_mode: opt_runs = cerebro.run(tradehistory=False)
                    else: opt_runs = cerebro.run(tradehistory=False, runonce = False, Exactbars=True)
            
                    firstStrat = opt_runs[0]
            
                    if opt_mode:
                        # CREATE A LIST VARIABLE THAT CONTAINS RESULTS
                        final_results_list = []
                        for run in opt_runs:
                            for strategy in run:
                                value = round(strategy.broker.get_value(), 2)
                                PnL = round(value - startcash, 2)
                                period = strategy.params.period
                                rsi_low = strategy.params.rsi_low
                                rsi_high = strategy.params.rsi_high
                                final_results_list.append([period, rsi_low, rsi_high, PnL])
            
                        # Sort Results List
                        by_period = sorted(final_results_list, key=lambda x: x[0])
                        by_PnL = sorted(final_results_list, key=lambda x: x[3], reverse=True)
            
                        # PRINT RESULTS IN OPTIMISATION AND FILTER TOP 3
                        result_number = 0
                        print('Results: Ordered by Profit:')
                        for result in by_PnL:
                            if result_number < 3:
                                print('Asset: {} Start: {}, End: {}, Period: {}, rsi_low: {}, rsi_high: {}, PnL: {}'.format(filename, periods['FROM'][index], periods['TO'][index], result[0], result[1], result[2], result[3]))
                                result_number = result_number + 1
            
                    # Timing the operation
                    time_at_end = time.time()
                    time_elapsed = round(time_at_end - time_at_start,2)
            
            
                    print('Time elapsed: {} seconds'.format(time_elapsed))
                    if opt_mode == False:
                        # print the analyzers
                        # printTradeAnalysis(firstStrat.analyzers.ta.get_analysis())
                        # printSQN(firstStrat.analyzers.sqn.get_analysis())
            
                        #Get final portfolio Value
                        portvalue = cerebro.broker.getvalue()
            
                        #Print out the final result
                        print('Final Portfolio Value: ${}'.format(portvalue))
                        cerebro.plot(style='candlestick')
            
                    #cerebro.plot(style='candlestick')
            
            

            for completeness here is the data that I am using

            DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO
            19900102,11520.0000,11520.0000,11316.0000,11322.0000,19779,38563,0,0,0.0000
            19900103,11322.0000,11460.0000,11298.0000,11442.0000,16270,39377,0,0,0.0000
            19900104,11472.0000,11520.0000,11340.0000,11346.0000,12743,39220,0,0,0.0000
            19900105,11298.0000,11466.0000,11286.0000,11418.0000,11808,39932,0,0,0.0000
            19900108,11448.0000,11808.0000,11442.0000,11784.0000,20844,40091,0,0,0.0000
            19900109,11730.0000,11820.0000,11682.0000,11778.0000,19457,39502,0,0,0.0000
            19900110,11760.0000,11778.0000,11598.0000,11724.0000,16266,39703,0,0,0.0000
            19900111,11712.0000,11790.0000,11706.0000,11754.0000,13699,39704,0,0,0.0000
            19900112,11826.0000,11844.0000,11634.0000,11640.0000,12677,38444,0,0,0.0000
            19900115,11598.0000,11688.0000,11580.0000,11652.0000,6067,38307,0,0,0.0000
            19900116,11610.0000,11640.0000,11490.0000,11496.0000,13691,37511,0,0,0.0000
            19900117,11484.0000,11616.0000,11484.0000,11502.0000,12378,38223,0,0,0.0000
            19900118,11508.0000,11580.0000,11496.0000,11538.0000,8570,37584,0,0,0.0000
            19900119,11562.0000,11568.0000,11400.0000,11406.0000,14732,36865,0,0,0.0000
            19900122,11400.0000,11448.0000,11316.0000,11358.0000,15163,36837,0,0,0.0000
            19900123,11382.0000,11508.0000,11376.0000,11466.0000,13141,36918,0,0,0.0000
            19900124,11472.0000,11514.0000,11436.0000,11448.0000,10550,36551,0,0,0.0000
            19900125,11472.0000,11478.0000,11340.0000,11340.0000,14579,35655,0,0,0.0000
            19900126,11340.0000,11430.0000,11328.0000,11418.0000,7547,34389,0,0,0.0000
            19900129,11430.0000,11544.0000,11370.0000,11538.0000,11285,34198,0,0,0.0000
            19900130,11544.0000,11574.0000,11442.0000,11460.0000,12167,33318,0,0,0.0000
            19900131,11460.0000,11544.0000,11430.0000,11502.0000,10708,32328,0,0,0.0000
            19900201,11520.0000,11700.0000,11502.0000,11532.0000,17957,32443,0,0,0.0000
            19900202,11544.0000,11580.0000,11508.0000,11568.0000,14151,31419,0,0,0.0000
            19900205,11568.0000,11634.0000,11460.0000,11478.0000,14241,30695,0,0,0.0000
            
            J 1 Reply Last reply Reply Quote 0
            • J
              j45p41 @j45p41 last edited by

              @j45p41 Hi guys, with the additional info provided can you help me understand why my code is not processing trades? - thanks.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors