For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
Furtures data with missing days - how to handle
-
Hi,
I am trying to read in futures data in the following fomat:
DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO 19900531,104687.5000,104850.0000,104387.5000,104525.0000,18592,32108,0,0,0.0000 19900601,104750.0000,105512.5000,104612.5000,104762.5000,26697,27205,0,0,0.0000 19900604,104250.0000,104562.5000,103937.5000,104437.5000,15315,25228,0,0,0.0000 19900605,104575.0000,105250.0000,104425.0000,105137.5000,18054,25452,0,0,0.0000 19900606,105187.5000,105437.5000,105112.5000,105300.0000,10104,24872,0,0,0.0000 19900607,105250.0000,105450.0000,105137.5000,105425.0000,9931,24237,199009,199009,0.0000 19900608,103562.5000,103562.5000,103125.0000,103437.5000,12791,9134,0,0,-1600.0000I intend to trim the data so that it does not include the last three columns.
However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.
I suspect there is a setting somewhere which will allow this data to be handled.
I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html
but can't figure out how to handle this data - any suggestions pls?
-
This sounds suspicious. I use futures data with missing days but don't have this issue.
Do you load only one future when running your tests?Also, you should include your code.
-
@j45p41 said in Furtures data with missing days - how to handle:
I intend to trim the data so that it does not include the last three columns.
There is no need to trim anything. Extra columns will be ignored (if using
GenericCSVDatafor example)@j45p41 said in Furtures data with missing days - how to handle:
However when I run using this data, due to it not being continuous days cerebro does not process trades correctly - the graphs is plotted but the trades do not execute.
Bogus statement. Please provide proof that trades are not processed correctly.
@j45p41 said in Furtures data with missing days - how to handle:
I have read: https://www.backtrader.com/blog/posts/2016-08-31-rolling-over-futures/rolling-futures-over.html
but can't figure out how to handle this data - any suggestions pls?
The data shown above is completely normal. You show no code and the statements are not backed by any kind of evidence of anything. How can any suggestion be made?
-
My apologies - i thought there was a problem due to discontinues dates.
here is my code in which I have disabled my analysers as they end up giving me a keyerror. When disabled I get a graph that does not execute trades.
import datetime import time import os.path import sys import backtrader as bt import backtrader.feeds as btfeed import ccxt import csv import io import pandas as pd from collections import OrderedDict from multiprocessing import Pool, cpu_count import math import os # DECLARE MODE FOR PROGRAM - OPTOMISATION OR STRATEGY opt_mode = False DEBUG = False # LOG OUTPUT TO FILE class Logger(object): def __init__(self, filename="Default.log"): self.terminal = sys.stdout self.log = open(filename, "a") def write(self, message): self.terminal.write(message) self.log.write(message) def flush(self): pass # CSV INPUT FILE FORMAT CONFIGURATION class dataFeed(btfeed.GenericCSVData): params = ( ('dtformat', '%Y%m%d'), ('datetime', 0), ('open', 1), ('high', 2), ('low', 3), ('close', 4), ('volume', 5), ('openinterest', 6) ) class OrderObserver(bt.observer.Observer): lines = ('created', 'expired',) plotinfo = dict(plot=True, subplot=True, plotlinelabels=True) plotlines = dict( created=dict(marker='*', markersize=8.0, color='lime', fillstyle='full'), expired=dict(marker='s', markersize=8.0, color='red', fillstyle='full') ) def next(self): for order in self._owner._orderspending: if order.data is not self.data: continue if not order.isbuy(): continue # Only interested in "buy" orders, because the sell orders # in the strategy are Market orders and will be immediately # executed if order.status in [bt.Order.Accepted, bt.Order.Submitted]: self.lines.created[0] = order.created.price elif order.status in [bt.Order.Expired]: self.lines.expired[0] = order.created.price # MAIN STRATEGY DEFINITION - DEFINE VALUES HERE FOR NON-OPTOMISATION MODE class firstStrategy(bt.Strategy): params = ( ("period", 28), ("rsi_low", 30), ("rsi_high", 70), ) #TRADE LOGGING FUNCTION def log(self, txt, dt=None): ''' Logging function fot this strategy''' if not opt_mode: dt = dt or self.datas[0].datetime.datetime(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): self.startcash = self.broker.getvalue() self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period, safediv=True) #TRADE LOGGING FUNCTION def notify_trade(self, trade): if not trade.isclosed and not opt_mode: return self.log('TRADE INFO, PRICE %.2f, GROSS %.2f, NET %.2f' % (trade.price, trade.pnl, trade.pnlcomm)) def next(self): if not self.position: if self.rsi < self.params.rsi_low: self.buy(size=10) else: if self.rsi > self.params.rsi_high: self.sell(size=10) if opt_mode and DEBUG: print('period: {}, rsi low: {}, rsi high {}'.format(self.params.period,self.params.rsi_low,self.params.rsi_high)) # if opt_mode == False: # def printTradeAnalysis(analyzer): # ''' # Function to print the Technical Analysis results in a nice format. # ''' # #Get the results we are interested in # total_open = analyzer.total.open # total_closed = analyzer.total.closed # total_won = analyzer.won.total # total_lost = analyzer.lost.total # win_streak = analyzer.streak.won.longest # lose_streak = analyzer.streak.lost.longest # pnl_net = round(analyzer.pnl.net.total,2) # strike_rate = round((total_won / total_closed) * 100,2) # #Designate the rows # h1 = ['Total Open', 'Total Closed', 'Total Won', 'Total Lost'] # h2 = ['Strike Rate','Win Streak', 'Losing Streak', 'PnL Net'] # r1 = [total_open, total_closed,total_won,total_lost] # r2 = [strike_rate, win_streak, lose_streak, pnl_net] # #Check which set of headers is the longest. # if len(h1) > len(h2): # header_length = len(h1) # else: # header_length = len(h2) # #Print the rows # print_list = [h1,r1,h2,r2] # row_format ="{:<15}" * (header_length + 1) # print("Trade Analysis Results:") # for row in print_list: # print(row_format.format('',*row)) # # def printSQN(analyzer): # sqn = round(analyzer.sqn,2) # print('SQN: {}'.format(sqn)) # INPUT CONDITIONS TO FEED INTO CEREBRO IS ADDED HERE if __name__ == '__main__': sys.stdout = Logger("strat_evaluator.log") #periods = pd.DataFrame(columns=['FROM','TO'],index=[1,2,3,4,5,6,7,8,9,10,11,12]) periods = pd.DataFrame(columns=['FROM','TO'],index=[1]) periods.loc[1] = ('1990-1-01','2017-12-31') #periods.loc[2] = ('2017-02-01','2017-03-01') #periods.loc[3] = ('2017-03-01','2017-04-01') #periods.loc[4] = ('2017-04-01','2017-05-01') #periods.loc[5] = ('2017-05-01','2017-06-01') #periods.loc[6] = ('2017-06-01','2017-07-01') #periods.loc[7] = ('2017-07-01','2017-08-01') #periods.loc[8] = ('2017-08-01','2017-09-01') #periods.loc[9] = ('2017-09-01','2017-10-01') #periods.loc[10] = ('2017-10-01','2017-11-01') #periods.loc[11] = ('2017-11-01','2017-12-01') #periods.loc[12] = ('2017-12-01','2017-12-31') for index, row in periods.iterrows(): # Variable for our starting cash startcash = 10000 # Create an instance of cerebro cerebro = bt.Cerebro(optreturn=False) # Timing the whole operation time_at_start = time.time() if opt_mode: # ADD STRATEGY OPTIMISATION #cerebro.optstrategy(firstStrategy, period=range(11, 20), rsi_low=range(10, 50), rsi_high=range(51, 85)) cerebro.optstrategy(firstStrategy, period=range(13, 14), rsi_low=range(20, 21), rsi_high=range(40, 41)) cerebro.addanalyzer(bt.analyzers.SharpeRatio) else: #ADD STRATEGY cerebro.addstrategy(firstStrategy) #format dates for datafeed object fy,fm,fd = periods['FROM'][index].split('-') ty,tm,td = periods['TO'][index].split('-') #READ DATA FROM CSV FILE filename = 'tickerData/F_BO.txt' modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath,str(filename)) data = dataFeed(dataname=datapath, timeframe=bt.TimeFrame.Days, fromdate=datetime.datetime(int(fy),int(fm),int(fd)), todate=datetime.datetime(int(ty),int(tm),int(td)),) # Add the data to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(startcash) if not opt_mode: # Add the analyzers we are interested in cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name="ta") cerebro.addanalyzer(bt.analyzers.SQN, _name="sqn") #WRITER TEST #cerebro.addwriter(bt.WriterFile, csv=True, rounding=2) # RUN STRATEGY THROUGH CEREBRO USING INPUT DATA # Timing the operation time_at_end = time.time() time_elapsed = round(time_at_end - time_at_start,2) print('Time elapsed: {} seconds'.format(time_elapsed)) print ('Running Cerebro') if not opt_mode: opt_runs = cerebro.run(tradehistory=False) else: opt_runs = cerebro.run(tradehistory=False, runonce = False, Exactbars=True) firstStrat = opt_runs[0] if opt_mode: # CREATE A LIST VARIABLE THAT CONTAINS RESULTS final_results_list = [] for run in opt_runs: for strategy in run: value = round(strategy.broker.get_value(), 2) PnL = round(value - startcash, 2) period = strategy.params.period rsi_low = strategy.params.rsi_low rsi_high = strategy.params.rsi_high final_results_list.append([period, rsi_low, rsi_high, PnL]) # Sort Results List by_period = sorted(final_results_list, key=lambda x: x[0]) by_PnL = sorted(final_results_list, key=lambda x: x[3], reverse=True) # PRINT RESULTS IN OPTIMISATION AND FILTER TOP 3 result_number = 0 print('Results: Ordered by Profit:') for result in by_PnL: if result_number < 3: print('Asset: {} Start: {}, End: {}, Period: {}, rsi_low: {}, rsi_high: {}, PnL: {}'.format(filename, periods['FROM'][index], periods['TO'][index], result[0], result[1], result[2], result[3])) result_number = result_number + 1 # Timing the operation time_at_end = time.time() time_elapsed = round(time_at_end - time_at_start,2) print('Time elapsed: {} seconds'.format(time_elapsed)) if opt_mode == False: # print the analyzers # printTradeAnalysis(firstStrat.analyzers.ta.get_analysis()) # printSQN(firstStrat.analyzers.sqn.get_analysis()) #Get final portfolio Value portvalue = cerebro.broker.getvalue() #Print out the final result print('Final Portfolio Value: ${}'.format(portvalue)) cerebro.plot(style='candlestick') #cerebro.plot(style='candlestick')for completeness here is the data that I am using
DATE, OPEN, HIGH, LOW, CLOSE, VOL, OI, P, R, RINFO 19900102,11520.0000,11520.0000,11316.0000,11322.0000,19779,38563,0,0,0.0000 19900103,11322.0000,11460.0000,11298.0000,11442.0000,16270,39377,0,0,0.0000 19900104,11472.0000,11520.0000,11340.0000,11346.0000,12743,39220,0,0,0.0000 19900105,11298.0000,11466.0000,11286.0000,11418.0000,11808,39932,0,0,0.0000 19900108,11448.0000,11808.0000,11442.0000,11784.0000,20844,40091,0,0,0.0000 19900109,11730.0000,11820.0000,11682.0000,11778.0000,19457,39502,0,0,0.0000 19900110,11760.0000,11778.0000,11598.0000,11724.0000,16266,39703,0,0,0.0000 19900111,11712.0000,11790.0000,11706.0000,11754.0000,13699,39704,0,0,0.0000 19900112,11826.0000,11844.0000,11634.0000,11640.0000,12677,38444,0,0,0.0000 19900115,11598.0000,11688.0000,11580.0000,11652.0000,6067,38307,0,0,0.0000 19900116,11610.0000,11640.0000,11490.0000,11496.0000,13691,37511,0,0,0.0000 19900117,11484.0000,11616.0000,11484.0000,11502.0000,12378,38223,0,0,0.0000 19900118,11508.0000,11580.0000,11496.0000,11538.0000,8570,37584,0,0,0.0000 19900119,11562.0000,11568.0000,11400.0000,11406.0000,14732,36865,0,0,0.0000 19900122,11400.0000,11448.0000,11316.0000,11358.0000,15163,36837,0,0,0.0000 19900123,11382.0000,11508.0000,11376.0000,11466.0000,13141,36918,0,0,0.0000 19900124,11472.0000,11514.0000,11436.0000,11448.0000,10550,36551,0,0,0.0000 19900125,11472.0000,11478.0000,11340.0000,11340.0000,14579,35655,0,0,0.0000 19900126,11340.0000,11430.0000,11328.0000,11418.0000,7547,34389,0,0,0.0000 19900129,11430.0000,11544.0000,11370.0000,11538.0000,11285,34198,0,0,0.0000 19900130,11544.0000,11574.0000,11442.0000,11460.0000,12167,33318,0,0,0.0000 19900131,11460.0000,11544.0000,11430.0000,11502.0000,10708,32328,0,0,0.0000 19900201,11520.0000,11700.0000,11502.0000,11532.0000,17957,32443,0,0,0.0000 19900202,11544.0000,11580.0000,11508.0000,11568.0000,14151,31419,0,0,0.0000 19900205,11568.0000,11634.0000,11460.0000,11478.0000,14241,30695,0,0,0.0000
-
@j45p41 Hi guys, with the additional info provided can you help me understand why my code is not processing trades? - thanks.