create a CSV of the result of optstrategy



  • hello,
    i'm trying to print the result of optstrategy for this i to use something like this:

    import csv
    b = open('test.csv', 'a')
    a = csv.writer(b,delimiter=',')
    list = ['test', 'self.broker.startingcash', 'self.broker.getvalue()']
    a.writerows([list])
    b.close()
    

    that i try to insert in the def stop section. it doesn't work.
    anyone has a clue ?
    here the code i run:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import datetime  # For datetime objects
    import os.path  # To manage paths
    import sys  # To find out the script name (in argv[0])
    import backtrader.feeds as btfeeds
    import numpy as np
    import csv
    
    # Import the backtrader platform
    import backtrader as bt
    
    
    # Create a Stratey
    class TestStrategy(bt.Strategy):
        params = (
            ('maperiod', 15),
            ('printlog', False),
        )
    
        def log(self, txt, dt=None, doprint=False):
            ''' Logging function fot this strategy'''
            if self.params.printlog or doprint:
                dt = dt or self.datas[0].datetime.date(0)
                print('%s, %s' % (dt.isoformat(), txt))
    
        def __init__(self):
            # Keep a reference to the "close" line in the data[0] dataseries
            self.dataclose = self.datas[0].close
    
            # To keep track of pending orders and buy price/commission
            self.order = None
            self.buyprice = None
            self.buycomm = None
    
            # Add a MovingAverageSimple indicator
            self.sma = bt.indicators.SimpleMovingAverage(
                self.datas[0], period=self.params.maperiod)
    
        def notify_order(self, order):
            if order.status in [order.Submitted, order.Accepted]:
                # Buy/Sell order submitted/accepted to/by broker - Nothing to do
                return
    
            # Check if an order has been completed
            # Attention: broker could reject order if not enougth cash
            if order.status in [order.Completed, order.Canceled, order.Margin]:
                if order.isbuy():
                    self.log(
                        'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                        (order.executed.price,
                         order.executed.value,
                         order.executed.comm))
    
                    self.buyprice = order.executed.price
                    self.buycomm = order.executed.comm
                else:  # Sell
                    self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                             (order.executed.price,
                              order.executed.value,
                              order.executed.comm))
    
                self.bar_executed = len(self)
    
            # Write down: no pending order
            self.order = None
    
        def notify_trade(self, trade):
            if not trade.isclosed:
                return
    
            self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
                     (trade.pnl, trade.pnlcomm))
    
        def next(self):
            # Simply log the closing price of the series from the reference
            self.log('Close, %.2f' % self.dataclose[0])
    
            # Check if an order is pending ... if yes, we cannot send a 2nd one
            if self.order:
                return
    
            # Check if we are in the market
            if not self.position:
    
                # Not yet ... we MIGHT BUY if ...
                if self.dataclose[0] > self.sma[0]:
    
                    # BUY, BUY, BUY!!! (with all possible default parameters)
                    self.log('BUY CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.buy()
    
            else:
    
                if self.dataclose[0] < self.sma[0]:
                    # SELL, SELL, SELL!!! (with all possible default parameters)
                    self.log('SELL CREATE, %.2f' % self.dataclose[0])
    
                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.sell()
    
        def stop(self):
            self.log('(MA Period %2d) Ending Value %.2f' %
                     (self.params.maperiod, self.broker.getvalue()), doprint=True)
            list = [self.params.maperiod, self.broker.startingcash, self.broker.getvalue()]
            b = open('test.csv', 'a')
            a = csv.writer(b)
            a.writerows([list])
            b.close()
    
    
    if __name__ == '__main__':
        # Create a cerebro entity
        cerebro = bt.Cerebro()
    
        # Add a strategy
        strats = cerebro.optstrategy(
            TestStrategy,
            maperiod=range(10, 31))
    
        # Datas are in a subfolder of the samples. Need to find where the script is
        # because it could have been called from anywhere
        modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
        datapath = os.path.join(modpath, './GLDGDX/GLD20151130to20170124.csv')
    
        fromdate = datetime.datetime(2016, 10, 02)  # datetime.datetime.strptime(args.fromdate, '%Y-%m-%d')
        todate = datetime.datetime(2016, 11, 20)  # datetime.datetime.strptime(args.todate, '%Y-%m-%d')
    
        # Create the 2nd data
    
        data = btfeeds.GenericCSVData(
            dataname=datapath,
    
            fromdate=fromdate,
            todate=todate,
    
            nullvalue=0.0,
    
            dtformat=('%Y-%m-%d %H:%M:%S'),
            datetime=1,
            high=3,
            low=4,
            open=2,
            close=5,
            volume=6,
            openinterest=-1,
        )
        # Add the Data Feed to Cerebro
        cerebro.adddata(data)
    
        # Set our desired cash start
        cerebro.broker.setcash(10000000.0)
    
        # Add a FixedSize sizer according to the stake
        cerebro.addsizer(bt.sizers.FixedSize, stake=10)
    
        # Set the commission
        cerebro.broker.setcommission(commission=0.0)
    
        # Run over everything
        cerebro.run()
    


  • try this:
    cerebro.addwriter(bt.WriterFile, csv = True, out='your_strategy_results')



  • Maybe you need to try a.writerow([list]).

    What error was returned?



  • @ab_trader
    it was working in fact with an s or without and s.
    I didn't pay close attention enough.

    here the result with the originale code:

    14,10000000.0,9999912.600000035
    
    16,10000000.0,9999920.800000025
    
    12,10000000.0,9999908.300000027
    
    10,10000000.0,9999897.00000004
    
    17,10000000.0,9999929.200000012
    
    15,10000000.0,9999921.400000028
    
    13,10000000.0,9999907.600000022
    
    11,10000000.0,9999905.400000053
    
    18,10000000.0,9999936.400000017
    
    20,10000000.0,9999941.199999996
    
    22,10000000.0,9999932.200000025
    
    24,10000000.0,9999937.000000019
    
    21,10000000.0,9999940.200000029
    
    19,10000000.0,9999935.100000003
    
    23,10000000.0,9999937.500000019
    
    25,10000000.0,9999938.100000018
    
    30,10000000.0,9999946.000000006
    
    28,10000000.0,9999943.700000001
    
    26,10000000.0,9999942.499999996
    
    29,10000000.0,9999954.699999997
    
    27,10000000.0,9999943.599999996
    .
    

    I have space between lines I don't how to avoid to create them



  • @Brad-Lloyd
    hello Brad , I want to be able to control the format of the exit.
    when I used the line you proposed :

    Traceback (most recent call last):
      File "C:/Users/mteddy/PycharmProjects/datacollection/Datahackathon/letsoptimise.py", line 168, in <module>
        cerebro.run()
      File "C:\Users\mteddy\Anaconda2\lib\site-packages\backtrader\cerebro.py", line 801, in run
        self.runstrats = list(pool.map(self, iterstrats))
      File "C:\Users\mteddy\Anaconda2\lib\multiprocessing\pool.py", line 251, in map
        return self.map_async(func, iterable, chunksize).get()
      File "C:\Users\mteddy\Anaconda2\lib\multiprocessing\pool.py", line 567, in get
        raise self._value
    ValueError: I/O operation on closed file
    



  • administrators

    This is really an unexpected use case, because the writer works by collecting data in real time and there is single writer instance running in the master process and not an instance per worker. Hence the closed file error notification, because the process forking has invalidated the existing file descriptor in the master.

    It should be clear that if one tried to write to the same file from multiple processes, something unexpected would be happening.

    The writer (with csv=True will also print all the values the data feeds have produced and that wouldn't help with multiple processes which do actually process the same data feeds)

    Cerebro returns (a cut-down version unless optreturn=False) the strategies which have been executed. The best way to print things out is to rely on the values collected by the analyzers attached to the strategies. Of course, collecting value and cash in the stop method is a valid approach too.

    Let's assume the case in which a single strategy is being optimized

    
    class MyStrategy(bt.Strategy):
        ...
        def stop(self):
            self.thevalue = self.broker.get_value()
            self.thecash = self.broker.get_cash()
    
    cerebro.optstrategy(MyStrategy, myparam1=range(val1, valn), ...)
    
    strats = cerebro.run()  # list of list
    
    # list comprehensions separated for clarity
    # Get the element 0 (there was only 1 strategy in each run) of each optimization
    st0 = [s[0] for s in strats]
    
    # Get the period, value and cash in text format comma separated in a list from each strategy
    periodcashvalues = [','.join([str(s.p.maperiod), str(s.thecash), str(s.thevalue)]) for s in st0]
    
    # print out the generated list separating with newlines
    print('\n'.join(cashvalues))
    


  • @backtrader said in create a CSV of the result of optstrategy:

    The writer (with csv=True will also print all the values the data feeds have produced and that wouldn't help with multiple processes which do actually process the same data feeds)

    Cerebro returns (a cut-down version unless optreturn=False) the strategies which have been executed. The best way to print things out is to rely on the values collected by the analyzers attached to the strategies. Of course, collecting value and cash in the stop method is a valid approach too.

    How would I get the strategy to also print out the analyzers ? I am using optstrategy as well as all the standard analyzers:

    cerebro.addanalyzer(btanalyzers.SharpeRatio, _name='mySharpeRatio', timeframe=bt.TimeFrame.Days, compression=1, riskfreerate=0.0)
    cerebro.addanalyzer(btanalyzers.DrawDown, _name='myDrawDown')
    cerebro.addanalyzer(btanalyzers.AnnualReturn, _name='myAnnualReturn')
    #cerebro.addanalyzer(btanalyzers.GrossLeverage, _name='myGrossLeverage')
    cerebro.addanalyzer(btanalyzers.TimeDrawDown, _name='myTimeDrawDown')
    

    I saw your blog post on strategy selection, but i have a different use case - 1 strategy, multiple parameters.

    I changed the code from

    cerebro.addstrategy(MyStrat,
        timeframe_0_tf = "1Min",
        timeframe_1_tf = "1H",
        timeframe_0_num_bars_in_sr = 3,
        timeframe_1_num_bars_in_sr = 3,
        csv_file="data.csv",
        sr_indicator = False,
        min_spread = 0.0010,
        proximity_to_sr = 0.2
        )
    

    to

    cerebro.optstrategy(MyStrat,
        timeframe_0_tf = "1Min",
        timeframe_1_tf = "1H",
        timeframe_0_num_bars_in_sr = 3, 
        timeframe_1_num_bars_in_sr = 3,
        csv_file="data.csv",
        sr_indicator = False,
        min_spread = list(np.arange(0.0010,0.0020,0.0005)), 
        proximity_to_sr = 0.2
        )
    

    I.e. one line min_spread went from float to list

    Nothing else changed. I run the code, and i get this error

    Traceback (most recent call last):
      File "C:\Python36\lib\threading.py", line 916, in _bootstrap_inner
        self.run()
      File "C:\Python36\lib\threading.py", line 864, in run
        self._target(*self._args, **self._kwargs)
      File "C:\Python36\lib\multiprocessing\pool.py", line 429, in _handle_results
        task = get()
      File "C:\Python36\lib\multiprocessing\connection.py", line 251, in recv
        return _ForkingPickler.loads(buf.getbuffer())
    AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_95681bf7d78147dca4d7f72d5ed03808' on <module 'backtrader.lineseries' from 'C:\\Python36\\lib\\site-packages\\backtrader\\lineseries.py'>

  • administrators

    @Taewoo-Kim said in create a CSV of the result of optstrategy:

    How would I get the strategy to also print out the analyzers ?

    Either with writer=True to cerebro or adding your own writer as pointed out by @Brad-Lloyd

    cerebro.addwriter(bt.WriterFile, csv = True, out='your_strategy_results')

    But in a multiprocess scenario this isn't sensible because the output will get interleaved and be most probably unusable.

    That's why the blog post mentioned by you uses a standard print after the strategies have run.

    Nothing else changed

    Compared to the post, there are several changes, including the use of multiple data feeds. This obviously shows that the DataTrades observer has a problem with being pickled


  • administrators

    In any case and to avoid createing the DataTrades observer (which may be considered pointelss, since plotting in an optimization makes no sense) you may run with cerebro.run(stdstats=False)


  • administrators

    And upon inspection of the case it is not easily solvable. The DataTrades observer (which is used when multiple data feeds are in place) generates its lines automatically when the number of data feeds is known (that is: when the execution of each strategy starts).

    Because of this, the attempt to place the dynamically generated classes at module level to enable pickling is not successful.



  • is it possible to access the analyzers from within the strategy?

    I have in main:

    cerebro.addanalyzer(btanalyzers.DrawDown, _name='myDrawDown')
    

    If i do (in strategy)

    def stop(self):
        print(self.analyzers.mySharpeRatio.get_analysis())
    

    it would just print a blank dict

    However, if I call it after cerebro runs via:

    thestrats = cerebro.run()
    
    print(thestrats[0].analyzers.mySharpeRatio.get_analysis())
    

    it seems to work.

    Work around?


  • administrators

    @Taewoo-Kim said in create a CSV of the result of optstrategy:

    is it possible to access the analyzers from within the strategy?

    The SharpeRatio is calculated after the strategy has stopped. It's not meant as a rolling analyzer (which calculates during the development of the strategy) That's the only reason for the difference.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.