keep (custom) observer value in a list.

kav

I would like to keep the value of the observer in a list (rather than write it out to a csv and read the csv latter). Can this be done?

Here is my custom observer:

class PositionObserver(backtrader.observer.Observer):
    lines = ('position', 'price', )

    plotinfo = dict(plot=True, subplot=True, plotlinelabels=True)

    plotlines = dict(
        executed_position = dict(marker='*', markersize=8.0, color='lime', fillstyle='full'),
        executed_price    = dict(marker='*', markersize=8.0, color='lime', fillstyle='full')
    )

    def next(self):

        for order in self._owner._orderspending:

            if order.data is not self.data:
                continue

            self.lines.executed_position[0] = order.executed.size
            self.lines.executed_price[0]    = order.executed.price

backtrader

You probably need to elaborate a bit more

You have this

@kav said in keep (custom) observer value in a list.:

self.lines.executed_position[0] = order.executed.size

you can do this

self.executed_pos.append(order.executed.size)

So you probably mean something else.

kav

@backtrader : sorry, my question was poorly formulated.

What I meant is that after I run the code:

        cerebro   = backtrader.Cerebro()
        cerebro.addobserver(PositionObserver)
        # Add a strategy
        cerebro.addstrategy(StrategyBaseTrading)

        data = PandasDataStrat1(dataname     = dataframe, 
                                datetime     = -1, 
                                open         = -1, 
                                high         = -1, 
                                low          = -1, 
                                close        = -1,
                                volume       = -1,
                                openinterest = None)
        cerebro.adddata(data)
        cerebro.broker\
               .setcash(100000.0)
        cerebro.broker\
               .setcommission(commission = 2.0, 
                              margin     = 10000.0, 
                              mult       = 10.0)

is there a way to extract the content of lines.executed_position created by the Observer out of the cerebero object?

basically, right now, I do:

        cerebro.addwriter(backtrader.WriterFile, 
                          csv = True, 
                          out = 'results.csv')
        cerebro.run()

        def find_skipped_rows(file_name, string):
            sentinel = []
            counter  = 0
            with open(file_name) as fh:
                for line in fh:
                    counter += 1
                    if line.startswith(string):
                        sentinel.append(counter)
            return [i for i in range(counter) if i < sentinel[0] or i >= sentinel[1] - 1]

        skip_rows = find_skipped_rows(outfile, '===')
        executed_position  = pandas.read_csv(outfile, skiprows = skip_rows).executed_position

so basically I'm looking for a way to get the content of executed_position from the Cerebro object directly (without the write to csv, read from csv step)

thanks

backtrader

You can access the observers as attribute observers in the strategies returned by cerebro. And from there you have full access to the lines.

See: Docs - Cerebro and the section "Returning the Results"

kav

@backtrader Thanks for the pointer.. But I must be doing something wrong because the output I get don't seem to match the one from the csv file.

result[0].observers[2].lines.executed_size

In [44]: len(list(result[0].observers[2].lines.executed_size))
Out[44]: 3364

(this is the correct length: that of the data).

but I must be doing something wrong because these are all nan values:

In [39]: pandas.DataFrame(list(result[0].observers[2].lines.executed_size)).dropna()
Out[39]:
Empty DataFrame
Columns: [0]
Index: []

whereas when I export the result to a csv:

        cerebro.addwriter(backtrader.WriterFile, 
                          csv = True,
                          out = outfile)

I can clearly see the executed_size column is not all empty.

full code

class maCross(backtrader.Strategy):
    '''
    For an official backtrader blog on this topic please take a look at:

    https://www.backtrader.com/blog/posts/2017-04-09-multi-example/multi-example.html

    oneplot = Force all datas to plot on the same master.
    '''
    params = (
    ('sma1', 80),
    ('sma2', 200),
    ('oneplot', True),
    ('list_plus', [])
    )

    def __init__(self):
        '''
        Create an dictionary of indicators so that we can dynamically add the
        indicators to the strategy using a loop. This mean the strategy will
        work with any number of data feeds. 
        '''
        self.inds = dict()#           p0 = pandas.Series(self.data.close.get(size=5))
#           print(p0.head(5).transpose())
        for i, d in enumerate(self.datas):
            self.inds[d] = dict()
            self.inds[d]['sma1'] = backtrader.indicators.SMA(d.close, 
                                                             period=self.params.sma1)
            self.inds[d]['sma2'] = backtrader.indicators.SMA(d.close, 
                                                             period=self.params.sma2)
            if d._name in self.params.list_plus:
                self.inds[d]['cross'] = backtrader.indicators.CrossOver(self.inds[d]['sma1'],
                                                                        self.inds[d]['sma2'])
            else:
                self.inds[d]['cross'] = backtrader.indicators.CrossOver(self.inds[d]['sma2'],
                                                                        self.inds[d]['sma1']) 
          
            if i > 0: #Check we are not on the first loop of data feed:
                if self.p.oneplot == True:
                    d.plotinfo.plotmaster = self.datas[0]

    def next(self):
        for i, d in enumerate(self.datas):
            dt, dn = self.datetime.date(), d._name
            pos = self.getposition(d).size

            if not pos:  # no market / no orders
                if self.inds[d]['cross'][0] == 1:
                    self.buy(data=d, size=1)
                elif self.inds[d]['cross'][0] == -1:
                    self.sell(data=d, size=1)
            else:
                if self.inds[d]['cross'][0] == 1:
                    self.close(data=d)
                    self.buy(data=d, size=1)
                elif self.inds[d]['cross'][0] == -1:
                    self.close(data=d)
                    self.sell(data=d, size=1)

class SimpleObserver(backtrader.observer.Observer):
    lines = ('executed_size', )

    def next(self):

        for order in self._owner._orderspending:

            if order.data is not self.data:
                continue

            if order.status in [backtrader.Order.Completed]:
                self.lines.executed_size[0]  = order.executed.size

then

        dataframe = self.get_data(file_name)
        margin    = 10000
        # Create a cerebro entity
        cerebro   = backtrader.Cerebro()
        cerebro.addobserver(SimpleObserver)
        # Add a strategy
        cerebro.addstrategy(maCross)
        data      = PandasDataStrat1(dataname     = dataframe, 
                                     datetime     = -1, 
                                     open         = -1, 
                                     high         = -1, 
                                     low          = -1, 
                                     close        = -1,
                                     volume       = -1,
                                     openinterest = None)
        cerebro.adddata(data)
        start_cash = 100000.0
        cerebro.broker\
               .setcash(start_cash)
        cerebro.broker\
               .setcommission(commission = 2.0, 
                              margin     = margin, 
                              mult       = 10.0)
        result = cerebro.run()

backtrader

@kav said in keep (custom) observer value in a list.:

In [44]: len(list(result[0].observers[2].lines.executed_size))

If you have read the docs, you know that the lines don't follow the indexing convention in Python to make them fit the trading pattern. [0] is the actual time and [-1] is the last value delivered before the current point (i.e.: one instant in time before)

See: Docs - Platform Concepts and Slicing. The get method allows you to get the actual data. There are some other methods available for a line like getzero, plot or you can even access the underlying array attribute which holds the values.

kav

@backtrader: Perfect. Thanks for the pointer, it indeed addresses my problem. Just to be sure, could you confirm that the backtrader-preferred way to add an observer as a new column to the original dataframe (that was fed to backtrader) is:

dataframe['observer'] = result[0].observers[2].executed_size.get(size = dataframe.shape[0])

thank you