Programmatically extending a datafeed

tmorgan4

This issue has had me stumped for at 2+ weeks and I've decided to reach out for help.

A common question I see about backtrader is how to extend a datafeed to include additional columns in the input data file (e.g. indicators calculated outside of backtrader). This has been thoroughly documented and a good example is shown in pandas-data-optix.py which is partially shown below:

class PandasDataOptix(btfeeds.PandasData):

    lines = ('optix_close', 'optix_pess', 'optix_opt',)
    params = (('optix_close', -1),
              ('optix_pess', -1),
              ('optix_opt', -1))

    datafields = btfeeds.PandasData.datafields + (
        ['optix_close', 'optix_pess', 'optix_opt'])

What if we didn't know the names of the additional columns (or there were hundreds of columns that are unique to each data file) and want to automatically add every column not in the standard list (below)?

['datetime', 'open', 'high', 'low', 'close', 'volume', 'openinterest']

The heavy use of Metaclasses is beyond my knowledge level at the moment and using the debugger to follow how the PandasDataOptix class is created step-by-step is difficult.

The issue is that the lines, params, and datafields variables must be populated when the PandasDataOptix class is created. The MetaBase.__call__() method calls Lines.__init__() which cycles through the supplied lines and creates a LineBuffer for each. Since this all happens before PandasDataOptix.__init__() no arguments can be passed to the class (such as lines, params, and datafields).

I got it working by manually re-running several *._derive() methods to overwrite some of the PandasDataOptix class attributes after it was created but this is very hacky and surely not the best way:

PandasDataOptix.linealias = la = newcls.linealias._derive('la_' + name_cls, newlalias, oblalias)
PandasDataOptix.lines = newcls.lines._derive(name_cls, newlines, extralines, morebaseslines, linesoverride, lalias=la)
PandasDataOptix.plotlines = newcls.plotlines._derive('pl_' + name_cls, newplotlines, morebasesplotlines, recurse=True)

It seems that some of the steps which initialize the PandasData class could be manually called afterward to include additional arguments but so far I'm stumped. Thanks in advance.

backtrader

May have you gone too deep into the details?

datafields is actually a relic and should be removed at some point in time (or at least deprecated and not needed)

Extending the data feed in real-time should be possible without having to call the internal _derive methods.

thenewlines = ['newline1', 'newline2']

mydict = dict(
    lines=tuple(thenewlines),
    datafieds=bt.feeds.PandasData.datafields + thenewlines,
)

MyPandasClass = type('mypandasname', (bt.feeds.PandasData,), mydict)

This is 100% Python idiomatic and the usual way to create classes dynamically.

You can of course add a plotinfo and plotlines dictionaries (with the same syntax as in manual class declaration) to mydict and they will be taken into account.

tmorgan4

You are absolutely correct! Despite hours of searching for how to pass arguments to a class as it's created I never came across this alternative method for creating a class dynamically. It works perfectly.

You have taught me so much about Python just from snooping through backtrader code. Thank you Daniel!

In case anyone else finds this useful here is the data-pandas-optix.py file (backtrader/samples/data-pandas) with all parameters defined externally from the PandasData class. The original class creation is included but commented out.

#!/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-
###############################################################################
#
# Copyright (C) 2015, 2016 Daniel Rodriguez
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
###############################################################################
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import argparse
import backtrader as bt
import backtrader.feeds as btfeeds
import pandas


# class PandasDataOptix(btfeeds.PandasData):
#
#     lines = ('optix_close', 'optix_pess', 'optix_opt',)
#     params = (('optix_close', -1),
#               ('optix_pess', -1),
#               ('optix_opt', -1))
#
#     datafields = btfeeds.PandasData.datafields + (
#         ['optix_close', 'optix_pess', 'optix_opt'])


######## Dynamic class creation ##########

lines = ('optix_close', 'optix_pess', 'optix_opt',)
params = (('optix_close', -1),
          ('optix_pess', -1),
          ('optix_opt', -1))

datafields = btfeeds.PandasData.datafields + (
    ['optix_close', 'optix_pess', 'optix_opt'])

mydict = dict(
    lines=tuple(lines),
    params=params,
    datafields=bt.feeds.PandasData.datafields + list(lines),
    )

PandasDataOptix = type('PandasDataOptix', (btfeeds.PandasData,), mydict)


class StrategyOptix(bt.Strategy):

    def next(self):
        print('%03d %f %f, %f' % (
            len(self),
            self.data.optix_close[0],
            self.data.lines.optix_pess[0],
            self.data.optix_opt[0],))


def runstrat():
    args = parse_args()

    # Create a cerebro entity
    cerebro = bt.Cerebro(stdstats=False)

    # Add a strategy
    cerebro.addstrategy(StrategyOptix)

    # Get a pandas dataframe
    datapath = ('../../datas/2006-day-001-optix.txt')

    # Simulate the header row isn't there if noheaders requested
    skiprows = 1 if args.noheaders else 0
    header = None if args.noheaders else 0

    dataframe = pandas.read_csv(datapath,
                                skiprows=skiprows,
                                header=header,
                                parse_dates=True,
                                index_col=0)

    if not args.noprint:
        print('--------------------------------------------------')
        print(dataframe)
        print('--------------------------------------------------')

    # Pass it to the backtrader datafeed and add it to the cerebro
    data = PandasDataOptix(dataname=dataframe)

    cerebro.adddata(data)

    # Run over everything
    cerebro.run()

    # Plot the result
    if not args.noplot:
        cerebro.plot(style='bar')


def parse_args():
    parser = argparse.ArgumentParser(
        description='Pandas test script')

    parser.add_argument('--noheaders', action='store_true', default=False,
                        required=False,
                        help='Do not use header rows')

    parser.add_argument('--noprint', action='store_true', default=False,
                        help='Print the dataframe')

    parser.add_argument('--noplot', action='store_true', default=False,
                        help='Do not plot the chart')

    return parser.parse_args()


if __name__ == '__main__':
    runstrat()

LAbrk

Hi all.
Im new on BT, im researching the code to know if is compatible with mi live data. I'm seeing that bt is an incredible tool!

@backtrader my doubt, is this example valid for live datafeed?
Or do I have to add the indicators in another way?

Thanks a lot!

backtrader

The extension mechanism is generic. How to fill the values is what's data feed specific.

backtrader

@backtrader said in Programmatically extending a datafeed:

datafields is actually a relic and should be removed at some point in time (or at least deprecated and not needed)

The need to use datafields for the PandasData was already removed some versions ago

LAbrk

@backtrader
Great. Thank you for the reply.
Good to have an active community.
I'm still trying.

btr

@backtrader @tmorgan4 Hey, I am trying to understand this and the question is about programatically extending the data feed when you do not know the additional column names of df. After the discussion, @tmorgan4 posted a code where ('optix_close', 'optix_pess', 'optix_opt',) lines are already known. How is this created programatically? Am I missing something?

thanks you in advance for the response

run-out

@btr What part do you feel like you are missing?

btr

@run-out I needed a break and coffee. It was silly

run-out

@btr said in Programmatically extending a datafeed:

I needed a break and coffee. It was silly

Hilarious! Hope you had a nice break!