How to add event information into strategy

[brettelliot]

Hi everyone,

I'm a total noob to BackTrader but first let me say, this platform is awesome! Thanks to everyone for making this available!

I do have a question, hopefully an easy one... I want to add event data to a backtest (and a live strategy for that matter) but I'm unsure how.

Let's start with backtesting....

I'd think like to add this data feed so that when next gets called the value shows up just like any other data (like Close or whatever).

First, is that how experienced backtraders would expect this new data to appear?

Second, this is what I did (and where I'm stuck).

I made a csv for the events that just appends an "Event" Column. I'm starting simple, with one event:

Date,Open,High,Low,Close,Adj Close,Volume,Event
2014-01-03 07:30:00,,,,,,,4

Then I made a new feed...

class GenericCSV_Event(GenericCSVData):

    # Add a 'event' line to the inherited ones from the base class
    lines = ('event',)

    # openinterest in GenericCSVData has index 7 ... add 1
    # add the parameter to the parameters inherited from the base class
    params = (('Event', 8),)

I created the feed and added it alongside a stock feed:

    cerebro.adddata(data)
    cerebro.adddata(event_data)

And, im stuck. I can't around a few errors like:

AttributeError: 'AutoInfoClass_LineRoot_LineMultiple_LineSeries_Dat' object has no attribute 'event'

I'm kinda confused about the capitalization of the "Event" in my csv file (which I copied from the orcl-2014.txt file) and the use of event in the GenericCSV_Event class.

Anyway, here is the whole example... perhaps someone can point me in the right direction?

Thanks!

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime  # For datetime objects
import os.path  # To manage paths
import sys  # To find out the script name (in argv[0])
from backtrader.feeds import GenericCSVData

# Import the backtrader platform
import backtrader as bt

class GenericCSV_Event(GenericCSVData):

    # Add a 'event' line to the inherited ones from the base class
    lines = ('event',)

    # openinterest in GenericCSVData has index 7 ... add 1
    # add the parameter to the parameters inherited from the base class
    params = (('event', 8),)


# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        event = self.datas[0].event[0]
        print('%s, %s' % (dt.isoformat(), txt))
        print("Event: {0}".format(event))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('Close, %.2f' % self.dataclose[0])


if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    # Datas are in a subfolder of the samples. Need to find where the script is
    # because it could have been called from anywhere
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    datapath = os.path.join(modpath, '../backtrader/datas/orcl-2014.txt')

    # Create a Data Feed
    data = bt.feeds.YahooFinanceCSVData(
        dataname=datapath,
        # Do not pass values before this date
        fromdate=datetime.datetime(2014, 1, 1),
        # Do not pass values before this date
        todate=datetime.datetime(2014, 1, 5),
        # Do not pass values after this date
        reverse=False)

    # now add the event data:
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    datapath = os.path.join(modpath, '../backtrader/datas/events.csv')
    # Create a Data Feed
    event_data = GenericCSV_Event(
        dataname=datapath,
        # Do not pass values before this date
        fromdate=datetime.datetime(2014, 1, 1),
        # Do not pass values before this date
        todate=datetime.datetime(2014, 1, 5),
        # Do not pass values after this date
        reverse=False)


    # Add the Data Feed to Cerebro
    cerebro.adddata(data)
    cerebro.adddata(event_data)

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

Also, I loved the line: "Blistering Barnacles!!! The system made money ... something must be wrong"

[brettelliot]

OK, so this morning, I tried again... I got it! Posting here for others.

BUT, I do have a question... I'm getting events for dates that don't appear in my event list. It's like the event data is being backfilled (ie: the last event data is being passed to the current bar). Is there a way to prevent that? I've tried backfill=False but that doesn't seem to work (possibly not a property of the CSV feed).

Here's the example... this event file has two events in the same week. But instead of just showing up in the bar of those two days, there's an event for every bar.

Thanks!

CSV File:

TIMESTAMP,EVENT
2014-01-07,4
2014-01-09,24

Output: (how do I prevent the events from showing up on the 3 days of the week without events?)

Starting Portfolio Value: 100000.00
2014-01-07, Close, 37.85
2014-01-07, Event, 4
2014-01-08, Close, 37.72
2014-01-08, Event, 4
2014-01-09, Close, 37.65
2014-01-09, Event, 24
2014-01-10, Close, 38.11
2014-01-10, Event, 24
Final Portfolio Value: 100000.00

Algo:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime  # For datetime objects
import os.path  # To manage paths
import sys  # To find out the script name (in argv[0])
from backtrader.feeds import GenericCSVData

# Import the backtrader platform
import backtrader as bt


class EventCSV(GenericCSVData):
    linesoverride = True  # discard usual OHLC structure
    # datetime must be present and last
    lines = ('event', 'datetime')
    #lines = ('datetime', 'event')
    # datetime (always 1st) and then the desired order for
    params = (
        ('dtformat', '%Y-%m-%d'),
        ('datetime', 0), # inherited from parent class
        ('event', 1),  # default field pos 1
    )

# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))


    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close
        self.event_data = self.datas[1]

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('Close, %.2f' % self.dataclose[0])
        self.log('Event, %d' % self.event_data[0])



if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    # Datas are in a subfolder of the samples. Need to find where the script is
    # because it could have been called from anywhere
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    datapath = os.path.join(modpath, '../backtrader/datas/orcl-2014.txt')

    # Create a Data Feed
    data = bt.feeds.YahooFinanceCSVData(
        dataname=datapath,
        # Do not pass values before this date
        fromdate=datetime.datetime(2014, 1, 6),
        # Do not pass values after this date
        todate=datetime.datetime(2014, 1, 11),
        reverse=False)

    # now add the event data:
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    datapath = os.path.join(modpath, '../backtrader/datas/events_02.csv')
    # Create a Data Feed
    event_data = EventCSV(
        dataname=datapath,
        # Do not pass values before this date
        fromdate=datetime.datetime(2014, 1, 6),
        # Do not pass values after this date
        todate=datetime.datetime(2014, 1, 11),
        backfill=False,
        reverse=False)


    # Add the Data Feed to Cerebro
    cerebro.adddata(data)
    cerebro.adddata(event_data, name='event_data')

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

Thanks!

[Laurent Michelizza]

Hi,

Another way of doing this could be to create an indicator events, load the csv in a panda df during the init and at each bar check whether or not the date is in the df. So the indicator would return true if the date is an event and false if not.

[brettelliot]

Yes!!! This is (potentially) great! It might even solve my problem of using CSVs for live trading. I'll attempt to create an indicator that reads the csv into a df in next() . This way it always grabs the latest file on disk.

Stay tuned...

Thanks!

[backtrader]

@brettelliot said in How to add event information into strategy:

this event file has two events in the same week. But instead of just showing up in the bar of those two days, there's an event for every bar.

This is because you have implemented your event as a data feed. And the platform presents you with the latest available data point. It's not actually showing twice (or three times). You are reading it twice without noticing that the date to which the data point refers has not changed.

The same holds true if you for example have the SP500 and the DAX. The trading calendars are not equal and you may get a new data point for the DAX, whilst the SP500 may present you the data point from the previous day (because the markets in the USA are closed for the 4th of July)

If you implement it as an Indicator as suggested by @Laurent-Michelizza (which has already been discussed several times) you can decide in the next method of the indicator if the data point of your event has to be presented or not (for example by checking if the date for the data on which the indicator is applied is the same as the one for the event)

[brettelliot]

Thank you @backtrader and @Laurent-Michelizza ! I've gotten this working with an indicator. Code posted below for posterity:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime  # For datetime objects
import os.path  # To manage paths
import sys  # To find out the script name (in argv[0])
import pandas as pd
import numpy as np

# Import the backtrader platform
import backtrader as bt

class EventIndicator(bt.Indicator):
    lines = ('event_line',)

    def next(self):

        df = pd.read_csv("../backtrader/datas/events_02.csv",
                                    parse_dates=True,
                                    index_col=0)

        dt = self.datas[0].datetime.date(0)
        try:
            event = df.loc[dt, 'EVENT']
            #print("{0}, Event: {1}".format(dt, event))
            self.lines.event_line[0] = event
        except KeyError:
            pass



# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print("{}: {}".format(dt.isoformat(), txt))


    def __init__(self):
        # Keep a reference to the event indicator
        self.event_indicator = EventIndicator(self.data)

    def next(self):
        # Simply log the whenever there was an event from the event indicator
        event = self.event_indicator[0]
        if not np.isnan(event):
            self.log("Event: {}".format(event))
        if event >= 10:
            self.buy()


if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    # Datas are in a subfolder of the samples. Need to find where the script is
    # because it could have been called from anywhere
    modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
    datapath = os.path.join(modpath, "../backtrader/datas/orcl-2014.txt")

    # Create a Data Feed
    data = bt.feeds.YahooFinanceCSVData(
        dataname=datapath,
        # Do not pass values before this date
        fromdate=datetime.datetime(2014, 1, 6),
        # Do not pass values after this date
        todate=datetime.datetime(2014, 1, 11),
        reverse=False)

    # Add the Data Feed to Cerebro
    cerebro.adddata(data)

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

[ab_trader]

I did approximately the same to consider stock splits.