btoandav20 - Accessing indicator values in strategy

cdpy

I'm hoping I might get some help here despite btoandav20 not being a part of backtrader core.

I have developed a custom Indicator to detect swing highs and this is working well with GenericCSVData downloaded from the Oanda v20 API. I can access values stored in the Indicator lines in my strategy like so:

    def next(self):
        print(f'len(self) is {len(self)}, prior high is {self.swing_highs.prior_high[0]}')

This works & prints out the current bar and the current value of the prior_high line e.g.

...
len(self) is 5024, prior high is 1.18408
len(self) is 5025, prior high is 1.18408
len(self) is 5026, prior high is 1.18408
len(self) is 5027, prior high is 1.18704
len(self) is 5028, prior high is 1.18704
len(self) is 5029, prior high is 1.18704
len(self) is 5030, prior high is 1.18704
len(self) is 5031, prior high is 1.18704
len(self) is 5032, prior high is 1.18704
len(self) is 5033, prior high is 1.18704
len(self) is 5034, prior high is 1.18808
len(self) is 5035, prior high is 1.18808
len(self) is 5036, prior high is 1.18808
...

I am now attempting to update my backtest to use btoandav20 for data instead of downloading it from the API & saving as a csv file. However by doing so, I have found that I am unable to access the values stored in my Indicator lines like before, now all that is printed is nan:

...

len(self) is 5024, prior high is nan
len(self) is 5025, prior high is nan
len(self) is 5026, prior high is nan
len(self) is 5027, prior high is nan
len(self) is 5028, prior high is nan
len(self) is 5029, prior high is nan
len(self) is 5030, prior high is nan
len(self) is 5031, prior high is nan
len(self) is 5032, prior high is nan
len(self) is 5033, prior high is nan
len(self) is 5034, prior high is nan
len(self) is 5035, prior high is nan
len(self) is 5036, prior high is nan
...

See below for the full backtest script, note that nothing has changed in the Strategy or Indicator, I am simply switching between what data feed is loaded by adding either data or data0:

import datetime as dt
import backtrader as bt
import backtrader.feeds as btfeeds
from GenericCSVDataForex import GenericCSVDataForex
import btoandav20
import exampleauth as auth
from test_strategy import TestStrategy

StoreCls = btoandav20.stores.OandaV20Store
DataCls = btoandav20.feeds.OandaV20Data

account_id, token = auth.example_auth()

data = GenericCSVDataForex(
    dataname = 'EUR_USD_H1_2020-01-01.csv',
    timeframe=bt.TimeFrame.Minutes,
    compression=60,

    fromdate=dt.datetime(2010, 1, 1),
    todate=dt.datetime.today(),

    dtformat=('%Y-%m-%dT%H:%M:%S.000000000Z'),

    datetime=1,
    open=2,
    high=3,
    low=4,
    close=5,
    volume=6,
    cc=7,
    openinterest=-1
)

if __name__ == '__main__':
    cerebro = bt.Cerebro(stdstats=True)
    cerebro.broker.set_cash(2000)

    storekwargs = dict(
        token=token,
        account=account_id,
        practice=True)

    store = StoreCls(**storekwargs)

    DataFactory = store.getdata

    datakwargs = dict(
        dataname='EUR_USD',
        timeframe=bt.TimeFrame.Minutes, compression=60,
        qcheck=0.5,
        historical=True,
        fromdate=dt.datetime(2020, 1, 1),
        todate=dt.datetime.today(),
        bidask=False,
        useask=False,
        backfill_start=True,
        backfill=True,
        tz='Europe/London'
    )

    data0 = DataFactory(**datakwargs)

    cerebro.adddata(data0)

    cerebro.addstrategy(TestStrategy)

    cerebro.run(exactbars=False, tradehistory=True)

    cerebro.plot(style='candlestick')

The plotting works and looks exactly the same for both data feeds, so I can see that the indicator values are still being calculated correctly, I just can't seem to access them in the Strategy when using the btoandav20 data feed.

Any help would be greatly appreciated!

dasch

try add resampledata with desired timeframe and compression when adding data feed when using live data. when using live data, btoandav20 will use ticks instead of desired timeframe.

cdpy

@dasch Unfortunately resampledata doesn't seem to make any difference, modified the code to the below & the result is the same, indicator values are all nan, plotting is still fine. As I am using the Oanda practice server I don't think it should be getting live data?

I also tried with the btoandav20 data feed instead of store, with the candles parameter set to True, which per the source code should wait for candles rather than streaming tick data, unfortunately the result is still the same.

Updated script with resampledata:

if __name__ == '__main__':
    cerebro = bt.Cerebro(stdstats=True)
    cerebro.broker.set_cash(2000)

    storekwargs = dict(
        token=token,
        account=account_id,
        practice=True)

    store = StoreCls(**storekwargs)

    no_store = False

    DataFactory = DataCls if no_store else store.getdata

    datakwargs = dict(
        dataname='EUR_USD',
        timeframe=bt.TimeFrame.Ticks, compression=1,
        qcheck=0.5,
        historical=True,
        fromdate=dt.datetime(2020, 1, 1),
        todate=dt.datetime.today(),
        bidask=False,
        useask=False,
        backfill_start=True,
        backfill=True,
        tz='Europe/London'
    )

    data0 = DataFactory(**datakwargs)

    cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)

    # cerebro.adddata(data0)

    cerebro.addstrategy(TestStrategy)

    cerebro.run(exactbars=False, tradehistory=True)

    cerebro.plot(style='candlestick')

dasch

@cdpy not really sure what self.swing_highs is doing, but if it plots, the data is there. Check your strategy for errors or post the strategy code to see where the issue is.

cdpy

@dasch The Strategy is just adding the indicators and attempting to print the values of the indicators during next():

import backtrader as bt
from swing_high_indicator import SwingHighInd
from swing_low_indicator import SwingLowInd
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.dataclose = self.datas[0].close
        self.swing_highs = SwingHighInd(self.datas[0])
        self.swing_lows = SwingLowInd(self.datas[0])
        self.order = None
        self.buy_order = None
        self.sell_order = None

    def notify_order(self, order):

        if not order.status == order.Completed:
            return  # discard any other notification

        if not self.position:  # we left the market
            print('SELL@price: {:.2f}'.format(order.executed.price))
            return

        # We have entered the market
        print('BUY @price: {:.2f}'.format(order.executed.price))
        
    def next(self):
        print(f'len(self) is {len(self)}, prior high is {self.swing_highs.prior_high[0]}')
        
    def stop(self):
        print(self.data0._name)

Here is the indicator code for swing_highs:

import backtrader as bt

class SwingHighInd(bt.Indicator):
    '''
    A Simple swing high indicator that measures swing highs (the highest value)
    within a given time period.
    '''
    lines = ('local_high', 'prior_high')
    params = (('period',4),)

    plotinfo = dict(
        subplot=False,
        plotlinelabels=True
    )

    plotlines = dict(
        local_high=dict(marker='v', markersize=5.0, color='green', fillstyle='full', ls=''),
        prior_high=dict(marker='_', markersize=8.0, color='blue', fillstyle='full', ls=''),
    )
 
    def __init__(self):
        #Set the swing range - The number of bars before and after the swing
        #needed to identify a swing
        self.swing_range = (self.p.period * 2) + 1
        self.addminperiod(self.swing_range)
 
    def next(self):

        highs = self.data.high.get(size=self.swing_range)
        lows = self.data.low.get(size=self.swing_range)

        middle_high = highs.pop(self.p.period)
        middle_low = lows.pop(self.p.period)

        if middle_high > max(highs):
            self.lines.local_high[-self.p.period] = middle_high
            self.lines.prior_high[-self.p.period + 1] = middle_high
        else:
            self.lines.prior_high[-self.p.period + 1] = self.lines.prior_high[-self.p.period]

dasch

@cdpy you are setting the values in your indicator for -self.p.period and -self.p.period + 1, so if you try to access the last value, it will always be nan, since it was not set yet.

if middle_high > max(highs):
            self.lines.local_high[-self.p.period] = middle_high
            self.lines.prior_high[-self.p.period + 1] = middle_high
        else:
            self.lines.prior_high[-self.p.period + 1] = self.lines.prior_high[-self.p.period]

cdpy

@dasch :facepalm: Thank you so much! It's so obvious now, the only reason it was working before with non-live data was because preload parameter defaults to true with non-liva data. If I manually set preload as false when using GenericCSVData, the same behaviour occurs & the current value of the indicator is alway nan.

Updated my strategy so it now looks at the indicator values in the past, it is working as intended.