Thanks, in the end that is pretty much what I did.
There's likely a much more elegant way to do this than how I have done it though

# define_indicators.py

def save_indicators():

  indicators = {

    'SMA': {'ind': bt.ind.SMA, 'data': 0, 'args': {'period':10}},
    'trend': {'ind': bt.ind.SMA, 'data': 3, 'args': {'period':10}},
    'rsi_1M': { 'ind': bt.ind.RSI, 'data': 0, 'args': None },
    'rsi_5M': { 'ind': bt.ind.RSI, 'data': 1, 'args': None},
    'rsi_30M': { 'ind': bt.ind.RSI, 'data': 2, 'args': None },
    'rsi_1H': { 'ind': bt.ind.RSI, 'data': 3, 'args': None },
    'rsi_4H': {'ind': bt.ind.RSI, 'data': 4, 'args': None},
    'sto_1M': {'ind': StochasticRSI, 'data': 'rsi_1M', 'args': None},
    .....
  }

  DATA_DIRNAME.mkdir(parents=True, exist_ok=True)
  with open(INDICATORS_FILENAME, 'wb') as f:
    pickle.dump(indicators, f, pickle.HIGHEST_PROTOCOL)


def build_indicators(datas, indicators):

    ind = dict()
    for k, v in indicators.items():

        kwargs = {}
        if v['args']:
            kwargs = v['args']

        if type(v['data']) is str:
            data = [ind[v['data']]]
        elif type(v['data']) is list:
            data = list()
            for d in v['data']:
                if type(d) is tuple:
                    data.append(getattr(ind[d[0]], d[1]))
                else:
                    data.append(ind[d])
        else:
            data = [datas[v['data']]]
        
        ind[k] = v['ind'](*data, plotname=k, **kwargs)
    
    return ind

if __name__ == '__main__':
  save_indicators()

# strategy.py
import backtrader as bt
from define_indicators import build_indicators

class St(bt.Strategy):

  def __init__(self):

        f = open(INDICATORS_FILENAME, 'rb')
        indicators = pickle.load(f)

        self.indicators = build_indicators(self.datas, indicators)

I am trying to figure out a way to save the configuration of all of my indicators in one strategy and then load the same configuration into a different strategy (there are more than 50 indicators and the indicators may change from time to time so it needs to be done programmatically).

For example, say in strategy1 I have an SMA and an RSI, I want to be able to load a file from disk that will tell strategy2 to create an SMA and an RSI with the same config (including plotname) as in Strategy1. Strategy 1 and Strategy 2 have different logic and both use different feeds so the rest of the cerebro instance is quite different between Strategy1 and Strategy2.

I had tried pickling but keep getting cannot serialize '_io.TextIOWrapper' object exceptions, which is also discussed in this thread. However pickling probably wouldn't work in any case because each indicator holds references to the cerebro context and those get pickled as well.

Is there a way to save a config, along with the indicator plotname etc that could then be used to instantiate an indentical copy in another strategy?

Nevermind, I had taken the idea of using the self._timeframe from the ccxt feed when instead I should have been using self.p.timeframe which doesn't get modified by the replaydata. Works perfectly now.

I have created a custom feed for coinbase data using their websocket interface and am trying to get the replaydata to work. The feed supports both historic and realtime data so if I want to replay ticks on minute bars for example I would do the following

# Start with 255 mins of historic data before going to LIVE
hist_start_date = datetime.utcnow() - timedelta(seconds=60*255)

# Get a tick data feed as the underlying data
data = CoinbaseProFeed(symbol="BTC-USD",
                       name="btc_usd_tick",
                       fromdate=hist_start_date,
                       timeframe=bt.TimeFrame.Ticks)

# replay the tick feed as minute bars
cerebro.replaydata(data,
                   timeframe=bt.TimeFrame.Minutes,
                   compression=1)

However, the feed only sees the timeframe of Minutes (self._timeframe) which is what the strategy would expect, but the feed needs to know that the underlying data is ticks because that's what cerebro is expecting in order to do the minute bar construction over the top.

I see that there is a self.replaying flag to check that the data is being replayed but I don't see anywhere to fetch the underlying timeframe from.

@blonc said in marketstore NEW data feed complete code & example:

I am integrating MS because it as well allows for real-time streaming via a websocket of the bars from the db. IMO this will allow for consistency between realtime and historical testing. Data is stored in tick compression and then using On-disk Aggregate updates the downsample data compressed into any timeframe you want. Easiest way to get this to work is to build MS from source in a docker, its already integrated with GDAX and works out the box.

It looks like you were originally trying to have MarketStore feed both historical and live data into backtrader. I have the MarketStore running with the GDAX feeder running, however the feeder is only using the historical data API to fetch the data even if the timeframe is 1Min, which means that its always a little behind real-time. And likewise the code above also uses the polling API to retrieve the data from the MarketStore.

Did you ever get it working with websocket data feeding in and a websocket connection from backtrader to the MarketStore?

@blonc said in marketstore NEW data feed complete code & example:

The next step for me is to integrate polygon.io which is REALTIME trade by trade SIP data for 200$ a month, latency is under 10ms too if you are colocated in or around ny4.

What additional latency are you getting on the GDAX connection (compared to a direct GDAX API connection) when streaming via the MarketStore websocket connector. e.g. By the time it comes down from the GDAX API, writes to disk in MarketStore, then gets re-streamed out, how far behind the real-time market data are you?

Do you think you could run this with real-time L1 or L2 data feed?

I noticed something strange with the ccxt fetchohlc() in that sometimes it returns candles up to the last closed candle and sometimes it returns the current live (not yet closed candle).

I used the follow code to get the last 20 candles which uses the same method that the ccxt backtrader uses to get live candle data.

df = pd.DataFrame(gdax.fetchOhlcv('BTC/USD', timeframe='1m', limit=20))
df['Date'] = pd.to_datetime(df[0], unit='ms')
df.set_index('Date', inplace=True)
df.drop(0, axis=1, inplace=True)
df.columns = ['open', 'high','low','close','volume']
df['highlow'] = df['high'] - df['low']; df

This outputs a dataframe:

                           
Date	                open    high    low     close   volume        highlow
2018-08-21 09:15:00	6452.99	6453.00	6452.99	6452.99	0.382287	0.01
2018-08-21 09:16:00	6453.00	6453.00	6452.99	6453.00	0.231581	0.01
2018-08-21 09:17:00	6453.00	6453.00	6453.00	6453.00	0.033058	0.00
2018-08-21 09:18:00	6453.00	6453.00	6452.99	6453.00	0.244911	0.01
2018-08-21 09:19:00	6453.00	6453.00	6452.99	6452.99	2.142046	0.01
2018-08-21 09:20:00	6453.00	6453.00	6438.91	6441.62	10.994071	14.09
2018-08-21 09:21:00	6441.62	6445.98	6441.62	6445.98	1.862376	4.36
2018-08-21 09:22:00	6445.97	6445.97	6431.64	6437.10	12.625005	14.33
2018-08-21 09:23:00	6437.10	6445.67	6437.10	6443.40	1.902326	8.57
2018-08-21 09:24:00	6443.39	6443.40	6443.39	6443.39	0.531487	0.01
2018-08-21 09:25:00	6443.40	6443.40	6435.62	6435.62	1.435376	7.78
2018-08-21 09:26:00	6436.35	6437.32	6433.71	6433.71	1.582333	3.61
2018-08-21 09:27:00	6435.23	6435.23	6435.22	6435.22	0.054494	0.01
2018-08-21 09:28:00	6435.22	6436.00	6435.22	6436.00	0.099945	0.78
2018-08-21 09:29:00	6435.99	6435.99	6433.71	6433.71	1.374433	2.28
2018-08-21 09:30:00	6433.71	6433.71	6431.00	6432.77	1.015340	2.71
2018-08-21 09:31:00	6433.62	6433.62	6432.76	6433.00	0.172707	0.86
2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24
2018-08-21 09:34:00	6431.63	6431.63	6431.63	6431.63	0.059508	0.00

When I ran this 4 times consecutively within less than 1 second I got the following results (only last 3 lines shown)

2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24
2018-08-21 09:34:00	6431.63	6431.63	6431.63	6431.63	0.059508	0.00

2018-08-21 09:31:00	6433.62	6433.62	6432.76	6433.00	0.172707	0.86
2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24

2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24
2018-08-21 09:34:00	6431.63	6431.63	6431.63	6431.63	0.062508	0.00

2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24
2018-08-21 09:34:00	6431.63	6431.63	6431.63	6431.63	0.059508	0.00

As you can see, sometimes it returns the 9:34 candle (which at the time had not closed) and sometimes it returns the 9:33 candle (which has closed) as the most recent.

When I ran this a few minutes later I get

2018-08-21 09:32:00	6432.99	6433.00	6432.99	6432.99	0.555686	0.01
2018-08-21 09:33:00	6432.99	6432.99	6427.75	6431.62	3.599003	5.24
2018-08-21 09:34:00	6431.63	6433.00	6431.62	6433.00	4.551569	1.38
2018-08-21 09:35:00	6433.38	6434.68	6433.38	6434.68	12.601649	1.30
2018-08-21 09:36:00	6436.36	6444.00	6436.36	6443.99	3.252023	7.64

As you can see the 9:34 candle closed at 6433.00 not at 6431.63.

This means that when you are getting 1m live data from backtrader you may actually be getting a live candle instead of the actual data because backtrader/ccxt will record the time of the latest candle and only give candles with timestamps later than the previous.

In fact this actually looks like what I'm getting

***** DATA NOTIF: LIVE
***** NEXT: 2018-08-21 09:04:00 btc_usd_min 6459.99 6460.0 6459.99 6460.0 0.02234232 Minute 3
***** NEXT: 2018-08-21 09:05:00 btc_usd_min 6455.06 6455.06 6455.06 6455.06 0.05325761 Minute 4
***** NEXT: 2018-08-21 09:06:00 btc_usd_min 6455.06 6460.92 6455.06 6460.92 0.75941057 Minute 5
***** NEXT: 2018-08-21 09:07:00 btc_usd_min 6460.95 6460.95 6460.95 6460.95 0.00362068 Minute 6
***** NEXT: 2018-08-21 09:08:00 btc_usd_min 6457.84 6457.84 6457.84 6457.84 0.03702458 Minute 7
***** NEXT: 2018-08-21 09:09:00 btc_usd_min 6452.82 6452.82 6452.82 6452.82 1.21548925 Minute 8
***** NEXT: 2018-08-21 09:10:00 btc_usd_min 6452.81 6452.81 6452.81 6452.81 0.0226 Minute 9

Minute 5 appears to be a full candle whereas Minute 7 (for example) appears to be a partially completed live candle

I've been using backtrader for sometime but I'm new to the ccxt live data. Thank you @Ed-Bartosh for all the work you've done on this. amazing!

When using live data I'm noticing quite a significant delay of the data by about 30 seconds or so. Is this expected behavior? My hope is to run this as close to real-time as possible.

Even though the data status shows LIVE, the first data is actually about 15mins behind real-time but starts to catch up after around 30 seconds, although it never actually fully catches up and ends up always being about 10-15 seconds behind the real-time data. I'm using TradingView and pro.coinbase.com as my benchmark as I'm using gdax as the feed.

***** DATA NOTIF: LIVE
***** NEXT: 2018-08-21 08:15:33.492003 btc_usd_tick 6475.01 6475.01 6475.01 6475.01 0.01 Tick 1
***** NEXT: 2018-08-21 08:15:33.532004 btc_usd_tick 6475.91 6475.91 6475.91 6475.91 0.57164401 Tick 2
***** NEXT: 2018-08-21 08:15:46.381995 btc_usd_tick 6475.91 6475.91 6475.91 6475.91 0.0016879 Tick 3
***** NEXT: 2018-08-21 08:15:53.060997 btc_usd_tick 6475.9 6475.9 6475.9 6475.9 0.0034 Tick 4
***** NEXT: 2018-08-21 08:16:04.493001 btc_usd_tick 6475.91 6475.91 6475.91 6475.91 0.0028 Tick 5
***** NEXT: 2018-08-21 08:16:16.126000 btc_usd_tick 6475.9 6475.9 6475.9 6475.9 0.0097 Tick 6
***** NEXT: 2018-08-21 08:16:28.048999 btc_usd_tick 6475.9 6475.9 6475.9 6475.9 0.1667 Tick 7
***** NEXT: 2018-08-21 08:16:45.003001 btc_usd_tick 6475.9 6475.9 6475.9 6475.9 1.0 Tick 8
***** NEXT: 2018-08-21 08:16:45.043002 btc_usd_tick 6474.6 6474.6 6474.6 6474.6 0.001 Tick 9
***** NEXT: 2018-08-21 08:16:49.953004 btc_usd_tick 6471.01 6471.01 6471.01 6471.01 0.00345867 Tick 10
***** NEXT: 2018-08-21 08:16:50.023995 btc_usd_tick 6471.01 6471.01 6471.01 6471.01 0.045419 Tick 11
***** NEXT: 2018-08-21 08:17:05.876002 btc_usd_tick 6471.01 6471.01 6471.01 6471.01 0.15756512 Tick 12
***** NEXT: 2018-08-21 08:17:10.691003 btc_usd_tick 6471.01 6471.01 6471.01 6471.01 0.0079974 Tick 13
***** NEXT: 2018-08-21 08:17:17.021002 btc_usd_tick 6471.01 6471.01 6471.01 6471.01 0.00315375 Tick 14
***** NEXT: 2018-08-21 08:17:21.066002 btc_usd_tick 6468.35 6468.35 6468.35 6468.35 0.00062 Tick 15
***** NEXT: 2018-08-21 08:17:21.703003 btc_usd_tick 6467.06 6467.06 6467.06 6467.06 0.01 Tick 16
***** NEXT: 2018-08-21 08:17:49.377999 btc_usd_tick 6465.19 6465.19 6465.19 6465.19 0.03187985 Tick 17
***** NEXT: 2018-08-21 08:18:01.357998 btc_usd_tick 6465.19 6465.19 6465.19 6465.19 0.0515 Tick 18
***** NEXT: 2018-08-21 08:18:04.042001 btc_usd_tick 6465.2 6465.2 6465.2 6465.2 0.00384762 Tick 19
***** NEXT: 2018-08-21 08:18:11.765002 btc_usd_tick 6465.19 6465.19 6465.19 6465.19 0.00383559 Tick 20
***** NEXT: 2018-08-21 08:18:11.801997 btc_usd_tick 6465.0 6465.0 6465.0 6465.0 4.99793959 Tick 21

class TestStrategy(bt.Strategy):
    def next(self):
        for data in self.datas:
            print('*' * 5, 'NEXT:', bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0],
                  data.low[0], data.close[0], data.volume[0],
                  bt.TimeFrame.getname(data._timeframe), len(data))

    def notify_order(self, order):
        print('*' * 5, "NOTIFY ORDER", order)
        
    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)

def runstrategy(argv):
    # Create a cerebro
    cerebro = bt.Cerebro()

    # Create broker
    broker_config = {'urls': {'api': 'https://api.pro.coinbase.com'},
                     'apiKey': '{}'.format(gdax.apiKey),
                     'secret': '{}'.format(gdax.secret),
                     'password': '{}'.format(gdax.password)
                    }

    # Create data feeds
    data_ticks = bt.feeds.CCXT(exchange='gdax', symbol='BTC/USD', name="btc_usd_tick",
                               timeframe=bt.TimeFrame.Ticks, compression=1)
    cerebro.adddata(data_ticks)

    # Add the strategy
    cerebro.addstrategy(TestStrategy)

    # Run the strategy
    cerebro.run()

Re: PyFolio analyzer multiple data sources not working.

I see the above topic was posted back in October 2017 but I'm wondering if this was resolved somewhere else. I'm also finding the exact same thing when using multiple data sources with the pyfolio analyzer. The source code line mentioned in the original post refers to

ps = [[k] + v[-2:] for k, v in iteritems(pss)]

where v contains the positions for each of the data feeds. [-2:] means that its only using the last two entries (the last symbol and the cash positions). If the line instead is changed to

ps = [[k] + v for k, v in iteritems(pss)]

Then positions for all datafeeds are saved and pyfolio gives the correct position and full tear sheets. Otherwise pyfolio thinks there was only one symbol in the portfolio.

A workaround of course is to add your own PositionValues analyzer

# existing pyfolio analyzer
cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio',
                        timeframe=bt.TimeFrame.Days)
pyfoliozer = strat.analyzers.getbyname('pyfolio',)
returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items() #<--positions only holds data for one symbol here

# Add an extra PositionsValue analyzer to capture positions for all symbols
from backtrader.utils.py3 import iteritems
cerebro.addanalyzer(bt.analyzers.PositionsValue, headers=True, cash=True, _name='mypositions')
p = strat.analyzers.getbyname('mypositions').get_analysis()
mypositions = [[k] + v for k, v in iteritems(p)] #<-- Now positions will hold data for all symbols
cols = mypositions.pop(0)  # headers are in the first entry
mypositions = pd.DataFrame.from_records(mypositions, index=cols[0], columns=cols)

pf.create_full_tear_sheet(
    returns, # from pyfolio analyzer
    positions=mypositions, #<-- from additional PositionsValue analyzer 
    transactions=transactions, # from pyfolio analyzer
    gross_lev=gross_lev, # from pyfolio analyzer
    round_trips=True)

I have a few hundred data feeds that Im running a portfolio strategy over. I would like to plot just the observer results and perhaps one of the data feeds as a reference but I haven't figured out how to turn off the data labels showing for the trades observer.

I am setting plotinfo.plot = False for all the data feeds except the first one which seems to work when I use just 6 of the data feeds but when I add all the data feeds I get the plot show below.

With all feeds

With just 6 feeds

#run
cerebro = bt.Cerebro()
cerebro.addstrategy(St)
cerebro.broker.setcash(20000.0)
cerebro.addobserver(bt.observers.TimeReturn, timeframe=bt.TimeFrame.NoTimeFrame)
cerebro.addobserver(bt.observers.Benchmark, data=benchmark, timeframe=bt.TimeFrame.NoTimeFrame)
cerebro.adddata(benchmark, name='SPY')
for feed in feeds:
    cerebro.adddata(feed[1], name=feed[0])
results = cerebro.run()
cerebro.plot(iplot=False, style='candle')

#strategy
for d in self.datas:
    d.plotinfo.plot = False
self.data1.plotinfo.plot = True