@backtrader Awesome. Thank you for the suggestions. I will give them a try.

Hello All,
Just wondering if anyone has experience with extending a datafeed and plotting the newly extended lines on a subplot (like an indicator)

I have some COT data coming in on the weekly and would like to plot the commercial / noncommmercial lines under the daily price chart.

I have played around trying to add an indicator that simply assigns the extended datafeed lines to the lines of the indicator. However, I am running into a couple of speed bumps...

• Because my COT data has no OHLCV values, I think I need turn plotting of the data feed off using plotinfo.plot so I avoid getting the error: ValueError Axis limits cannot be NaN or Inf.
• My guess (could be wrong) is that this now means any indicator assigned to that datafeed will not plot (because I turned plotting off - That would seem logical!).

My current thinking is to merge the weekly COT data with some dummy OHLCV data or resampled daily data so there are some values to plot and create subplots from. However, it won't look so nice on the final chart and I am wondering if anyone can think of a better way?

Or perhaps I am extending the data feed incorrectly? Here is my extended code. It works with a pandas dataframe downloaded directly from Quandl

class COT_Feed(bt.feeds.PandasData):

    # Add a 'pe' line to the inherited ones from the base class
    lines = (
        'ncl',  # Noncommercial long
        'ncs',  # Noncommercial Short
        'cl',   # Commercial Long
        'cs',   # Commercial Short
        )

    params = (
        ('open', -1),
        ('high', -1),
        ('low', -1),
        ('close', -1),
        ('volume', -1),
        ('openinterest', 0),
        ('ncl', 1),
        ('ncs', 2),
        ('cl', 4),
        ('cs', 5)
        )

The feed works in a simple strategy where I can access the values from the feed using d.ncl[0]... It is only the plotting part I am having some trouble with.

Maybe I am over thinking this and there is a simple solution?

I noticed that the when using multiple data feeds, the buy and sell markers can appear on different dates even though notify_order() reports them both being filled on the same date.

Not sure if this is a misunderstanding on my part or not....

Anyway, I created an example script to reproduce the phenomenon. The script will attempt to use order_target_percent() at fixed dates. As such, I would expect them to be filled on the same date (which they appear to be!)

import backtrader as bt
from datetime import datetime

class TestStrategy(bt.Strategy):

    def __init__(self):
        pass

    def next(self):

        for i, d in enumerate(self.datas):
            dt = d.datetime.datetime().date()
            dn = d._name
            pos = self.getposition(d).size
            #print(dt,type(dt))
            if dt == datetime(2017,6,1).date():
                print('{} Sending Order: {} | Current Pos: {}'.format(dt,dn,pos))

                if dn == 'AAPL':
                    self.order_target_percent(d, target=0.3)
                else:
                    self.order_target_percent(d, target=0.7)

            if dt == datetime(2017,9,1).date():
                print('{} Sending Order: {} | Current Pos: {}'.format(dt,dn,pos))
                if dn == 'AAPL':
                    self.order_target_percent(d, target=0.1)
                else:
                    self.order_target_percent(d, target=0.1)


    def notify_order(self, order):
        date = self.data.datetime.datetime().date()

        if order.status == order.Completed:
            print('{} >> Order Completed >> Stock: {},  Ref: {}, Size: {}, Price: {}'.format(

                                                        date,
                                                        order.data._name,
                                                        order.ref,
                                                        order.size,
                                                        'NA' if not order.price else round(order.price,5)
                                                        ))


    def notify_trade(self, trade):
        date = self.data.datetime.datetime().date()
        if trade.isclosed:
            print('{} >> Notify Trade >> Stock: {}, Close Price: {}, Profit, Gross {}, Net {}'.format(
                                                date,
                                                trade.data._name,
                                                trade.price,
                                                round(trade.pnl,2),
                                                round(trade.pnlcomm,2)))

startcash = 10000

#Create an instance of cerebro
cerebro = bt.Cerebro()

# data lists
data = bt.feeds.Quandl(
    dataname='AAPL',
    fromdate = datetime(2017,1,1),
    todate = datetime(2018,1,1),
    buffered= True
    )

data2 = bt.feeds.Quandl(
    dataname='F',
    fromdate = datetime(2017,1,1),
    todate = datetime(2018,1,1),
    buffered= True
    )

data_list = [(data, 'AAPL'), (data2, 'F')]

#Add our strategy
cerebro.addstrategy(TestStrategy)

# Create a Data Feed
for asset in data_list:

    # Add the data
    cerebro.adddata(
                asset[0],
                name=asset[1])


# Set our desired cash start
cerebro.broker.setcash(startcash)
cerebro.broker.set_checksubmit(False)

# Run over everything
cerebro.run()

cerebro.plot(style='candlestick')

This will then result in an output and chart that looks like this:

Am I being a dumbo and missing something obvious?

@sirhc

Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?

Yes - That is correct. The reason it is slow is because the broker will try to check your cash/balance on every call of next(). (If I remember correctly it makes 2 or 3 checks in total between each call of next). Since, the store will make use rest API calls to do these checks, this can add quite a few seconds delay between each call of next(). (Because of the rest API rate limit).

So in my fork, I cheat a bit. The cash and balance checks return the last values received from the last rest call. They do not make another call. Instead, the user must manually check the balance if they need up to date balance information before entering a position. (at which point the latest values will be saved).

Not an ideal solution but it works for me.

@backtrader Awesome! Thanks, I will take a look

@vandan-chopra

You can load multiple strategies into cerebro as far as I am aware. However, perhaps you just need to rethink your planned architecture?

If the "mothership" will be finally responsible for buying and selling (creating the portfolio), then it sounds like you just need to generate unique signals on specific data feeds (assets).

If that is the case, you can create indicators that generate signals according to your buy and sell principles. Then just allocate those indicators to the specific asset classes you are interested in. Finally, in next(), you can look at all of the signals and decide how to build your portfolio. No need for multiple instances.

Just a suggestion.

@chuka

You can just setup a live algorithm and backfill from a file or even the live source if they have enough data for training available.

Hello All,
I have run into some peculiar results when trying to optimise a strategy that uses multiple data feeds. The results I am getting also seem to be effected by maxcpus. The key word here is "seems". I am not really sure how they are interrelated (if at all).

I have put together a little demonstration code to see if others can reproduce it. However, first let me describe what I am seeing.

1. If I run a simple script with a single data feed, I can optimise without issue.
2. If I add a second data feed, my strategy exits with an AttributeError
3. If I reduce the maxcpus to 1, essentially disabling multicore support, then the optimisation runs but all of the final results are the same.

The test script is below. To drive it, we just need to change the following 4 lines

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 1
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

And the script:

import backtrader as bt
from datetime import datetime


class singleTestStrategy(bt.Strategy):
    params = (
        ('period',21),
        )

    def __init__(self):
        self.startcash = self.broker.getvalue()
        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period)

    def next(self):
        if not self.position:
            if self.rsi < 30:
                self.buy(size=100)
        else:
            if self.rsi > 70:
                self.sell(size=100)



class multTestStrategy(bt.Strategy):
    params = (
        ('period',21),
        )

    def __init__(self):
        self.startcash = self.broker.getvalue()
        self.data_dict = dict()

        for i, d in enumerate(self.datas):
            self.data_dict[d] = dict()
            self.data_dict[d]['rsi'] = bt.indicators.RSI_SMA(d, period=self.params.period)

    def next(self):

        for i, d in enumerate(self.datas):
            pos = self.broker.getposition(d).size
            if not pos:
                if self.data_dict[d]['rsi'] < 30:
                    self.buy(d,size=10)
            else:
                if self.data_dict[d]['rsi'] > 70:
                    self.sell(d,size=10)


if __name__ == '__main__':
    #Variable for our starting cash
    startcash = 10000

    # ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = singleTestStrategy
    second_data = False
    # -------------------------------------------------------------------------/

    #Create an instance of cerebro
    cerebro = bt.Cerebro(optreturn=False, maxcpus=multicore)

    #Add our strategy
    if optimize:
        cerebro.optstrategy(testStrategy, period=[14,21,28])
    else:
        cerebro.addstrategy(testStrategy, period=21)


    data = bt.feeds.Quandl(
        dataname='AAPL',
        fromdate = datetime(2016,1,1),
        todate = datetime(2017,1,1),
        buffered= True
        )

    data2 = bt.feeds.Quandl(
        dataname='AMZN',
        fromdate = datetime(2016,1,1),
        todate = datetime(2017,1,1),
        buffered= True
        )

    #Add the data to Cerebro
    cerebro.adddata(data)

    #Add the data to Cerebro
    if second_data:
        cerebro.adddata(data2)

    # Set our desired cash start
    cerebro.broker.setcash(startcash)

    # Run over everything
    if optimize:
        opt_runs = cerebro.run()

        # Generate results list
        final_results_list = []
        for run in opt_runs:
            for strategy in run:
                value = round(strategy.broker.get_value(),2)
                PnL = round(value - startcash,2)
                period = strategy.params.period
                final_results_list.append([period,PnL])

        #Sort Results List
        by_period = sorted(final_results_list, key=lambda x: x[0])
        by_PnL = sorted(final_results_list, key=lambda x: x[1], reverse=True)

        #Print results
        print('Results: Ordered by period:')
        for result in by_period:
            print('Period: {}, PnL: {}'.format(result[0], result[1]))
        print('Results: Ordered by Profit:')
        for result in by_PnL:
            print('Period: {}, PnL: {}'.format(result[0], result[1]))
    else:
        # Run over everything
        cerebro.run()

        #Get final portfolio Value
        portvalue = cerebro.broker.getvalue()
        pnl = portvalue - startcash

        #Print out the final result
        print('Final Portfolio Value: ${}'.format(portvalue))
        print('P/L: ${}'.format(pnl))

        #Finally plot the end results
        cerebro.plot(style='candlestick')

Test Case 1 - Single Data with Optimization

This is the baseline - Working example

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = singleTestStrategy
    second_data = False
# -------------------------------------------------------------------------/

Output:

Results: Ordered by period:
Period: 14, PnL: 2153.0
Period: 21, PnL: 1710.0
Period: 28, PnL: 1327.0
Results: Ordered by Profit:
Period: 14, PnL: 2153.0
Period: 21, PnL: 1710.0
Period: 28, PnL: 1327.0

Test Case 2 - Multi-Data / Multicore with Optimization

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 463, in _handle_results
    task = get()
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_aa890439e0cc4922a6091c60d2c6e113' on <module 'backtrader.lineseries' from 'C:\\Users\\Dave\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\backtrader\\lineseries.py'>

Test Case 3 - Multi-Data / Multicore WITHOUT Optimization

Again, just a positive working test case showing that things are working on the multTestStrategy

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = False
    multicore = 0
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Final Portfolio Value: $10046.32851701423
P/L: $46.328517014229874

Test Case 4 - Multi-Data / Single Core with Optimization

The final test case is also quite perculiar. Limiting maxcpus to 1. I get the correct period from the strategyreturned by cerebrobut the get_value() broker calls all seem to return the same figure.

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 1
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Results: Ordered by period:
Period: 14, PnL: 333.98
Period: 21, PnL: 333.98
Period: 28, PnL: 333.98
Results: Ordered by Profit:
Period: 14, PnL: 333.98
Period: 21, PnL: 333.98
Period: 28, PnL: 333.98

As we can see the period is changing but the PnL remains the same.

So overall, I am not sure what is causing it but I am also not sure how where to begin with debugging!

@totoro

Regarding the partial candle data, even though the PR wasn't merged, you can pick up fork of the Ed's and the others awesome work here:

https://github.com/Dave-Vallance/bt-ccxt-store

The repo contains made a few changes I made including an attempt to address the partial candle issue (although the solution is different to the PR). These changes work for me on the few exchanges I am looking at but your mileage may vary and I don't really plan on supporting it :)

There are some other changes such as speeding up backfilling and allowing you to map some broker keys and strings if the defaults don't work amongst other things. (for example, since CCXT does not have a unified stop/limit order implementation, Kraken expects you to create a stoploss with 'stop-loss', whereas bitmex expects just 'stop'.

For a full list of changes you can checkout the readme.

@backtrader

Sure, I appreciate that thanks.

So in the end I decided to do something a bit hacky and add a attribute to the CCXT broker that can be overwritten from inside the strategy. I.e When I set it to false, the broker will just return the last cash and value figures that were downloaded. If it is true the broker will make the proper store method call.

Probably not the best idea but needs must :) I can now warm up the indicators without waiting a lifetime.