@sirhc

Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?

Yes - That is correct. The reason it is slow is because the broker will try to check your cash/balance on every call of next(). (If I remember correctly it makes 2 or 3 checks in total between each call of next). Since, the store will make use rest API calls to do these checks, this can add quite a few seconds delay between each call of next(). (Because of the rest API rate limit).

So in my fork, I cheat a bit. The cash and balance checks return the last values received from the last rest call. They do not make another call. Instead, the user must manually check the balance if they need up to date balance information before entering a position. (at which point the latest values will be saved).

Not an ideal solution but it works for me.

@backtrader Awesome! Thanks, I will take a look

@vandan-chopra

You can load multiple strategies into cerebro as far as I am aware. However, perhaps you just need to rethink your planned architecture?

If the "mothership" will be finally responsible for buying and selling (creating the portfolio), then it sounds like you just need to generate unique signals on specific data feeds (assets).

If that is the case, you can create indicators that generate signals according to your buy and sell principles. Then just allocate those indicators to the specific asset classes you are interested in. Finally, in next(), you can look at all of the signals and decide how to build your portfolio. No need for multiple instances.

Just a suggestion.

@chuka

You can just setup a live algorithm and backfill from a file or even the live source if they have enough data for training available.

Hello All,
I have run into some peculiar results when trying to optimise a strategy that uses multiple data feeds. The results I am getting also seem to be effected by maxcpus. The key word here is "seems". I am not really sure how they are interrelated (if at all).

I have put together a little demonstration code to see if others can reproduce it. However, first let me describe what I am seeing.

1. If I run a simple script with a single data feed, I can optimise without issue.
2. If I add a second data feed, my strategy exits with an AttributeError
3. If I reduce the maxcpus to 1, essentially disabling multicore support, then the optimisation runs but all of the final results are the same.

The test script is below. To drive it, we just need to change the following 4 lines

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 1
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

And the script:

import backtrader as bt
from datetime import datetime


class singleTestStrategy(bt.Strategy):
    params = (
        ('period',21),
        )

    def __init__(self):
        self.startcash = self.broker.getvalue()
        self.rsi = bt.indicators.RSI_SMA(self.data.close, period=self.params.period)

    def next(self):
        if not self.position:
            if self.rsi < 30:
                self.buy(size=100)
        else:
            if self.rsi > 70:
                self.sell(size=100)



class multTestStrategy(bt.Strategy):
    params = (
        ('period',21),
        )

    def __init__(self):
        self.startcash = self.broker.getvalue()
        self.data_dict = dict()

        for i, d in enumerate(self.datas):
            self.data_dict[d] = dict()
            self.data_dict[d]['rsi'] = bt.indicators.RSI_SMA(d, period=self.params.period)

    def next(self):

        for i, d in enumerate(self.datas):
            pos = self.broker.getposition(d).size
            if not pos:
                if self.data_dict[d]['rsi'] < 30:
                    self.buy(d,size=10)
            else:
                if self.data_dict[d]['rsi'] > 70:
                    self.sell(d,size=10)


if __name__ == '__main__':
    #Variable for our starting cash
    startcash = 10000

    # ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = singleTestStrategy
    second_data = False
    # -------------------------------------------------------------------------/

    #Create an instance of cerebro
    cerebro = bt.Cerebro(optreturn=False, maxcpus=multicore)

    #Add our strategy
    if optimize:
        cerebro.optstrategy(testStrategy, period=[14,21,28])
    else:
        cerebro.addstrategy(testStrategy, period=21)


    data = bt.feeds.Quandl(
        dataname='AAPL',
        fromdate = datetime(2016,1,1),
        todate = datetime(2017,1,1),
        buffered= True
        )

    data2 = bt.feeds.Quandl(
        dataname='AMZN',
        fromdate = datetime(2016,1,1),
        todate = datetime(2017,1,1),
        buffered= True
        )

    #Add the data to Cerebro
    cerebro.adddata(data)

    #Add the data to Cerebro
    if second_data:
        cerebro.adddata(data2)

    # Set our desired cash start
    cerebro.broker.setcash(startcash)

    # Run over everything
    if optimize:
        opt_runs = cerebro.run()

        # Generate results list
        final_results_list = []
        for run in opt_runs:
            for strategy in run:
                value = round(strategy.broker.get_value(),2)
                PnL = round(value - startcash,2)
                period = strategy.params.period
                final_results_list.append([period,PnL])

        #Sort Results List
        by_period = sorted(final_results_list, key=lambda x: x[0])
        by_PnL = sorted(final_results_list, key=lambda x: x[1], reverse=True)

        #Print results
        print('Results: Ordered by period:')
        for result in by_period:
            print('Period: {}, PnL: {}'.format(result[0], result[1]))
        print('Results: Ordered by Profit:')
        for result in by_PnL:
            print('Period: {}, PnL: {}'.format(result[0], result[1]))
    else:
        # Run over everything
        cerebro.run()

        #Get final portfolio Value
        portvalue = cerebro.broker.getvalue()
        pnl = portvalue - startcash

        #Print out the final result
        print('Final Portfolio Value: ${}'.format(portvalue))
        print('P/L: ${}'.format(pnl))

        #Finally plot the end results
        cerebro.plot(style='candlestick')

Test Case 1 - Single Data with Optimization

This is the baseline - Working example

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = singleTestStrategy
    second_data = False
# -------------------------------------------------------------------------/

Output:

Results: Ordered by period:
Period: 14, PnL: 2153.0
Period: 21, PnL: 1710.0
Period: 28, PnL: 1327.0
Results: Ordered by Profit:
Period: 14, PnL: 2153.0
Period: 21, PnL: 1710.0
Period: 28, PnL: 1327.0

Test Case 2 - Multi-Data / Multicore with Optimization

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 0
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 463, in _handle_results
    task = get()
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'Lines_LineSeries_LineIterator_DataAccessor_ObserverBase_Observer_DataTrades_aa890439e0cc4922a6091c60d2c6e113' on <module 'backtrader.lineseries' from 'C:\\Users\\Dave\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\site-packages\\backtrader\\lineseries.py'>

Test Case 3 - Multi-Data / Multicore WITHOUT Optimization

Again, just a positive working test case showing that things are working on the multTestStrategy

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = False
    multicore = 0
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Final Portfolio Value: $10046.32851701423
P/L: $46.328517014229874

Test Case 4 - Multi-Data / Single Core with Optimization

The final test case is also quite perculiar. Limiting maxcpus to 1. I get the correct period from the strategyreturned by cerebrobut the get_value() broker calls all seem to return the same figure.

Settings:

# ------------------------ PLAY WITH THESE PARAMS -------------------------/
    optimize  = True
    multicore = 1
    testStrategy = multTestStrategy
    second_data = True
# -------------------------------------------------------------------------/

Output:

Results: Ordered by period:
Period: 14, PnL: 333.98
Period: 21, PnL: 333.98
Period: 28, PnL: 333.98
Results: Ordered by Profit:
Period: 14, PnL: 333.98
Period: 21, PnL: 333.98
Period: 28, PnL: 333.98

As we can see the period is changing but the PnL remains the same.

So overall, I am not sure what is causing it but I am also not sure how where to begin with debugging!

@totoro

Regarding the partial candle data, even though the PR wasn't merged, you can pick up fork of the Ed's and the others awesome work here:

https://github.com/Dave-Vallance/bt-ccxt-store

The repo contains made a few changes I made including an attempt to address the partial candle issue (although the solution is different to the PR). These changes work for me on the few exchanges I am looking at but your mileage may vary and I don't really plan on supporting it :)

There are some other changes such as speeding up backfilling and allowing you to map some broker keys and strings if the defaults don't work amongst other things. (for example, since CCXT does not have a unified stop/limit order implementation, Kraken expects you to create a stoploss with 'stop-loss', whereas bitmex expects just 'stop'.

For a full list of changes you can checkout the readme.

@backtrader

Sure, I appreciate that thanks.

So in the end I decided to do something a bit hacky and add a attribute to the CCXT broker that can be overwritten from inside the strategy. I.e When I set it to false, the broker will just return the last cash and value figures that were downloaded. If it is true the broker will make the proper store method call.

Probably not the best idea but needs must :) I can now warm up the indicators without waiting a lifetime.

Hello All

Is there way to prevent the broker from attempting to call getcash() and getvalue() during a backfill? I.e when self._state = self._ST_HISTORBACK ?

I guess this might be an illogical thing to ask since we can have multiple feeds and some might be delayed, reconnecting backfilling etc whilst others are live....

The reason I am asking is because I have been playing around with the CCXT implementation from Mr Bartosh and noticed that Backfilling takes a long time as a rest API call is made to access cash and value before the bar comes in and after.

So I am trying to think of a way to speed things up a little.

Cheers!

Hi Jimmy,
You can just take the last JPYUSD close price and perform a currency conversion at that price.

For example, if the last close was 100 JPY to 1 USD and your profit was 500 JPY, then you made 5 USD profit.

It gets more complex if your account currency is in USD and you were trading GBPEUR or something without USD. In that case, you would need to add another currency feed to the system and then do a conversion from the feed with the USD pair.

ok, noted thank you!