Backtrader Community

run-out

You may have already checked this, but in case you haven't.

Getting zero size for every trade could imply there's not enough cash to execute the transaction. Check to see if you are getting margined out. Read "Orders -Order Status Values"

run-out

@Mexflubber said in Resampling issue, duplicated records.:

How can I spot where the data issue is ?

Check the length of the data line. For datas[0] for example insert...

len(self.datas[0])

into the next function in the same line you are printing your information. By checking the length of the data line you can check to see if the data is simply being displayed again or you have new data. If length is the same for your duplicated data, you are just seeing the same data twice and have no duplicated data.

run-out

Can you provide examples of your code?

run-out

The end of this might help.

run-out

@backtrader Thanks!

run-out

I'm trying to implement a daily strategy. Ideally I would like to check the prices at open, adjust the size of my order, and send the order. I've used cheat-on-open in back testing and it worked fine.

Now trying to trade in paper with IB. When I run the algorithm, it automatically executes all the instructions within the timer function right away instead of waiting for the timer. I'm sure I'm missing a setting or misunderstanding how this works.

Here is my relevant code... cheat is cancelled in this run.

class DoubleTrendLS(bt.Strategy):
    """
    Back test implementation of trading strategy called Double Trend Long/Short Strategy
    """

    params = dict(
        # when=(datetime.now() + timedelta(seconds=60)).time(),
        when=time(15, 44, 0, 0), # 5 minutes ahead of run time.
        # when=bt.timer.SESSION_START,
        timer=True,
        cheat=False,
        offset=timedelta(),
        repeat=timedelta(),
        weekdays=[1, 2, 3, 4, 5],
        weekcarry=False,
        monthdays=[],
        monthcarry=True,
        buffer=0.2,
    )

    def __init__(self):
        """Set up indicators line level"""
        # Create timer
        self.cheating = self.cerebro.p.cheat_on_open
        self.data_live = False

        if self.p.timer:
            self.add_timer(
                when=self.p.when,
                offset=self.p.offset,
                repeat=self.p.repeat,
                weekdays=self.p.weekdays,
                weekcarry=self.p.weekcarry,
                monthdays=self.p.monthdays,
                monthcarry=self.p.monthcarry,
                # tzdata=self.datas[0],
            )
        if self.cheating:
            self.add_timer(
                when=self.p.when,
                offset=self.p.offset,
                repeat=self.p.repeat,
                weekdays=self.p.weekdays,
                weekcarry=self.p.weekcarry,
                monthdays=self.p.monthdays,
                monthcarry=self.p.monthcarry,
                # tzdata=self.datas[0],
                cheat=True,
            )

   def notify_timer(self, timer, when, *args, **kwargs):

        if not self.data_live:
            return

        print(
            "strategy notify_timer with tid {}, when {} cheat {}".format(
                timer.p.tid, when, timer.p.cheat
            )
        )

        self.logdata("Timer:")
        self.operate() # this has the algorithm

def next(self):
    # just prints logs...

def operate(self):
    # does stuff and trades...

def run_strat(args=None):
    # Cerebro create
    cerebro = bt.Cerebro(stdstats=False, cheat_on_open=False) # currently trying it without cheat...

    # Open the IB Store.
    store = bt.stores.IBStore(host="127.0.0.1", port=7497, clientId=147)
    cerebro.broker = store.getbroker(**eval("dict(" + args.broker + ")"))

    stockkwargs = dict(
        timeframe=bt.TimeFrame.Days,
        rtbar=False,  # use RealTime 5 seconds bars
        historical=bool(int(args.historical)),  # only historical download
        what="TRADES",
        useRTH=False,  # historical - download only Regular Trading Hours
        qcheck=0.5,  # timeout in seconds (float) to check for events
        backfill_start=True,  # do backfilling at the start
        backfill=True,  # do backfilling when reconnecting
        fromdate=date_start,  # get data from..
        todate=date_end,  # get data from..
        latethrough=False,  # let late samples through
        tradename=None,  # use a different asset as order target
    )

    # Add Nasdaq data line.
    data0 = store.getdata(dataname=args.data0, **stockkwargs)
    cerebro.adddata(data0)

    # Add S&P500 data line.
    data1 = store.getdata(dataname=args.data1, **stockkwargs)
    cerebro.adddata(data1)

    # Add S&P short data line.
    data2 = store.getdata(dataname=args.data2, **stockkwargs)
    cerebro.adddata(data2)

And running:

main_live.py --buffer 0.025 --data0 TQQQ-STK-ISLAND-USD --data1 SPXL-STK-ISLAND-USD --data2 SPXS-STK-ISLAND-USD --historical 0 --cerebro "optreturn=False, optdatas=True, stdstats=False" --broker "coo=False" --ploton 0

With no active positions, and the timer set to 5 minutes ahead of run time, when run, the order fills immediately.

Print from next: 2019-08-31 03:59:59, TQQQ-STK-ISLAND-USD, close: 60.3  
Print from next: 2019-09-03 00:00:00, TQQQ-STK-ISLAND-USD, close: 58.69  
***** DATA NOTIF: LIVE
***** DATA NOTIF: LIVE
strategy notify_timer with tid 0, when 2019-09-03 15:44:00 cheat False
Called from: Timer:
TQQQ-STK-ISLAND-USD, 557, 2019-09-03, 58.69, 58.69, 58.69, 58.69, 1.00
SPXL-STK-ISLAND-USD, 557, 2019-09-02, 48.75, 49.17, 48.00, 48.64, 3139.00
SPXS-STK-ISLAND-USD, 557, 2019-09-02, 18.62, 18.82, 18.40, 18.59, 16446.00
Nasdaq 30 day: 59.03, S&P 12 mo: 46.41
This is the 'unrate': 3.7 and unrate indicator: 3.75, -0.05


Called from: Operate
TQQQ-STK-ISLAND-USD, 557, 2019-09-03, 58.69, 58.69, 58.69, 58.69, 1.00
SPXL-STK-ISLAND-USD, 557, 2019-09-02, 48.75, 49.17, 48.00, 48.64, 3139.00
SPXS-STK-ISLAND-USD, 557, 2019-09-02, 18.62, 18.82, 18.40, 18.59, 16446.00
Nasdaq 30 day: 59.03, S&P 12 mo: 46.41
This is the 'unrate': 3.7 and unrate indicator: 3.75, -0.05


2019-09-03 Send BUY for SPXL-STK-ISLAND-USD,  value: 10000.00,	size:205,	close 48.64
Print from next: 2019-09-03 19:42:53, TQQQ-STK-ISLAND-USD, close: 58.69  
Print from next: 2019-09-03 19:42:56, TQQQ-STK-ISLAND-USD, close: 58.68  
***** DATA NOTIF: LIVE
Print from next: 2019-09-03 19:43:01, TQQQ-STK-ISLAND-USD, close: 58.69  
Print from next: 2019-09-03 19:43:02, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 19:43:02, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
2019-09-03, SPXL-STK-ISLAND-USD Order 1 Status Partial
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65  
2019-09-03, SPXL-STK-ISLAND-USD Order 1 Status Completed
2019-09-03, BUY EXECUTED for SPXL-STK-ISLAND-USD, Price: 48.65, Cost: 9972.25, Comm 1.54
-- No longer alive main Ref
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65

As always, appreciate any advice and guidance.

run-out

run-out

This is not an indicator, but an added function that will return empyrical financial ratios at the end of the back test.

This example is for an equal risk parity rebalancing algorithm that has five assets. A dataframe is returned that provides financial ratios for not only the strategy but for each of the datas. This works best with daily data. Two of the calculations are for annual returns and will throw an error if the test is less than one year.

                   Strategy  MSCI_ACWI  SP500  TBOND20   gold  iSharesREIT
annual_return         0.048      0.080  0.114    0.024  0.027        0.055
annual_volatility     0.069      0.150  0.123    0.130  0.136        0.142
cagr                  0.048      0.080  0.114    0.024  0.027        0.055
calmar                0.507      0.350  0.873    0.133  0.138        0.322
cumm_return           0.148      0.261  0.381    0.073  0.084        0.174
max_drawdown         -0.096     -0.230 -0.130   -0.179 -0.197       -0.171
sharpe                0.719      0.590  0.938    0.246  0.265        0.449
sortino               1.015      0.805  1.336    0.342  0.389        0.613
tail_ratio            1.064      1.003  1.012    0.940  1.070        0.864

The Strategy column is the result of the back test, and the other five columns are the datas used.

To implement this, you need the following libraries:

import pandas as pd
import empyrical as ep     # https://github.com/quantopian/empyrical

Add two analyzers:

class CashMarket(bt.analyzers.Analyzer):
    """
    Analyzer returning cash and market values
    """

    def start(self):
        super(CashMarket, self).start()

    def create_analysis(self):
        self.rets = {}
        self.vals = 0.0

    def notify_cashvalue(self, cash, value):
        self.vals = (cash, value)
        self.rets[self.strategy.datetime.datetime()] = self.vals

    def get_analysis(self):
        return self.rets


class OHLCV(bt.analyzers.Analyzer):
    """This analyzer reports the OHLCV of each of datas.
    Params:
      - timeframe (default: ``None``)
        If ``None`` then the timeframe of the 1st data of the system will be
        used
      - compression (default: ``None``)
        Only used for sub-day timeframes to for example work on an hourly
        timeframe by specifying "TimeFrame.Minutes" and 60 as compression
        If ``None`` then the compression of the 1st data of the system will be
        used
    Methods:
      - get_analysis
        Returns a dictionary with returns as values and the datetime points for
        each return as keys
    """

    def start(self):
        tf = min(d._timeframe for d in self.datas)
        self._usedate = tf >= bt.TimeFrame.Days

    def next(self):
        # Cycle through the datas at each next and store the OHLCV
        # (d.l.lines.__len__())
        pvals = {}
        # Some datas I use have only 'Close' value used for signalling. The try statement
        # avoids errors on these datas.
        for d in self.datas:
            try:
                d.open[0]
                d.high[0]
                d.low[0]
                d.volume[0]
            except:
                continue
            else:
                pvals = [d.open[0], d.high[0], d.low[0], d.close[0], d.volume[0]]

                if self._usedate:
                    self.rets[(self.strategy.datetime.date(), d._name)] = pvals
                else:
                    self.rets[(self.strategy.datetime.datetime(), d._name)] = pvals

    def get_analysis(self):
        return self.rets

Add function at the top level for returning financial ratios.

def fin_funcs(returns_series, risk_free_rate=0):
    """
    Financial calculations taken from Quantopians Empirical Library.

    :param df: pd.Series containing daily returns calculated on a percentage change and also by log scale.
    :return: Dictionary of financial ratios both for percent change returns and log returns.
    """
    returns_pct = returns_series

    # Calculate each of the functions.
    annual_return_pct = ep.annual_return(
        returns_pct, period="daily", annualization=None
    )
    cumm_return_pct = ep.cum_returns(returns_pct, starting_value=0).iloc[-1]
    cagr_pct = ep.cagr(returns_pct, period="daily", annualization=None)
    sharpe_pct = ep.sharpe_ratio(
        returns_pct, risk_free=risk_free_rate, period="daily", annualization=None
    )
    annual_volatility_pct = ep.annual_volatility(
        returns_pct, period="daily", alpha=2.0, annualization=None
    )
    max_drawdown_pct = ep.max_drawdown(returns_pct)
    calmar_pct = ep.calmar_ratio(returns_pct, period="daily", annualization=None)
    sortino_pct = ep.sortino_ratio(
        returns_pct,
        required_return=0,
        period="daily",
        annualization=None,
        _downside_risk=None,
    )
    tail_ratio_pct = ep.tail_ratio(returns_pct)

    # Collect ratios into dictionary.
    financials = {
        "annual_return": annual_return_pct,
        "cumm_return": cumm_return_pct,
        "cagr": cagr_pct,
        "sharpe": sharpe_pct,
        "annual_volatility": annual_volatility_pct,
        "max_drawdown": max_drawdown_pct,
        "calmar": calmar_pct,
        "sortino": sortino_pct,
        "tail_ratio": tail_ratio_pct,
    }

    return financials

Add analyzers...

cerebro.addanalyzer(CashMarket, _name="cash_market")
cerebro.addanalyzer(OHLCV, _name="ohlcv")

After running cerebro...
Cerebro run

strat = cerebro.run(**eval("dict(" + args.cerebro + ")"))

Create dictionary for collecting all the financial results to then convert to pandas dataframe.
First collect the strategy results, then the datas results.

fin_results = {}

# Calculate the financial functions for the strategy results
# First, get market values for the algorithm, second item in list in dict from analyzer cash_market
dict_mv = strat[0].analyzers.getbyname("cash_market").get_analysis()

# Create lists for values and keys for pd.Series, then create pd.Series with pct_change.
v = [x[1] for x in dict_mv.values()]
d = [x.date() for x in dict_mv.keys()]
returns_series = pd.Series(v, index=d, name="Strategy")
returns_series = returns_series[int(args.rperiod) :].pct_change()

# Call fin_funcs and get dictionary back. Add to new dictionary tracking all the financials.
fin_results["Strategy"] = fin_funcs(returns_series)

Then collect data for each datas and get financial ratios.

# Get the OHLCV for each data.
dict_ohlcv = strat[0].analyzers.getbyname("ohlcv").get_analysis()

# Create a dataframe from the analyzer ohlcv, percent change by date.
df = (
    pd.DataFrame(pd.DataFrame.from_dict(dict_ohlcv).unstack())
    .loc[pd.IndexSlice[:, :, 3], :]
    .droplevel(2)
    .reset_index()
    .pivot(index="level_0", columns="level_1", values=0)
    .pct_change()
)

# Get financial ratios for each security column and add to fin_results dictionary.
for n in range(len(df.columns)):
    fin_results[df.columns[n]] = fin_funcs(df.iloc[:, n])

Convert results dictionary to a DataFrame for presenting results.

df_results = pd.DataFrame.from_dict(fin_results)
print(df_results.round(3))

run-out

This is a working example of a long/short strategy with multiple datas.
****This was my first backtrader project so you'll excuse any crudities.

Essentially, I was calculating the daily long and shorts from a 5,000 stock universe, resulting in 50 long and 50 shorts a day. This part was using an external module and importing the long and short recommendations per day as a dictionary. Then, I would adjust the portfolio with buys and sells each day according to the dictionary.

I hope there's some useful reference. Again, this was my first project, so use with a grain of salt.

https://bit.ly/2kcMfSZ

run-out

@CooleRnax You may wish to consider changing the error line to:

self.atr = abs(self.rsi_ma(-1) - self.rsi_ma)

Your indicator runs for me using this.

run-out

@Jose-Larghi Do you know the full universe you are selecting from ahead of time? Are you randomly selecting stocks from a known universe?

run-out

How are you selecting assets through time?

run-out

@Bayne Try order.ref

run-out

Backtrader shows me :
2019-06-30, Close, 295.65
2019-06-30, Close, 295.65
2019-06-30, Close, 295.85
But how can I have H-M-S too in the log function ?

Try changing

dt = dt or self.datas[0].datetime.date(0)

to ...

dt = dt or self.datas[0].datetime.datetime(0)

run-out

Thanks for your support. I appreciate it and all the work you do.

I tried running historical and was still hanging up, so I did what your'e not supposed to do, I deleted and reinstalled backtrader, IbPy2, and TWS all at the same time.

When I reinstalled TWS I installed the 'older' version.

And now, no problem, historical info coming in at light speed. Which is great.

But since I also did the other uninstalls and installs, I can only speculate that the latest/best/greatest stable TWS was the problem? Personally I think that was it.

Regarding your side note, what do you normally use for running your programs if not an IDE like pycharm?

Thanks again DR for all your help! Awesome package. Keep up the great work.

run-out

Thanks for the reply. Turns out I was selecting the wrong item from inside pycharm. I'm now using IbPy2 but still the program is hanging up at

/home/runout/Documents/quant/ibtest/venv/bin/python /home/runout/Documents/quant/ibtest/ibsimple.py
Server Version: 76
TWS Time at connection:20190825 13:02:17 EST
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** DATA NOTIF: DELAYED

The program is still active here.

Also, the TWS is showing an API Connection status of 'accepted', not connected.

API Connections Status

When I break/stop the program, I get the following:

Traceback (most recent call last):
  File "/home/runout/Documents/quant/ibtest/ibsimple.py", line 95, in <module>
    run()
  File "/home/runout/Documents/quant/ibtest/ibsimple.py", line 91, in run
    cerebro.run()
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/cerebro.py", line 1542, in _runnext
    drets.append(d.next(ticks=False))
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/feed.py", line 407, in next
    ret = self.load()
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/feed.py", line 479, in load
    _loadret = self._load()
  File "/home/runout/Documents/quant/ibtest/venv/lib/python3.7/site-packages/backtrader/feeds/ibdata.py", line 442, in _load
    self.qlive.get(timeout=self._qcheck))
  File "/home/runout/anaconda3/lib/python3.7/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/home/runout/anaconda3/lib/python3.7/threading.py", line 300, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

Process finished with exit code 1

run-out

Connecting to Interactive Broker gets stuck at delay. I am using a simplified script to isolate the problem.

def run(args=None):
    cerebro = bt.Cerebro(stdstats=False)
    store = bt.stores.IBStore(host='127.0.0.1', port=7497, clientId=147)
    cerebro.broker = store.getbroker()
    # also: 'TQQQ-STK-ISLAND-USD'
    data = store.getdata(dataname='AAPL-STK-SMART-USD', timeframe=bt.TimeFrame.Days)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Days, compression=1)

    cerebro.addstrategy(St)
    cerebro.run()

The error message is:

Server Version: 76
TWS Time at connection:20190825 07:09:20 EST
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm.nj>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:eufarm>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds.nj>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>
***** DATA NOTIF: DELAYED

I have verified the connection to TWS works using IBDataCache script which connects using ibConnection. I am connecting using an approved and funded account and connected to a paper trading account.

When debugging, I can get as far as notify_data. Script does not get to next().

I've installed Interactive Brokers Python API Version 0.9/70 Author Troy Melhase. I've also created a new 'clean' envirnment to reduce the posibility of conflicts. The same error comes up.

I'm using:
Python 3.7
Backtrader 1.9.74.123
ib-api 0.9.70
linux-ubuntu 18.04.3 LTS

Thanks for any assistance, I've exhausted the docs...

run-out

There's a good example here. Fixed Income Scheme

A class is created as follows:

class FixedCommisionScheme(bt.CommInfoBase):
    '''
    This is a simple fixed commission scheme
    '''
    params = (
        ('commission', 5),
        ('stocklike', True),
        ('commtype', bt.CommInfoBase.COMM_FIXED),
        )

    def _getcommission(self, size, price, pseudoexec):
        return self.p.commission

You could modify the _getcommission method to read something like:

    def _getcommission(self, size, price, pseudoexec):
        if abs(size * price) < 10000:
            return 10
        else:
            return 20

Once made, add this to cerebro:

#Set commissions
comminfo = FixedCommisionScheme()
cerebro.broker.addcommissioninfo(comminfo)

run-out

In the meantime use Jupyter Notebook and run the back test from there, use analyzers to extract the basic information, and then pandas and plotly to graph any way you wish.

run-out

@backtrader My bad...

run-out

@jesseliu0 You could try...

self.datas[0].datetime.date(1)

But you'll get an error on the last datas.

Posts made by run-out