@rluc99 Thanks for the update!

Sorry I haven't used the demo account. You might want to search https://groups.io/g/twsapi .

First let's get some historical data. Try doing it like this. Set up your stock key word arguments as follows:

stockkwargs = dict(
       timeframe=bt.TimeFrame.Minutes,
       rtbar=False,  # use RealTime 5 seconds bars
       historical=True,  # only historical download
       qcheck=0.5,  # timeout in seconds (float) to check for events
       fromdate=datetime(2019, 9, 24),  # get data from..
       todate=datetime(2019, 9, 25),  # get data from..
       latethrough=False,  # let late samples through
       tradename=None  # use a different asset as order target
   )

Then get your data using:

data0 = store.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)

Then add your data using:

cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=5)

If you print from next() in your strategy, or call a log function from there, you should get some results. Make sure TWS is open and connected.

This whole code works for historical. Once you have that working, switch to live and try again when the market is open.

#!/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-e

from __future__ import absolute_import, division, print_function, unicode_literals
import backtrader as bt
import datetime

class St(bt.Strategy):

    def logdata(self):
            txt = []
            txt.append("{}".format(self.data._name))
            txt.append("{}".format(len(self)))
            txt.append("{}".format(self.data.datetime.date(0)))
            txt.append("{:.2f}".format(self.data.open[0]))
            txt.append("{:.2f}".format(self.data.high[0]))
            txt.append("{:.2f}".format(self.data.low[0]))
            txt.append("{:.2f}".format(self.data.close[0]))
            txt.append("{:.2f}".format(self.data.volume[0]))
            print(", ".join(txt))


    def next(self):
        self.logdata()


def run():

    cerebro = bt.Cerebro(stdstats=False)

    store = bt.stores.IBStore(host="127.0.0.1", port=7497, clientId= ###insert_your_number_here### )
    cerebro.broker = store.getbroker()

    stockkwargs = dict(
        timeframe=bt.TimeFrame.Minutes,
        rtbar=False,  # use RealTime 5 seconds bars
        historical=True,  # only historical download
        qcheck=0.5,  # timeout in seconds (float) to check for events
        fromdate=datetime.datetime(2019, 9, 24),  # get data from..
        todate=datetime.datetime(2019, 9, 25),  # get data from..
        latethrough=False,  # let late samples through
        tradename=None  # use a different asset as order target
    )
    data0 = store.getdata(dataname="AAPL-STK-SMART-USD", **stockkwargs)
    cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=5)

    cerebro.addstrategy(St)
    cerebro.run()




if __name__ == "__main__":

    run()

The output starts like this...

TWS Time at connection:20190926 17:50:20 EST
AAPL-STK-SMART-USD, 1, 2019-09-24, 219.90, 219.90, 219.80, 219.80, 12.00
AAPL-STK-SMART-USD, 2, 2019-09-24, 219.80, 219.80, 219.80, 219.80, 0.00
AAPL-STK-SMART-USD, 3, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 1.00
AAPL-STK-SMART-USD, 4, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 0.00
AAPL-STK-SMART-USD, 5, 2019-09-24, 219.90, 219.90, 219.90, 219.90, 0.00
AAPL-STK-SMART-USD, 6, 2019-09-24, 219.96, 220.30, 219.96, 220.30, 12.00
AAPL-STK-SMART-USD, 7, 2019-09-24, 220.30, 220.67, 220.30, 220.39, 35.00
AAPL-STK-SMART-USD, 8, 2019-09-24, 220.21, 220.21, 220.16, 220.16, 16.00
AAPL-STK-SMART-USD, 9, 2019-09-24, 220.05, 220.06, 220.03, 220.06, 4.00

You may have already checked this, but in case you haven't.

Getting zero size for every trade could imply there's not enough cash to execute the transaction. Check to see if you are getting margined out. Read "Orders -Order Status Values"

@Mexflubber said in Resampling issue, duplicated records.:

How can I spot where the data issue is ?

Check the length of the data line. For datas[0] for example insert...

len(self.datas[0])

into the next function in the same line you are printing your information. By checking the length of the data line you can check to see if the data is simply being displayed again or you have new data. If length is the same for your duplicated data, you are just seeing the same data twice and have no duplicated data.

Can you provide examples of your code?

The end of this might help.

@backtrader Thanks!

I'm trying to implement a daily strategy. Ideally I would like to check the prices at open, adjust the size of my order, and send the order. I've used cheat-on-open in back testing and it worked fine.

Now trying to trade in paper with IB. When I run the algorithm, it automatically executes all the instructions within the timer function right away instead of waiting for the timer. I'm sure I'm missing a setting or misunderstanding how this works.

Here is my relevant code... cheat is cancelled in this run.

class DoubleTrendLS(bt.Strategy):
    """
    Back test implementation of trading strategy called Double Trend Long/Short Strategy
    """

    params = dict(
        # when=(datetime.now() + timedelta(seconds=60)).time(),
        when=time(15, 44, 0, 0), # 5 minutes ahead of run time.
        # when=bt.timer.SESSION_START,
        timer=True,
        cheat=False,
        offset=timedelta(),
        repeat=timedelta(),
        weekdays=[1, 2, 3, 4, 5],
        weekcarry=False,
        monthdays=[],
        monthcarry=True,
        buffer=0.2,
    )

    def __init__(self):
        """Set up indicators line level"""
        # Create timer
        self.cheating = self.cerebro.p.cheat_on_open
        self.data_live = False

        if self.p.timer:
            self.add_timer(
                when=self.p.when,
                offset=self.p.offset,
                repeat=self.p.repeat,
                weekdays=self.p.weekdays,
                weekcarry=self.p.weekcarry,
                monthdays=self.p.monthdays,
                monthcarry=self.p.monthcarry,
                # tzdata=self.datas[0],
            )
        if self.cheating:
            self.add_timer(
                when=self.p.when,
                offset=self.p.offset,
                repeat=self.p.repeat,
                weekdays=self.p.weekdays,
                weekcarry=self.p.weekcarry,
                monthdays=self.p.monthdays,
                monthcarry=self.p.monthcarry,
                # tzdata=self.datas[0],
                cheat=True,
            )

   def notify_timer(self, timer, when, *args, **kwargs):

        if not self.data_live:
            return

        print(
            "strategy notify_timer with tid {}, when {} cheat {}".format(
                timer.p.tid, when, timer.p.cheat
            )
        )

        self.logdata("Timer:")
        self.operate() # this has the algorithm

def next(self):
    # just prints logs...

def operate(self):
    # does stuff and trades...

def run_strat(args=None):
    # Cerebro create
    cerebro = bt.Cerebro(stdstats=False, cheat_on_open=False) # currently trying it without cheat...

    # Open the IB Store.
    store = bt.stores.IBStore(host="127.0.0.1", port=7497, clientId=147)
    cerebro.broker = store.getbroker(**eval("dict(" + args.broker + ")"))

    stockkwargs = dict(
        timeframe=bt.TimeFrame.Days,
        rtbar=False,  # use RealTime 5 seconds bars
        historical=bool(int(args.historical)),  # only historical download
        what="TRADES",
        useRTH=False,  # historical - download only Regular Trading Hours
        qcheck=0.5,  # timeout in seconds (float) to check for events
        backfill_start=True,  # do backfilling at the start
        backfill=True,  # do backfilling when reconnecting
        fromdate=date_start,  # get data from..
        todate=date_end,  # get data from..
        latethrough=False,  # let late samples through
        tradename=None,  # use a different asset as order target
    )

    # Add Nasdaq data line.
    data0 = store.getdata(dataname=args.data0, **stockkwargs)
    cerebro.adddata(data0)

    # Add S&P500 data line.
    data1 = store.getdata(dataname=args.data1, **stockkwargs)
    cerebro.adddata(data1)

    # Add S&P short data line.
    data2 = store.getdata(dataname=args.data2, **stockkwargs)
    cerebro.adddata(data2)

And running:

main_live.py --buffer 0.025 --data0 TQQQ-STK-ISLAND-USD --data1 SPXL-STK-ISLAND-USD --data2 SPXS-STK-ISLAND-USD --historical 0 --cerebro "optreturn=False, optdatas=True, stdstats=False" --broker "coo=False" --ploton 0

With no active positions, and the timer set to 5 minutes ahead of run time, when run, the order fills immediately.

Print from next: 2019-08-31 03:59:59, TQQQ-STK-ISLAND-USD, close: 60.3  
Print from next: 2019-09-03 00:00:00, TQQQ-STK-ISLAND-USD, close: 58.69  
***** DATA NOTIF: LIVE
***** DATA NOTIF: LIVE
strategy notify_timer with tid 0, when 2019-09-03 15:44:00 cheat False
Called from: Timer:
TQQQ-STK-ISLAND-USD, 557, 2019-09-03, 58.69, 58.69, 58.69, 58.69, 1.00
SPXL-STK-ISLAND-USD, 557, 2019-09-02, 48.75, 49.17, 48.00, 48.64, 3139.00
SPXS-STK-ISLAND-USD, 557, 2019-09-02, 18.62, 18.82, 18.40, 18.59, 16446.00
Nasdaq 30 day: 59.03, S&P 12 mo: 46.41
This is the 'unrate': 3.7 and unrate indicator: 3.75, -0.05


Called from: Operate
TQQQ-STK-ISLAND-USD, 557, 2019-09-03, 58.69, 58.69, 58.69, 58.69, 1.00
SPXL-STK-ISLAND-USD, 557, 2019-09-02, 48.75, 49.17, 48.00, 48.64, 3139.00
SPXS-STK-ISLAND-USD, 557, 2019-09-02, 18.62, 18.82, 18.40, 18.59, 16446.00
Nasdaq 30 day: 59.03, S&P 12 mo: 46.41
This is the 'unrate': 3.7 and unrate indicator: 3.75, -0.05


2019-09-03 Send BUY for SPXL-STK-ISLAND-USD,  value: 10000.00,	size:205,	close 48.64
Print from next: 2019-09-03 19:42:53, TQQQ-STK-ISLAND-USD, close: 58.69  
Print from next: 2019-09-03 19:42:56, TQQQ-STK-ISLAND-USD, close: 58.68  
***** DATA NOTIF: LIVE
Print from next: 2019-09-03 19:43:01, TQQQ-STK-ISLAND-USD, close: 58.69  
Print from next: 2019-09-03 19:43:02, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 19:43:02, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
2019-09-03, SPXL-STK-ISLAND-USD Order 1 Status Partial
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.68  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65  
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65  
2019-09-03, SPXL-STK-ISLAND-USD Order 1 Status Completed
2019-09-03, BUY EXECUTED for SPXL-STK-ISLAND-USD, Price: 48.65, Cost: 9972.25, Comm 1.54
-- No longer alive main Ref
Print from next: 2019-09-03 23:59:59, TQQQ-STK-ISLAND-USD, close: 58.65  

As always, appreciate any advice and guidance.

run-out

This is not an indicator, but an added function that will return empyrical financial ratios at the end of the back test.

This example is for an equal risk parity rebalancing algorithm that has five assets. A dataframe is returned that provides financial ratios for not only the strategy but for each of the datas. This works best with daily data. Two of the calculations are for annual returns and will throw an error if the test is less than one year.

                   Strategy  MSCI_ACWI  SP500  TBOND20   gold  iSharesREIT
annual_return         0.048      0.080  0.114    0.024  0.027        0.055
annual_volatility     0.069      0.150  0.123    0.130  0.136        0.142
cagr                  0.048      0.080  0.114    0.024  0.027        0.055
calmar                0.507      0.350  0.873    0.133  0.138        0.322
cumm_return           0.148      0.261  0.381    0.073  0.084        0.174
max_drawdown         -0.096     -0.230 -0.130   -0.179 -0.197       -0.171
sharpe                0.719      0.590  0.938    0.246  0.265        0.449
sortino               1.015      0.805  1.336    0.342  0.389        0.613
tail_ratio            1.064      1.003  1.012    0.940  1.070        0.864

The Strategy column is the result of the back test, and the other five columns are the datas used.

To implement this, you need the following libraries:

import pandas as pd
import empyrical as ep     # https://github.com/quantopian/empyrical

Add two analyzers:

class CashMarket(bt.analyzers.Analyzer):
    """
    Analyzer returning cash and market values
    """

    def start(self):
        super(CashMarket, self).start()

    def create_analysis(self):
        self.rets = {}
        self.vals = 0.0

    def notify_cashvalue(self, cash, value):
        self.vals = (cash, value)
        self.rets[self.strategy.datetime.datetime()] = self.vals

    def get_analysis(self):
        return self.rets


class OHLCV(bt.analyzers.Analyzer):
    """This analyzer reports the OHLCV of each of datas.
    Params:
      - timeframe (default: ``None``)
        If ``None`` then the timeframe of the 1st data of the system will be
        used
      - compression (default: ``None``)
        Only used for sub-day timeframes to for example work on an hourly
        timeframe by specifying "TimeFrame.Minutes" and 60 as compression
        If ``None`` then the compression of the 1st data of the system will be
        used
    Methods:
      - get_analysis
        Returns a dictionary with returns as values and the datetime points for
        each return as keys
    """

    def start(self):
        tf = min(d._timeframe for d in self.datas)
        self._usedate = tf >= bt.TimeFrame.Days

    def next(self):
        # Cycle through the datas at each next and store the OHLCV
        # (d.l.lines.__len__())
        pvals = {}
        # Some datas I use have only 'Close' value used for signalling. The try statement
        # avoids errors on these datas.
        for d in self.datas:
            try:
                d.open[0]
                d.high[0]
                d.low[0]
                d.volume[0]
            except:
                continue
            else:
                pvals = [d.open[0], d.high[0], d.low[0], d.close[0], d.volume[0]]

                if self._usedate:
                    self.rets[(self.strategy.datetime.date(), d._name)] = pvals
                else:
                    self.rets[(self.strategy.datetime.datetime(), d._name)] = pvals

    def get_analysis(self):
        return self.rets

Add function at the top level for returning financial ratios.

def fin_funcs(returns_series, risk_free_rate=0):
    """
    Financial calculations taken from Quantopians Empirical Library.

    :param df: pd.Series containing daily returns calculated on a percentage change and also by log scale.
    :return: Dictionary of financial ratios both for percent change returns and log returns.
    """
    returns_pct = returns_series

    # Calculate each of the functions.
    annual_return_pct = ep.annual_return(
        returns_pct, period="daily", annualization=None
    )
    cumm_return_pct = ep.cum_returns(returns_pct, starting_value=0).iloc[-1]
    cagr_pct = ep.cagr(returns_pct, period="daily", annualization=None)
    sharpe_pct = ep.sharpe_ratio(
        returns_pct, risk_free=risk_free_rate, period="daily", annualization=None
    )
    annual_volatility_pct = ep.annual_volatility(
        returns_pct, period="daily", alpha=2.0, annualization=None
    )
    max_drawdown_pct = ep.max_drawdown(returns_pct)
    calmar_pct = ep.calmar_ratio(returns_pct, period="daily", annualization=None)
    sortino_pct = ep.sortino_ratio(
        returns_pct,
        required_return=0,
        period="daily",
        annualization=None,
        _downside_risk=None,
    )
    tail_ratio_pct = ep.tail_ratio(returns_pct)

    # Collect ratios into dictionary.
    financials = {
        "annual_return": annual_return_pct,
        "cumm_return": cumm_return_pct,
        "cagr": cagr_pct,
        "sharpe": sharpe_pct,
        "annual_volatility": annual_volatility_pct,
        "max_drawdown": max_drawdown_pct,
        "calmar": calmar_pct,
        "sortino": sortino_pct,
        "tail_ratio": tail_ratio_pct,
    }

    return financials

Add analyzers...

cerebro.addanalyzer(CashMarket, _name="cash_market")
cerebro.addanalyzer(OHLCV, _name="ohlcv")

After running cerebro...
Cerebro run

strat = cerebro.run(**eval("dict(" + args.cerebro + ")"))

Create dictionary for collecting all the financial results to then convert to pandas dataframe.
First collect the strategy results, then the datas results.

fin_results = {}

# Calculate the financial functions for the strategy results
# First, get market values for the algorithm, second item in list in dict from analyzer cash_market
dict_mv = strat[0].analyzers.getbyname("cash_market").get_analysis()

# Create lists for values and keys for pd.Series, then create pd.Series with pct_change.
v = [x[1] for x in dict_mv.values()]
d = [x.date() for x in dict_mv.keys()]
returns_series = pd.Series(v, index=d, name="Strategy")
returns_series = returns_series[int(args.rperiod) :].pct_change()

# Call fin_funcs and get dictionary back. Add to new dictionary tracking all the financials.
fin_results["Strategy"] = fin_funcs(returns_series)

Then collect data for each datas and get financial ratios.

# Get the OHLCV for each data.
dict_ohlcv = strat[0].analyzers.getbyname("ohlcv").get_analysis()

# Create a dataframe from the analyzer ohlcv, percent change by date.
df = (
    pd.DataFrame(pd.DataFrame.from_dict(dict_ohlcv).unstack())
    .loc[pd.IndexSlice[:, :, 3], :]
    .droplevel(2)
    .reset_index()
    .pivot(index="level_0", columns="level_1", values=0)
    .pct_change()
)

# Get financial ratios for each security column and add to fin_results dictionary.
for n in range(len(df.columns)):
    fin_results[df.columns[n]] = fin_funcs(df.iloc[:, n])

Convert results dictionary to a DataFrame for presenting results.

df_results = pd.DataFrame.from_dict(fin_results)
print(df_results.round(3))