@hangouts91 Your lines declaration is a string, not a tuple (your are missing the trailing comma). Change it to

lines = ('numtrd',)

@hfrog713 It's hard to really provide any helpful information without knowing what "different results" means. Are you getting different EMA values? Are you getting different buy/sell signals from the crossover indicator? Are the signals the same but they result in different orders?

Additionally, without the full script and the data file, it's going to be hard to try it out locally. I ran your script against some AAPL prices and it just generates one sell signal during the period of test. Is that right or wrong? Is it consistent with what TradeView would show? It's hard for me to know.

@li-mike I don't think BT supports "boxed" positions (simultaneous long and short in the same security) but I would think you could fake it by adding the same feed for the security twice and just treating one of them as the long and the other as the short.

@kjiessar You are always receiving data in "bars". A bar is a summary of all the trades over a certain time interval. Each interval will have its own Open, High, Low and Close. You receive the bar innext() just after the interval is complete so the Close is never in the future, it always just happened. The Open will be further in the past by the interval of time of that bar.

Typically if you are using daily data, then next() is being called at the end of the trading day. If you place an order, the first opportunity for it to get filled is at the next days's Open price.

@ruiarruda said in Transactions happen for times that don't exist in source data:

However, the most confusing issue still stands: the RSI indicator, which I initialize but do NOT use (intentionally, at least), is somehow necessary for the code to run (if you comment it out on your code, the code breaks).

Here is the issue

self.interval_low = min(self.data.low.get(size=3))

In next you are trying to access the last 3 low values but in the first call to next, there is only one value in self.data.low at this point. When you leave in the line for the RSI indicator, BT buffers the first 14 data points before calling next so the RSI indicator will be available (whether you access it or not).

You could change the offending line to:

self.interval_low = min(self.data.low.get(size=min(3, len(self.data.low))))

or just create some indicator with a minimum period of 3 to work around this.

@dizzy0ny I think you'd need a custom feed derived from PandasData like in my example with all the additional "lines". But I supposed you could create one that just dynamically added the lines based on the columns in your DataFrame. But doing it explicitly isn't that complicated, as my example shows.

As for the vector vs. event-based backtesting, I think it's probably true that a vector-based approach is more powerful in some ways and bound to be a lot faster, but I think there is some logic that is easier to express in an event-driven approach which might lead to fewer mistakes (although I'm speculating a bit). The event-driven approach is also "safer" in that you can't cheat and accidentally look at future values. Finally, Backtrader makes is pretty straightforward to switch from backtesting to live trading. That might be more challenging with a vector-based system.

BTW, here's a vector-based Python backtesting project I found that looks interesting: vectorbt

@new_trader Here is how limit orders work:

• A buy limit order will only get filled at or below the limit price. The limit is the maximum you are willing to pay. You put a 149.5 limit on it and got filled below your limit. That's good.
• A sell limit order will only get filled at or above the limit price. The limit is the minimum you are willing to receive. Your sell limit was way above the market so you did not get filled.
• A sell stop order is an order that gets activated once the stop price is reached. You always put sell stops below the current market price or they will be immediately activated. Once they are activated, they become market orders and will be executed at the current market price. You put a sell stop order way above the market so it was immediately active and it got filled.
• A sell stop limit order has two prices associated with it: the stop price is the price at which it becomes activated at which time it becomes a sell limit order at the limit price. And like any sell limit order, it is the minimum price you are will to receive. You did not use this order type. You used a sell stop order but not a sell stop limit order.

Hope that helps.

@punajunior It looks like the date format in your file is "%m/%d/%Y" but you specified "%Y-%m-%d" in your script. Try changing the dtformat argument of GenericCSVData.

@catvin When you create expressions using lines and indicators, you need to use operations that Backtrader understands for those operations. It supports the basic arithmetic operators like +, -, *, / but doesn't know about math.log. But there is a helper indicator called ApplyN that can apply a function across a line to create a new indicator, which fits your case nicely. The only caveat is that ApplyN generally works with a list of values over a period, not a scalar, so below I've used a lambda to just pass the one value in that period to the math.log function:

self.lines.x = bt.ind.ApplyN(self.close, func=lambda c: math.log(c[0]), period=1)

If you look at the implementation of ApplyN here, (which derives from BaseApplyN, which itself derives from OperationN). You can see that it simply just calls the function in next like you were originally doing!

    def next(self):
        self.line[0] = self.func(self.data.get(size=self.p.period))

So ApplyN is really just syntactic sugar for your original implementation. But I think it's more readable.

@kuky I believe BT assumes the timestamp of a bar is for the end of that time interval. Thus at the simulator time of 22:30, you are seeing the 5m bar that ended at 22:30 (i.e. 22:25-22:30) and the 1h bar that ended at 22:00 (which is the last complete 1h bar at this point).

@the-world Yes, it uses C# but it's a completely different system than BT so you can't compare the two simply on implementation language. It may have something to do with how they process historical data? (I'm guessing here). I'd have to try writing two identical algorithms in each to really understand how they compare and it's not something I've tried or plan to do.

@the-world LEAN is not as fast as BT in my experience but I personally don't think that's the most important characteristic of a backtesting environment. You'll end up spending more time figuring out how to implement, analyze and debug your strategy, so finding the tool that allows you to work productively is - to me - the most important quality of a backtesting platform.

I think LEAN's ease of use with Docker, integrated VS Code debugging and good documentation with a very active community are really attractive features, though.

@kjiessar In that analyzer, self.rets does not track trades, it tracks entries which is a list of position size, price, data name and proceeds. Additionally, the position already aggregates multiple executions for multiple trades in a given data name (in notify_order), so there isn't an issue here using datetime as a key. It simply gives you a time series record of all position metrics at each bar in the analysis for all data names.

Here is the full function for context:

def next(self):
    # super(Transactions, self).next()  # let dtkey update
    entries = []
    for i, dname in self._idnames:
        pos = self._positions.get(dname, None)
        if pos is not None:
            size, price = pos.size, pos.price
            if size:
                entries.append([size, price, i, dname, -size * price])

    if entries:
        self.rets[self.strategy.datetime.datetime()] = entries

    self._positions.clear()

@sarah_james You can use order_target_percent(data, target=0.05) to buy 5% of your current portfolio value. Then when you want to sell, you can use order_target_percent(data, target=0.0)

@jacksonx I would think you could create an Analyzer to do this. It has access to the strategy, and therefore all the lines within it, so you could calculate on each day your margin requirement and then at the end of the analysis, sum up the total margin cost. You would need to incorporate that into your net P&L, though.

They other approach would be a custom broker (derived from the standard BackBroker) that only overrides the commission calculations, but that might be more work as it doesn't appear that brokers were designed to be as easily extensible as some of the other classes.

@auth87 Yes, the pre_next method in Strategy will be called each time and you can simply call next from it. But you then need to check which of the data items in self.datas have data ready. Something like:

def pre_next(self):
    self.next()

def next(self):
    available = [d for d in self.datas if len(d)]
    # do something with available list...

@sky_aka41 I think you are pretty close. I tried this all in a strategy (not a custom indicator) and it seems to be working.

Note I changed the prob_change and probability expressions slightly.

class St(bt.Strategy):
    params = (("momLength", 34), ("maLength", 55), ("pLength", 13))

    def __init__(self) :
        self.momentum = bt.ind.Momentum(self.datas[0].close, period=self.p.momLength)
        self.accelerate = bt.ind.Momentum(self.momentum, period=self.p.momLength)
        self.prob_change = bt.ind.Momentum(self.datas[0].close, period=1) > 0
        self.probability = bt.ind.SumN(self.prob_change) / self.p.pLength 
        
        self.adjusted_close = self.datas[0].close + (self.momentum + self.accelerate*0.5 ) * self.probability
        
        self.lines.MaMA = bt.ind.SMA(self.adjusted_close, period=self.p.maLength)

@brunof I think Option 2 sounds pretty reasonable. It may seem a bit kludgy to replace the Open field with a different value, but I looked at the execute method in BackBroker and it's pretty involved so subclassing and overriding would be complicated. Note you could replace the Open with whatever you like: VWAP, the first tick 10 seconds after the open, etc. - whatever you think is a reasonable fill price.

@brunof This sounds to me like an issue with the source data bars, not with Backtrader. If you are truly aggregating tick data into bars, then a given tick would only be included in one bar. So if the close price of one bar is the same as the open price of the next bar, that would only occur if there were two separate ticks at the same price (which is quite common, btw). It would be interesting to look at the underlying tick data for some of your ten-minute bars and see if they are consistent with what you'd expect.

I think Backtrader's assumption of filling at the open of the next bar is as good of an assumption as you can make. If indeed, there were two ticks at the same level that crossed the boundary of a bar, then you would expect to get filled at the same level as the previous bar's close.

If you want a much more realistic simulation of execution behavior on an exchange, I think you'd ultimately need to use tick data.

@jialeen You can use order_target_percent(data, target=1) to have BT use 100% of the portfolio value for the size of the order.