Adding individual futures data with rolloverdata feed - Import #158

[backtrader]

From https://github.com/mementum/backtrader/issues/158 - for reference purposes

Hi,

The below is an excerpt of your sample code in the doc:

cerebro = bt.Cerebro()
data0 = bt.feeds.MyFeed(dataname='Expiry0')
data1 = bt.feeds.MyFeed(dataname='Expiry1')
...
dataN = bt.feeds.MyFeed(dataname='ExpiryN')

drollover = cerebro.rolloverdata(data0, data1, ..., dataN, name='MyRoll', **kwargs)

cerebro.run()

Is there a way to add separate futures feed in a loop? Maybe something conceptually like below:

cerebro = bt.Cerebro()
for i in range(0, len(data_DB)):
    data_temp = bt.feeds.MyFeed(dataname=data_DB.ix[:,i])
    drollover = cerebro.rolloverdata(data_temp, name='MyRoll', **kwargs)
cerebro.run()

Thanks for your attention in advance. Please allow me to let you know I have been having great fun with your platform and I am greatly thankful for your work.

[backtrader]

I probably misunderstand your question and therefore my simple answer. This seems to be a matter of *args expansion.

cerebro = bt.Cerebro()

dtemp = [bt.feeds.MyFeed(dataname=data_DB.ix[:,i]) for i in range(0, len(data_DB))]
drollover = cerebro.rolloverdata(*dtemp, name='MyRoll', **kwargs)
cerebro.run()

rolloverdata has actually no name argument. It will look into the **kwargs to allow the behavior from the small snippet.

[backtrader]

Thanks - will give a try.

One slightly irrelevant question to this specific data feeding issue, please:
Would an existing position be rolled automatically (i.e. sold/bought for a long position) once the futures time series is rolled to the next contract?

[backtrader]

Would an existing position be rolled automatically (i.e. sold/bought for a long position) once the futures time series is rolled to the next contract?

That's an easy one: No. But not because of what you express in the question.

The cerebro.rolloverdata creates a data feed named MyRoll (in the example) and that's the trading instrument for the internal broker and this instrument trades continuously without interruptions. The broker has no notion of expirations.

There is a real data feed/broker (Visual Chart) for which there is support: The data feed can reference the continuous future, but takes an additional parameter to understand which is the current underlying asset to trade on.

[backtrader]

Understood - thanks for the swift response on this.

What I am trying to achieve here is to simulate the pnl of the actual futures trading where I am likely to hold multiple futures positions over multiple rollovers (i.e. holding period of 3-6 months). It is not the crucial part of the pnl attribution but I am just seeking to get the realistic aspect of the algo to certain degree.

What do you think it would be my best approach using backtrader? I don't have access to Visual Chart... if I understood correctly it's like a 3rd party broker like IB?

[backtrader]

VisualChart is a real data provider (with an associated broker mostly like TradeStation) with footprint in Europe. It was only meant as an example. It shouldn't apply to your case for simulation purposes.

There are 2 keys for long-term futures that span over several expirations:

• Commissions. Since there are no automatic roll-overs with the continuous feed, the commissions have to be accounted for ... manually. From a practical point of view this is for sure not a major

• Price difference: In some instances the price difference between the current future and the next is HUGE. This is not the regular case, but the EuroStoxx50 (well traded) happens to have this sometimes (2%-3% difference)

  Your code sample seems to indicate that you use raw futures data and not backadjusted data, which already flattens/reduces the gap.

If the price differences are small amongst the different assets, then the simulation shall be good. But if not, the results will not be accurate. Your system may even jump into buying/selling with the price difference spike.

At the moment the RollOverData data feed offers no direct way of knowing when the expiration has happened. There is information on which is the currently expired future, but any order issued would be evaluated against the new data.

Some thinking needs to go into this.

[backtrader]

Thanks for looking into this.

Yes - tricky issue once one wants to backtest something over a long time span in the futures spaces. I am happy with my workaround for now where I use the average adjusted futures time series to generate signals and use roll-adjusted futures time series separately to enter/exit and calculate pnl.

Like you mentioned, brokerage consideration is another factor but I generally know the average holding period of the trades and can assume 2-3 trips per trades manually.

[acon]

Bringing this back up because there was some relevant discussion earlier.

I've been learning backtrader, and have the futures rollover use case as well. Currently, I use my own home-rolled framework, and had to solve this. The basic problem is that futures roll needs to be handled carefully; as you mentioned previously, @backtrader, commissions and price difference can't be ignored in the real world. Unfortunately, for many frameworks, this inverts the data source → trade logic relationship, so is hard to implement. However, it's pretty important: basically every non-intraday futures strategy needs it.

The trade management side of a roll isn't too bad. Either you represent the roll as a new trade, or the same one and change details (changing the entry price to account for roll contract differences, account for commissions). The latter is typically more strategy-friendly—at least, it can be accomplished transparently.

There's three primary ways to handle the price difference:

• Always use backwards adjusted data.
  • Pros: Easiest to backtest on. Easy to obtain (from, for instance, Quandl). Latest prices are similar to spot prices. Strategies easily "see" the effects of the roll (consider a VIX strategy, with large natural time decay).
  • Cons: Every roll, the entire data set rotates out. Not conducive to live trading; typically frameworks ignore roll, and it becomes a manual step. The backtested prices are phantom—just the result of a calculation; no assurance that a strategy would have traded the same way.
• Handle data series transitions and rolls as they happen, and backwards adjust old data.
  • Pros: Represents what actually would have happened. Latest prices are similar to spot prices.
  • Cons: Real-time backwards adjusting means that old data changes (!!); this can affect internal strategy/indicator state, and has non-negligible performance overhead. In general, this means that strategies need to dump old state and recompute.
• Handle data series transitions and rolls as they happen, and forwards adjust new data (store an offset).
  • Pros: Fairly simple to implement, old data doesn't change, low performance impact. Strategies/indicators can be ignorant of what happened.
  • Cons: Non-exact representation to what actually happened. Latest prices are not similar to spot prices due to the offset, which could affect strategy decisions.

From my perspective, there's no clear solution—it's all tradeoffs. (1) seems to be the most straightforward solution for backtrader right now, but precludes real-time trading and reduces assurance that the strategy would have actually traded the same way.

I chose (2) for myself, since that yields the most accurate representation of what happened. The Cons for this, though, are pretty substantial: strategies need to have a method of reliably dumping state and rebuilding with the backwards adjusted data after a roll. Given that, however, it's fairly clean. Charts resemble backwards adjusted data that you'd get from a provider.

Of the "actively managed" solutions, (3) is fairly easy to implement, but distorts latest prices quite a bit. Most contracts would end up quite negative, which is a bit annoying to work with. The current price becomes a function of when you started. Simple things like calculating a contract size become very difficult.

Would love to start a discussion with other folks interested in solving this in backtrader. What's technically possible without re-working everything to accommodate this use case?

[ab_trader]

Thank you for the post! Currently I am looking on the testing couple ideas for american futures. The way that I think to go forward is to use back-adjusted prices and have continuous contacts. I've checked Quandl already and their continuous contracts are not suitable for me. I was thinking to chain data series based on max volume to match reality, so in the continuous contract I have prices only from the months with max volumes, but Quandl just chains contracts based on expiration months.

backtrader makes continuous contracts based on certain conditions (date and volume), but does not shift prices. As a result you will have gaps between contracts. So to make this shift ones need to re-write bt rollover method or just write a strategy which will make it and save prices into csv files. I am not a super-python guy, so I will probably go second way.

I don't think that adjusted prices in the past will harm the tests. I am going to use dollar value stats, no % stats. Also trades price delta will be the same as for real prices.