For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See:

Suggestions on making use of real time futures data?

  • My strategy trades futures ETNs, instead of futures. However it makes use of the futures term structure to generate trading signals. The usage of the signal is rather simplistic, like greater or less than a threshold or so.

    For backtesting I simply added the signal as an extra field to the data feed. For live trading, I use IB, and I have subscriptions for the futures data from it. But I'm not sure how to add it to the strategy.

    Some options I can think of:

    • Fetch the future data outside of backtrader and generate the trading signals there, and then maybe save them in a Redis store, and let backtrader access it;
    • Override the IBData class to fetch futures data, do the calculation, and incorporate an extra field of data;
    • Use another completely different trading platform.

    Which way is the least overwhelming? What do you guys think? Any idea is appreciated.

  • This actually seems straightforward (unless something is not being understood from the post):

    • Data feed 1 is the trading target
    • Data feed 2 is the trading signal generator

    As such it would seem that the obvious approach is:

    • Add Data feed 1
    • Add Data feed 2
    • Create the needed indicators for trading on Data Feed 2
    • When the signals say so: buy/sell on Data Feed 1

    If the trading signals are pre-generated, the same would almos apply:

    1. Add Data feed 1
    2. Add Data feed 2
    3. Create a fake indicator that simply uses the input in Data feed 2 to trigger a signal
    4. When the signals say so: buy/sell on Data Feed 1

    You may of course fuse steps 2. and 3. if the signals aren't a continuous data stream and you can simply load them as an array directly into the indicator.

  • @Paska-Houso Thank you for your suggestion. Normally, it would work like you said.

    The difficulty is that the futures are not a single, perpetual data stream, but rather individual ones with different settlement dates (and different days starting with which the data is available). It's similar to this post. I can't add them to backtrader as-is.

    But since I trade only futures ETNs, which are perpetual contracts, I can get away with it, as long as I somehow manage to feed the data in.

  • Maybe a fourth option:

    • Add each and every individual futures data as a data feed.

    In which case there will be a great lot of them, and I don't know how the backfilling will play out with their different timings?

  • @yangnw I believe IB can give you continuous futures data. At least I can see it in TWS along with actual future contract data, not sure if you can access it via API.

  • @yangnw said in Suggestions on making use of real time futures data?:

    The difficulty is that the futures are not a single, perpetual data stream, but rather individual ones with different settlement dates (and different days starting with which the data is available). It's similar to this post. I can't add them to backtrader as-is.

    The rolloverdata mentioned in the post should then be the solution. There is even an entire chapter in the docs dedicated to it, giving you fine control over the switch over between futures.

    What's missing then?

  • @ab_trader @Paska-Houso

    Thanks, first!

    The futures term structure consists of the prices of multiple futures contracts at the same time, instead of designating any one of them like a continuous futures does.

    However, now that you reminded me, I can probably rewrite the ReplayData class to carry out the calculation there. Thank you for the pointers!

    I'll see how I can share the results when I'm done with it.

  • @ab_trader @Paska-Houso

    No, rolloverdata cannot be rewritten to access multiple contracts at the same time. The way it's implemented, the "new" contract simply skips expired data. The only option with backtrader is to add each contract as a separate data feed.

    this post:

    rolloverdata assume (naively) that the data feeds passed to them are already in order and nowhere else in the system. The latter being what would prevent buying from different targets.
    An approach you may consider:

    • add each contract as a separate data feed,
    • manually manage index of the active contract,
    • override _getminperstatus() to hack the minimum period requirement.

    Apparently those who want to trade continuous futures will also have to add contracts separately.

    It's shouldn't be too hard, I presume. Need to override _getminperstatus() to just respect the minimum period of the first data only.

  • @yangnw I confused a bit. Do you use multiple active contracts at the same time to generate the trade signals? In this case I think they all need to be loaded to backtrader for live trading.

  • Experiment:

    • as stated by @backtrader, next() is only called after the last contract starts.
    • Each later data series has an abrupt start from 0 and an abrupt end from 0. like this: 0_1511238325585_Unknown.png
    • strategy.datas[0].datetime.datetime() stops updating after the first data feed stops.

    I can imagine some additional problems in live trading:

    • at backfilling start, some latest contracts may not have any data at all, for that particular time.
    • May not be possible to add contracts in the far future, such that their symbols are not available at all.

    @ab_trader Are you working on trading futures? I saw you in the other post.

  • @yangnw

    yes, I am slowly moving to futures trading. But I am going to use quite easy common approach:

    • signals will be generated based on continuous contract prices (historical)
    • trading will be done based on active most liquid contract

    The only reason why I can't use bt directly is that rollover function in bt doesn't shift prices when I move from one active contract to another. This stops me from implementing trading ideas, first I need to develop approach to generate continuous contract.

    Some thoughts on your post -

    • next() is called then all data feeds and indicators delivered, but bt has prenext() which is called every bar before 1st next() call. You may want to try to use it to process previous contracts.
    • to avoid these abrupt start and finish you may want to jump from one contract to another say 5 days before stop of real trading of contract. IB, for example, makes this automatically, you will not be able to trade november contract at october 30.

  • @ab_trader you might be able to take some inspiration from QtPyLib:
    that is a function to create a continuous future. Am not a fan of QtPyLib as there is no active community and it has too many dependencies for my liking. Hence I am looking at backtrader too. Having similar thoughts whether i can/should use backtrader for live trading. Am curious to hear what you decide ab_trader.

  • @ab_trader

    Lucky for you, IB has a continuous futures, available in API as real time bars.

    It's back-adjusted, so keep that in mind when generating signals.

    It should be quite simple to adjust ibdata to subscribe to it.

  • @yangnw

    unfortunately IB continuous contracts done in different way. Instead of price shift they use price ratio to eliminate gaps. Also looks like they use all contracts thru the year to build the continuous contract. Since only most liquid contracts are traded and continuous contract should be developed using these contracts only.

  • @steffen

    Thanks for the link. I was thinking to create simple strategy in bt which will go thru all of the contracts and return me continuous contract data feed built by my rules.

    Since I am using daily bars I am not so concerned about live trading right now. I is beneficial, but not mandatory.

Log in to reply