For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Backtesting with Futures data (Actual individual contract)



  • Dear All,

    I'm currently backtesting a Futures strategy that trades the entire curve (i.e. does not only trade front contract). It means that I need to work actual Futures contract rather than stitched futures contract (e.g. #1, #2, #3).
    What I do is to add every Futures contract as a data feed and then the strategy has code to retrieve the datafeeds (that corresponds to contract that have not expired) sorted by expiration date. One of the issue I have faced is that these individual contracts don't have all data for every date (let assume I perform a backtest over 10Y; it is easy to understand that Dec2019 contract had no data early 2010) and therefore the backtest is only performed with a subset of the contracts (i.e. Datafeed) and only from the date where the last datafeed added to Cerebro has data.

    I have temporary "resolved" this issue by reindexing my dataframes (and therefore adding nan date) before loading them to backtrader to ensure that all datafeeds have records from the start date till the end date I want. Is there by chance something more elegant than this I could do?

    Many thanks
    Regards,
    Lamp'


  • administrators



  • yes, for other strategies. but in this case, I need to be able to trade at different point of the forward curve (e.g. front contract and contract expiring in one year; a few weeks later front contract and contract expiring in 6 months). Plus in this case, it is actually useful to me to know which actual individual contract has been traded



  • If you load all contracts as separate data feeds into bt, than bt will build the date-time axis from the very first date of the very first contract to the last date of the last contract. If I remember correctly, this is how I did it some time ago.

    @lampalork said in Backtesting with Futures data (Actual individual contract):

    therefore the backtest is only performed with a subset of the contracts (i.e. Datafeed) and only from the date where the last datafeed added to Cerebro has data

    To make the backtest along the whole history put trading logic into both prenext() and next(). But accurately check if data available in the prenext() call.



  • Thanks, ab_trader and sorry for getting back so late. I'll double-check your claim and will update if it does not behave like you describe; for now, my trick works for it feels unnecessary.



  • Hi,
    I did as ab_trader suggested and can confirm that it works pretty well. I am actually loading a future as a chain in backtrader via a default_strategy that I inherit when I create a new strategy. So my code works on a specific future and I am able to trade the whole curve.
    Let me know if you have any questions.



  • thanks very much, guys. I finally took the time to look at this and it works. Furthermore solution is much simpler than expected. A simple override of prenext() does the job! hopefully, I manage to reduce the memory footprint of my backtest (20 years on a basket of 12 commodities futures with individual contracts; you can imagine how many Datafeed I have to handle!)

        def prenext(self):
            current_date = bt.utils.date.num2date(self.datetime[0])
            if current_date >= self.params.start_date:
                self.next()
    


  • If this is daily data that should work fine. I've had to do the same with an even bigger basket and I didn't have any issues.



  • @lampalork thank you very much for your sharing.I also explore how to backtest multi future contract in a basket. Are you sure it is Ok? when I read the ways to solve this question in this commuty, I find ,when you are in the prenext,if you call the self.next,your other contract that not in that trading day may pass through. In sipte of filtering can affect,but,maybe,you can't get the data in last year,I am not sure.



  • @lampalork I just run my code in prenext,don't use next, maybe,it is fine if you filter your data.

    ![4463f805-be55-4636-904a-61acb74b9ea4-image.png](/assets/uploads/files/1576564474483-4463f805-be55-4636-904a-61acb74b9ea4-image.png) code_text
    ```name	data_time	meta_time	open	close					
    meta	2010/1/4	2010/1/4	1	1					
    A1001.XDCE	2010/1/4	2010/1/4	4107	4108					
    A1003.XDCE	2010/1/4	2010/1/4	4080	4150					
    A1005.XDCE	2010/1/4	2010/1/4	4177	4165					
    A1007.XDCE	2010/1/4	2010/1/4	4202	4203					
    A1009.XDCE	2010/1/4	2010/1/4	4080	4057					
    A1011.XDCE	2010/1/4	2010/1/4	3970	4011					
    A1101.XDCE	2010/1/4	2010/1/4	4000	4004					
    A1103.XDCE	2010/1/4	2010/1/4	3989	4010					
    A1105.XDCE	2010/1/4	2010/1/4	4027	4027					
    A1107.XDCE	2011/7/14	2010/1/4	4237	4250	A1107.XDCE	2011/7/14	2010/1/6	4237	4250
    A1109.XDCE	2011/9/15	2010/1/4	4197	4197	A1109.XDCE	2011/9/15	2010/1/6	4197	4197
    A1111.XDCE	2011/11/14	2010/1/4	4016	4016	A1111.XDCE	2011/11/14	2010/1/6	4016	4016
    A1201.XDCE	2012/1/17	2010/1/4	4110	4100	A1201.XDCE	2012/1/17	2010/1/6	4110	4100
    A1203.XDCE	2012/3/14	2010/1/4	4180	4180	A1203.XDCE	2012/3/14	2010/1/6	4180	4180
    A1205.XDCE	2012/5/15	2010/1/4	4298	4298	A1205.XDCE	2012/5/15	2010/1/6	4298	4298
    A1207.XDCE	2012/7/13	2010/1/4	4450	4450	A1207.XDCE	2012/7/13	2010/1/6	4450	4450
    A1209.XDCE	2012/9/14	2010/1/4	4811	4811	A1209.XDCE	2012/9/14	2010/1/6	4811	4811
    A1211.XDCE	2012/11/14	2010/1/4	4511	4511	A1211.XDCE	2012/11/14	2010/1/6	4511	4511
    A1301.XDCE	2013/1/17	2010/1/4	4625	4625	A1301.XDCE	2013/1/17	2010/1/6	4625	4625
    A1303.XDCE	2013/3/14	2010/1/4	5193	5193	A1303.XDCE	2013/3/14	2010/1/6	5193	5193
    A1305.XDCE	2013/5/15	2010/1/4	4730	4720	A1305.XDCE	2013/5/15	2010/1/6	4730	4720
    A1307.XDCE	2013/7/12	2010/1/4	4674	4674	A1307.XDCE	2013/7/12	2010/1/6	4674	4674
    A1309.XDCE	2013/9/13	2010/1/4	4443	4443	A1309.XDCE	2013/9/13	2010/1/6	4443	4443
    A1311.XDCE	2013/11/14	2010/1/4	4445	4445	A1311.XDCE	2013/11/14	2010/1/6	4445	4445
    A1401.XDCE	2014/1/15	2010/1/4	4425	4425	A1401.XDCE	2014/1/15	2010/1/6	4425	4425
    A1403.XDCE	2014/3/14	2010/1/4	4891	4891	A1403.XDCE	2014/3/14	2010/1/6	4891	4891
    A1405.XDCE	2014/5/16	2010/1/4	4776	4776	A1405.XDCE	2014/5/16	2010/1/6	4776	4776
    A1407.XDCE	2014/7/14	2010/1/4	4573	4573	A1407.XDCE	2014/7/14	2010/1/6	4573	4573
    A1409.XDCE	2014/9/15	2010/1/4	4512	4512	A1409.XDCE	2014/9/15	2010/1/6	4512	4512
    A1411.XDCE	2014/11/14	2010/1/4	4515	4515	A1411.XDCE	2014/11/14	2010/1/6	4515	4515
    A1501.XDCE	2015/1/16	2010/1/4	4599	4599	A1501.XDCE	2015/1/16	2010/1/6	4599	4599
    A1503.XDCE	2015/3/13	2010/1/4	4076	4076	A1503.XDCE	2015/3/13	2010/1/6	4076	4076
    A1505.XDCE	2015/5/15	2010/1/4	4162	4162	A1505.XDCE	2015/5/15	2010/1/6	4162	4162
    A1507.XDCE	2015/7/14	2010/1/4	4081	4081	A1507.XDCE	2015/7/14	2010/1/6	4081	4081
    A1509.XDCE	2015/9/16	2010/1/4	4050	4050	A1509.XDCE	2015/9/16	2010/1/6	4050	4050
    A1511.XDCE	2015/11/13	2010/1/4	3846	3846	A1511.XDCE	2015/11/13	2010/1/6	3846	3846
    A1601.XDCE	2016/1/15	2010/1/4	3900	3900	A1601.XDCE	2016/1/15	2010/1/6	3900	3900
    A1603.XDCE	2016/3/14	2010/1/4	3377	3377	A1603.XDCE	2016/3/14	2010/1/6	3377	3377
    A1605.XDCE	2016/5/16	2010/1/4	3700	3700	A1605.XDCE	2016/5/16	2010/1/6	3700	3700
    A1607.XDCE	2016/7/14	2010/1/4	3900	3900	A1607.XDCE	2016/7/14	2010/1/6	3900	3900
    A1609.XDCE	2016/9/14	2010/1/4	3530	3620	A1609.XDCE	2016/9/14	2010/1/6	3530	3620
    A1611.XDCE	2016/11/14	2010/1/4	3760	3760	A1611.XDCE	2016/11/14	2010/1/6	3760	3760
    A1701.XDCE	2017/1/16	2010/1/4	4210	4200	A1701.XDCE	2017/1/16	2010/1/6	4210	4200
    A1703.XDCE	2017/3/14	2010/1/4	4000	4000	A1703.XDCE	2017/3/14	2010/1/6	4000	4000
    A1705.XDCE	2017/5/15	2010/1/4	3710	3713	A1705.XDCE	2017/5/15	2010/1/6	3710	3713
    A1707.XDCE	2017/7/14	2010/1/4	4004	4004	A1707.XDCE	2017/7/14	2010/1/6	4004	4004
    A1709.XDCE	2017/9/14	2010/1/4	3738	3738	A1709.XDCE	2017/9/14	2010/1/6	3738	3738
    A1711.XDCE	2017/11/14	2010/1/4	3511	3511	A1711.XDCE	2017/11/14	2010/1/6	3511	3511
    A1801.XDCE	2018/1/15	2010/1/4	3087	3148	A1801.XDCE	2018/1/15	2010/1/6	3087	3148
    A1803.XDCE	2018/3/14	2010/1/4	3298	3250	A1803.XDCE	2018/3/14	2010/1/6	3298	3250
    A1805.XDCE	2018/5/15	2010/1/4	3704	3703	A1805.XDCE	2018/5/15	2010/1/6	3704	3703
    A1807.XDCE	2018/7/13	2010/1/4	3565	3565	A1807.XDCE	2018/7/13	2010/1/6	3565	3565
    A1809.XDCE	2018/9/14	2010/1/4	3568	3595	A1809.XDCE	2018/9/14	2010/1/6	3568	3595
    A1811.XDCE	2018/11/14	2010/1/4	3430	3396	A1811.XDCE	2018/11/14	2010/1/6	3430	3396
    A1901.XDCE	2019/1/15	2010/1/4	3206	3206	A1901.XDCE	2019/1/15	2010/1/6	3206	3206
    A1903.XDCE	2019/3/14	2010/1/4	3277	3277	A1903.XDCE	2019/3/14	2010/1/6	3277	3277
    A1905.XDCE	2019/5/17	2010/1/4	3460	3460	A1905.XDCE	2019/5/17	2010/1/6	3460	3460
    A1907.XDCE	2019/7/12	2010/1/4	3380	3380	A1907.XDCE	2019/7/12	2010/1/6	3380	3380
    A1909.XDCE	2019/9/16	2010/1/4	3410	3410	A1909.XDCE	2019/9/16	2010/1/6	3410	3410
    A1911.XDCE	2019/11/14	2010/1/4	3276	3276	A1911.XDCE	2019/11/14	2010/1/6	3276	3276
    A2001.XDCE	2019/11/14	2010/1/4	3404	3403	A2001.XDCE	2019/11/14	2010/1/6	3404	3403
    A2003.XDCE	2019/11/14	2010/1/4	3439	3441	A2003.XDCE	2019/11/14	2010/1/6	3439	3441
    A2005.XDCE	2019/11/14	2010/1/4	3710	3699	A2005.XDCE	2019/11/14	2010/1/6	3710	3699
    A2007.XDCE	2019/11/14	2010/1/4	3763	3747	A2007.XDCE	2019/11/14	2010/1/6	3763	3747
    A2009.XDCE	2019/11/14	2010/1/4	3778	3769	A2009.XDCE	2019/11/14	2010/1/6	3778	3769


  • @Laurent-Michelizza @lampalork

    I try your way in my code,do you need filter the data? In my situation, there are some contracts pass by,which are not come in reality.

    581e3759-7f5c-4d28-8c6b-c779f9abe591-image.png code_text

    name	data_time	meta_time	open	close					
    meta	2010/1/4	2010/1/4	1	1	meta	2010/1/5	2010/1/5	1	1
    A1001.XDCE	2010/1/4	2010/1/4	4107	4108	A1001.XDCE	2010/1/5	2010/1/5	4107	4107
    A1003.XDCE	2010/1/4	2010/1/4	4080	4150	A1003.XDCE	2010/1/5	2010/1/5	4128	4128
    A1005.XDCE	2010/1/4	2010/1/4	4177	4165	A1005.XDCE	2010/1/5	2010/1/5	4172	4175
    A1007.XDCE	2010/1/4	2010/1/4	4202	4203	A1007.XDCE	2010/1/5	2010/1/5	4203	4217
    A1009.XDCE	2010/1/4	2010/1/4	4080	4057	A1009.XDCE	2010/1/5	2010/1/5	4067	4066
    A1011.XDCE	2010/1/4	2010/1/4	3970	4011	A1011.XDCE	2010/1/5	2010/1/5	4026	4028
    A1101.XDCE	2010/1/4	2010/1/4	4000	4004	A1101.XDCE	2010/1/5	2010/1/5	4018	4010
    A1103.XDCE	2010/1/4	2010/1/4	3989	4010	A1103.XDCE	2010/1/5	2010/1/5	4000	4016
    A1105.XDCE	2010/1/4	2010/1/4	4027	4027	A1105.XDCE	2010/1/5	2010/1/5	4027	4025
    A1107.XDCE	2011/7/14	2010/1/4	4237	4250	A1107.XDCE	2011/7/14	2010/1/5	4237	4250
    A1109.XDCE	2011/9/15	2010/1/4	4197	4197	A1109.XDCE	2011/9/15	2010/1/5	4197	4197
    A1111.XDCE	2011/11/14	2010/1/4	4016	4016	A1111.XDCE	2011/11/14	2010/1/5	4016	4016
    A1201.XDCE	2012/1/17	2010/1/4	4110	4100	A1201.XDCE	2012/1/17	2010/1/5	4110	4100
    A1203.XDCE	2012/3/14	2010/1/4	4180	4180	A1203.XDCE	2012/3/14	2010/1/5	4180	4180
    A1205.XDCE	2012/5/15	2010/1/4	4298	4298	A1205.XDCE	2012/5/15	2010/1/5	4298	4298
    A1207.XDCE	2012/7/13	2010/1/4	4450	4450	A1207.XDCE	2012/7/13	2010/1/5	4450	4450
    A1209.XDCE	2012/9/14	2010/1/4	4811	4811	A1209.XDCE	2012/9/14	2010/1/5	4811	4811
    A1211.XDCE	2012/11/14	2010/1/4	4511	4511	A1211.XDCE	2012/11/14	2010/1/5	4511	4511
    A1301.XDCE	2013/1/17	2010/1/4	4625	4625	A1301.XDCE	2013/1/17	2010/1/5	4625	4625
    A1303.XDCE	2013/3/14	2010/1/4	5193	5193	A1303.XDCE	2013/3/14	2010/1/5	5193	5193
    A1305.XDCE	2013/5/15	2010/1/4	4730	4720	A1305.XDCE	2013/5/15	2010/1/5	4730	4720
    A1307.XDCE	2013/7/12	2010/1/4	4674	4674	A1307.XDCE	2013/7/12	2010/1/5	4674	4674
    A1309.XDCE	2013/9/13	2010/1/4	4443	4443	A1309.XDCE	2013/9/13	2010/1/5	4443	4443
    A1311.XDCE	2013/11/14	2010/1/4	4445	4445	A1311.XDCE	2013/11/14	2010/1/5	4445	4445
    A1401.XDCE	2014/1/15	2010/1/4	4425	4425	A1401.XDCE	2014/1/15	2010/1/5	4425	4425
    A1403.XDCE	2014/3/14	2010/1/4	4891	4891	A1403.XDCE	2014/3/14	2010/1/5	4891	4891
    A1405.XDCE	2014/5/16	2010/1/4	4776	4776	A1405.XDCE	2014/5/16	2010/1/5	4776	4776
    A1407.XDCE	2014/7/14	2010/1/4	4573	4573	A1407.XDCE	2014/7/14	2010/1/5	4573	4573
    A1409.XDCE	2014/9/15	2010/1/4	4512	4512	A1409.XDCE	2014/9/15	2010/1/5	4512	4512
    A1411.XDCE	2014/11/14	2010/1/4	4515	4515	A1411.XDCE	2014/11/14	2010/1/5	4515	4515
    A1501.XDCE	2015/1/16	2010/1/4	4599	4599	A1501.XDCE	2015/1/16	2010/1/5	4599	4599
    A1503.XDCE	2015/3/13	2010/1/4	4076	4076	A1503.XDCE	2015/3/13	2010/1/5	4076	4076
    A1505.XDCE	2015/5/15	2010/1/4	4162	4162	A1505.XDCE	2015/5/15	2010/1/5	4162	4162
    A1507.XDCE	2015/7/14	2010/1/4	4081	4081	A1507.XDCE	2015/7/14	2010/1/5	4081	4081
    A1509.XDCE	2015/9/16	2010/1/4	4050	4050	A1509.XDCE	2015/9/16	2010/1/5	4050	4050
    A1511.XDCE	2015/11/13	2010/1/4	3846	3846	A1511.XDCE	2015/11/13	2010/1/5	3846	3846
    A1601.XDCE	2016/1/15	2010/1/4	3900	3900	A1601.XDCE	2016/1/15	2010/1/5	3900	3900
    A1603.XDCE	2016/3/14	2010/1/4	3377	3377	A1603.XDCE	2016/3/14	2010/1/5	3377	3377
    A1605.XDCE	2016/5/16	2010/1/4	3700	3700	A1605.XDCE	2016/5/16	2010/1/5	3700	3700
    A1607.XDCE	2016/7/14	2010/1/4	3900	3900	A1607.XDCE	2016/7/14	2010/1/5	3900	3900
    A1609.XDCE	2016/9/14	2010/1/4	3530	3620	A1609.XDCE	2016/9/14	2010/1/5	3530	3620
    A1611.XDCE	2016/11/14	2010/1/4	3760	3760	A1611.XDCE	2016/11/14	2010/1/5	3760	3760
    A1701.XDCE	2017/1/16	2010/1/4	4210	4200	A1701.XDCE	2017/1/16	2010/1/5	4210	4200
    A1703.XDCE	2017/3/14	2010/1/4	4000	4000	A1703.XDCE	2017/3/14	2010/1/5	4000	4000
    A1705.XDCE	2017/5/15	2010/1/4	3710	3713	A1705.XDCE	2017/5/15	2010/1/5	3710	3713
    A1707.XDCE	2017/7/14	2010/1/4	4004	4004	A1707.XDCE	2017/7/14	2010/1/5	4004	4004
    A1709.XDCE	2017/9/14	2010/1/4	3738	3738	A1709.XDCE	2017/9/14	2010/1/5	3738	3738
    A1711.XDCE	2017/11/14	2010/1/4	3511	3511	A1711.XDCE	2017/11/14	2010/1/5	3511	3511
    A1801.XDCE	2018/1/15	2010/1/4	3087	3148	A1801.XDCE	2018/1/15	2010/1/5	3087	3148
    A1803.XDCE	2018/3/14	2010/1/4	3298	3250	A1803.XDCE	2018/3/14	2010/1/5	3298	3250
    A1805.XDCE	2018/5/15	2010/1/4	3704	3703	A1805.XDCE	2018/5/15	2010/1/5	3704	3703
    A1807.XDCE	2018/7/13	2010/1/4	3565	3565	A1807.XDCE	2018/7/13	2010/1/5	3565	3565
    A1809.XDCE	2018/9/14	2010/1/4	3568	3595	A1809.XDCE	2018/9/14	2010/1/5	3568	3595
    A1811.XDCE	2018/11/14	2010/1/4	3430	3396	A1811.XDCE	2018/11/14	2010/1/5	3430	3396
    A1901.XDCE	2019/1/15	2010/1/4	3206	3206	A1901.XDCE	2019/1/15	2010/1/5	3206	3206
    A1903.XDCE	2019/3/14	2010/1/4	3277	3277	A1903.XDCE	2019/3/14	2010/1/5	3277	3277
    A1905.XDCE	2019/5/17	2010/1/4	3460	3460	A1905.XDCE	2019/5/17	2010/1/5	3460	3460
    A1907.XDCE	2019/7/12	2010/1/4	3380	3380	A1907.XDCE	2019/7/12	2010/1/5	3380	3380
    A1909.XDCE	2019/9/16	2010/1/4	3410	3410	A1909.XDCE	2019/9/16	2010/1/5	3410	3410
    A1911.XDCE	2019/11/14	2010/1/4	3276	3276	A1911.XDCE	2019/11/14	2010/1/5	3276	3276
    A2001.XDCE	2019/11/14	2010/1/4	3404	3403	A2001.XDCE	2019/11/14	2010/1/5	3404	3403
    A2003.XDCE	2019/11/14	2010/1/4	3439	3441	A2003.XDCE	2019/11/14	2010/1/5	3439	3441
    A2005.XDCE	2019/11/14	2010/1/4	3710	3699	A2005.XDCE	2019/11/14	2010/1/5	3710	3699
    A2007.XDCE	2019/11/14	2010/1/4	3763	3747	A2007.XDCE	2019/11/14	2010/1/5	3763	3747
    A2009.XDCE	2019/11/14	2010/1/4	3778	3769	A2009.XDCE	2019/11/14	2010/1/5	3778	3769


  • Hi @tianjixuetu, I have some logic in self.next() to chain these contracts. In other words, to know at any point of time what is the front contract (n=1) and all the other n contracts in the future.



  • @lampalork do you meet something strange about your position? for example,something your position cannot close successfully if you use many actual individual contract?
    total_position_value.png

    according my logic,during some trading days,my position will be closed for a period,just like 2010-2014,but unfortunately,after 2014,I got a steady positon?

    I close position use this code:

    for data in self.datas:
          if data in self.broker.positions:
               position=self.getposition(data)
               if position.size!=0:
                   self.close(data)
    

    do you meet this similar situation?

    Thank you very much.



  • @tianjixuetu hi, as always, if you expect people to help you, you have to prepare a mini reproduceable example with code and data that highlights the problem you are facing. You can't expect people to try to guess what you are doing. This is not intended to be mean but rather to help you get the support you need



  • @tianjixuetu You need to make sure to close the contracts that are expiring. In this case it seems a contract might have expired with a position.
    I have a table which gives me LTD and FND (if applicable) for all futures.



  • @Laurent-Michelizza Thank you very much! when I log the position,I also find this question and use almost your way to solve it.
    But in the future,I guess,we do a auto-close function when the future expires, it is more easy to use. After this four times reading the source code,I maybe try this function.


Log in to reply
 

});