Backtesting with Futures data (Actual individual contract)
-
Dear All,
I'm currently backtesting a Futures strategy that trades the entire curve (i.e. does not only trade front contract). It means that I need to work actual Futures contract rather than stitched futures contract (e.g. #1, #2, #3).
What I do is to add every Futures contract as a data feed and then the strategy has code to retrieve the datafeeds (that corresponds to contract that have not expired) sorted by expiration date. One of the issue I have faced is that these individual contracts don't have all data for every date (let assume I perform a backtest over 10Y; it is easy to understand that Dec2019 contract had no data early 2010) and therefore the backtest is only performed with a subset of the contracts (i.e. Datafeed) and only from the date where the last datafeed added to Cerebro has data.I have temporary "resolved" this issue by reindexing my dataframes (and therefore adding nan date) before loading them to backtrader to ensure that all datafeeds have records from the start date till the end date I want. Is there by chance something more elegant than this I could do?
Many thanks
Regards,
Lamp' -
Have you tried the rollover functionality?
Data Feeds - Rollover - https://www.backtrader.com/docu/data-rollover/rolling-futures-over/
-
yes, for other strategies. but in this case, I need to be able to trade at different point of the forward curve (e.g. front contract and contract expiring in one year; a few weeks later front contract and contract expiring in 6 months). Plus in this case, it is actually useful to me to know which actual individual contract has been traded
-
If you load all contracts as separate data feeds into
bt
, thanbt
will build the date-time axis from the very first date of the very first contract to the last date of the last contract. If I remember correctly, this is how I did it some time ago.@lampalork said in Backtesting with Futures data (Actual individual contract):
therefore the backtest is only performed with a subset of the contracts (i.e. Datafeed) and only from the date where the last datafeed added to Cerebro has data
To make the backtest along the whole history put trading logic into both
prenext()
andnext()
. But accurately check if data available in theprenext()
call. -
Thanks, ab_trader and sorry for getting back so late. I'll double-check your claim and will update if it does not behave like you describe; for now, my trick works for it feels unnecessary.
-
Hi,
I did as ab_trader suggested and can confirm that it works pretty well. I am actually loading a future as a chain in backtrader via a default_strategy that I inherit when I create a new strategy. So my code works on a specific future and I am able to trade the whole curve.
Let me know if you have any questions. -
thanks very much, guys. I finally took the time to look at this and it works. Furthermore solution is much simpler than expected. A simple override of prenext() does the job! hopefully, I manage to reduce the memory footprint of my backtest (20 years on a basket of 12 commodities futures with individual contracts; you can imagine how many Datafeed I have to handle!)
def prenext(self): current_date = bt.utils.date.num2date(self.datetime[0]) if current_date >= self.params.start_date: self.next()
-
If this is daily data that should work fine. I've had to do the same with an even bigger basket and I didn't have any issues.
-
@lampalork thank you very much for your sharing.I also explore how to backtest multi future contract in a basket. Are you sure it is Ok? when I read the ways to solve this question in this commuty, I find ,when you are in the prenext,if you call the self.next,your other contract that not in that trading day may pass through. In sipte of filtering can affect,but,maybe,you can't get the data in last year,I am not sure.
-
@lampalork I just run my code in prenext,don't use next, maybe,it is fine if you filter your data.
 code_text ```name data_time meta_time open close meta 2010/1/4 2010/1/4 1 1 A1001.XDCE 2010/1/4 2010/1/4 4107 4108 A1003.XDCE 2010/1/4 2010/1/4 4080 4150 A1005.XDCE 2010/1/4 2010/1/4 4177 4165 A1007.XDCE 2010/1/4 2010/1/4 4202 4203 A1009.XDCE 2010/1/4 2010/1/4 4080 4057 A1011.XDCE 2010/1/4 2010/1/4 3970 4011 A1101.XDCE 2010/1/4 2010/1/4 4000 4004 A1103.XDCE 2010/1/4 2010/1/4 3989 4010 A1105.XDCE 2010/1/4 2010/1/4 4027 4027 A1107.XDCE 2011/7/14 2010/1/4 4237 4250 A1107.XDCE 2011/7/14 2010/1/6 4237 4250 A1109.XDCE 2011/9/15 2010/1/4 4197 4197 A1109.XDCE 2011/9/15 2010/1/6 4197 4197 A1111.XDCE 2011/11/14 2010/1/4 4016 4016 A1111.XDCE 2011/11/14 2010/1/6 4016 4016 A1201.XDCE 2012/1/17 2010/1/4 4110 4100 A1201.XDCE 2012/1/17 2010/1/6 4110 4100 A1203.XDCE 2012/3/14 2010/1/4 4180 4180 A1203.XDCE 2012/3/14 2010/1/6 4180 4180 A1205.XDCE 2012/5/15 2010/1/4 4298 4298 A1205.XDCE 2012/5/15 2010/1/6 4298 4298 A1207.XDCE 2012/7/13 2010/1/4 4450 4450 A1207.XDCE 2012/7/13 2010/1/6 4450 4450 A1209.XDCE 2012/9/14 2010/1/4 4811 4811 A1209.XDCE 2012/9/14 2010/1/6 4811 4811 A1211.XDCE 2012/11/14 2010/1/4 4511 4511 A1211.XDCE 2012/11/14 2010/1/6 4511 4511 A1301.XDCE 2013/1/17 2010/1/4 4625 4625 A1301.XDCE 2013/1/17 2010/1/6 4625 4625 A1303.XDCE 2013/3/14 2010/1/4 5193 5193 A1303.XDCE 2013/3/14 2010/1/6 5193 5193 A1305.XDCE 2013/5/15 2010/1/4 4730 4720 A1305.XDCE 2013/5/15 2010/1/6 4730 4720 A1307.XDCE 2013/7/12 2010/1/4 4674 4674 A1307.XDCE 2013/7/12 2010/1/6 4674 4674 A1309.XDCE 2013/9/13 2010/1/4 4443 4443 A1309.XDCE 2013/9/13 2010/1/6 4443 4443 A1311.XDCE 2013/11/14 2010/1/4 4445 4445 A1311.XDCE 2013/11/14 2010/1/6 4445 4445 A1401.XDCE 2014/1/15 2010/1/4 4425 4425 A1401.XDCE 2014/1/15 2010/1/6 4425 4425 A1403.XDCE 2014/3/14 2010/1/4 4891 4891 A1403.XDCE 2014/3/14 2010/1/6 4891 4891 A1405.XDCE 2014/5/16 2010/1/4 4776 4776 A1405.XDCE 2014/5/16 2010/1/6 4776 4776 A1407.XDCE 2014/7/14 2010/1/4 4573 4573 A1407.XDCE 2014/7/14 2010/1/6 4573 4573 A1409.XDCE 2014/9/15 2010/1/4 4512 4512 A1409.XDCE 2014/9/15 2010/1/6 4512 4512 A1411.XDCE 2014/11/14 2010/1/4 4515 4515 A1411.XDCE 2014/11/14 2010/1/6 4515 4515 A1501.XDCE 2015/1/16 2010/1/4 4599 4599 A1501.XDCE 2015/1/16 2010/1/6 4599 4599 A1503.XDCE 2015/3/13 2010/1/4 4076 4076 A1503.XDCE 2015/3/13 2010/1/6 4076 4076 A1505.XDCE 2015/5/15 2010/1/4 4162 4162 A1505.XDCE 2015/5/15 2010/1/6 4162 4162 A1507.XDCE 2015/7/14 2010/1/4 4081 4081 A1507.XDCE 2015/7/14 2010/1/6 4081 4081 A1509.XDCE 2015/9/16 2010/1/4 4050 4050 A1509.XDCE 2015/9/16 2010/1/6 4050 4050 A1511.XDCE 2015/11/13 2010/1/4 3846 3846 A1511.XDCE 2015/11/13 2010/1/6 3846 3846 A1601.XDCE 2016/1/15 2010/1/4 3900 3900 A1601.XDCE 2016/1/15 2010/1/6 3900 3900 A1603.XDCE 2016/3/14 2010/1/4 3377 3377 A1603.XDCE 2016/3/14 2010/1/6 3377 3377 A1605.XDCE 2016/5/16 2010/1/4 3700 3700 A1605.XDCE 2016/5/16 2010/1/6 3700 3700 A1607.XDCE 2016/7/14 2010/1/4 3900 3900 A1607.XDCE 2016/7/14 2010/1/6 3900 3900 A1609.XDCE 2016/9/14 2010/1/4 3530 3620 A1609.XDCE 2016/9/14 2010/1/6 3530 3620 A1611.XDCE 2016/11/14 2010/1/4 3760 3760 A1611.XDCE 2016/11/14 2010/1/6 3760 3760 A1701.XDCE 2017/1/16 2010/1/4 4210 4200 A1701.XDCE 2017/1/16 2010/1/6 4210 4200 A1703.XDCE 2017/3/14 2010/1/4 4000 4000 A1703.XDCE 2017/3/14 2010/1/6 4000 4000 A1705.XDCE 2017/5/15 2010/1/4 3710 3713 A1705.XDCE 2017/5/15 2010/1/6 3710 3713 A1707.XDCE 2017/7/14 2010/1/4 4004 4004 A1707.XDCE 2017/7/14 2010/1/6 4004 4004 A1709.XDCE 2017/9/14 2010/1/4 3738 3738 A1709.XDCE 2017/9/14 2010/1/6 3738 3738 A1711.XDCE 2017/11/14 2010/1/4 3511 3511 A1711.XDCE 2017/11/14 2010/1/6 3511 3511 A1801.XDCE 2018/1/15 2010/1/4 3087 3148 A1801.XDCE 2018/1/15 2010/1/6 3087 3148 A1803.XDCE 2018/3/14 2010/1/4 3298 3250 A1803.XDCE 2018/3/14 2010/1/6 3298 3250 A1805.XDCE 2018/5/15 2010/1/4 3704 3703 A1805.XDCE 2018/5/15 2010/1/6 3704 3703 A1807.XDCE 2018/7/13 2010/1/4 3565 3565 A1807.XDCE 2018/7/13 2010/1/6 3565 3565 A1809.XDCE 2018/9/14 2010/1/4 3568 3595 A1809.XDCE 2018/9/14 2010/1/6 3568 3595 A1811.XDCE 2018/11/14 2010/1/4 3430 3396 A1811.XDCE 2018/11/14 2010/1/6 3430 3396 A1901.XDCE 2019/1/15 2010/1/4 3206 3206 A1901.XDCE 2019/1/15 2010/1/6 3206 3206 A1903.XDCE 2019/3/14 2010/1/4 3277 3277 A1903.XDCE 2019/3/14 2010/1/6 3277 3277 A1905.XDCE 2019/5/17 2010/1/4 3460 3460 A1905.XDCE 2019/5/17 2010/1/6 3460 3460 A1907.XDCE 2019/7/12 2010/1/4 3380 3380 A1907.XDCE 2019/7/12 2010/1/6 3380 3380 A1909.XDCE 2019/9/16 2010/1/4 3410 3410 A1909.XDCE 2019/9/16 2010/1/6 3410 3410 A1911.XDCE 2019/11/14 2010/1/4 3276 3276 A1911.XDCE 2019/11/14 2010/1/6 3276 3276 A2001.XDCE 2019/11/14 2010/1/4 3404 3403 A2001.XDCE 2019/11/14 2010/1/6 3404 3403 A2003.XDCE 2019/11/14 2010/1/4 3439 3441 A2003.XDCE 2019/11/14 2010/1/6 3439 3441 A2005.XDCE 2019/11/14 2010/1/4 3710 3699 A2005.XDCE 2019/11/14 2010/1/6 3710 3699 A2007.XDCE 2019/11/14 2010/1/4 3763 3747 A2007.XDCE 2019/11/14 2010/1/6 3763 3747 A2009.XDCE 2019/11/14 2010/1/4 3778 3769 A2009.XDCE 2019/11/14 2010/1/6 3778 3769
-
@Laurent-Michelizza @lampalork
I try your way in my code,do you need filter the data? In my situation, there are some contracts pass by,which are not come in reality.
code_text
name data_time meta_time open close meta 2010/1/4 2010/1/4 1 1 meta 2010/1/5 2010/1/5 1 1 A1001.XDCE 2010/1/4 2010/1/4 4107 4108 A1001.XDCE 2010/1/5 2010/1/5 4107 4107 A1003.XDCE 2010/1/4 2010/1/4 4080 4150 A1003.XDCE 2010/1/5 2010/1/5 4128 4128 A1005.XDCE 2010/1/4 2010/1/4 4177 4165 A1005.XDCE 2010/1/5 2010/1/5 4172 4175 A1007.XDCE 2010/1/4 2010/1/4 4202 4203 A1007.XDCE 2010/1/5 2010/1/5 4203 4217 A1009.XDCE 2010/1/4 2010/1/4 4080 4057 A1009.XDCE 2010/1/5 2010/1/5 4067 4066 A1011.XDCE 2010/1/4 2010/1/4 3970 4011 A1011.XDCE 2010/1/5 2010/1/5 4026 4028 A1101.XDCE 2010/1/4 2010/1/4 4000 4004 A1101.XDCE 2010/1/5 2010/1/5 4018 4010 A1103.XDCE 2010/1/4 2010/1/4 3989 4010 A1103.XDCE 2010/1/5 2010/1/5 4000 4016 A1105.XDCE 2010/1/4 2010/1/4 4027 4027 A1105.XDCE 2010/1/5 2010/1/5 4027 4025 A1107.XDCE 2011/7/14 2010/1/4 4237 4250 A1107.XDCE 2011/7/14 2010/1/5 4237 4250 A1109.XDCE 2011/9/15 2010/1/4 4197 4197 A1109.XDCE 2011/9/15 2010/1/5 4197 4197 A1111.XDCE 2011/11/14 2010/1/4 4016 4016 A1111.XDCE 2011/11/14 2010/1/5 4016 4016 A1201.XDCE 2012/1/17 2010/1/4 4110 4100 A1201.XDCE 2012/1/17 2010/1/5 4110 4100 A1203.XDCE 2012/3/14 2010/1/4 4180 4180 A1203.XDCE 2012/3/14 2010/1/5 4180 4180 A1205.XDCE 2012/5/15 2010/1/4 4298 4298 A1205.XDCE 2012/5/15 2010/1/5 4298 4298 A1207.XDCE 2012/7/13 2010/1/4 4450 4450 A1207.XDCE 2012/7/13 2010/1/5 4450 4450 A1209.XDCE 2012/9/14 2010/1/4 4811 4811 A1209.XDCE 2012/9/14 2010/1/5 4811 4811 A1211.XDCE 2012/11/14 2010/1/4 4511 4511 A1211.XDCE 2012/11/14 2010/1/5 4511 4511 A1301.XDCE 2013/1/17 2010/1/4 4625 4625 A1301.XDCE 2013/1/17 2010/1/5 4625 4625 A1303.XDCE 2013/3/14 2010/1/4 5193 5193 A1303.XDCE 2013/3/14 2010/1/5 5193 5193 A1305.XDCE 2013/5/15 2010/1/4 4730 4720 A1305.XDCE 2013/5/15 2010/1/5 4730 4720 A1307.XDCE 2013/7/12 2010/1/4 4674 4674 A1307.XDCE 2013/7/12 2010/1/5 4674 4674 A1309.XDCE 2013/9/13 2010/1/4 4443 4443 A1309.XDCE 2013/9/13 2010/1/5 4443 4443 A1311.XDCE 2013/11/14 2010/1/4 4445 4445 A1311.XDCE 2013/11/14 2010/1/5 4445 4445 A1401.XDCE 2014/1/15 2010/1/4 4425 4425 A1401.XDCE 2014/1/15 2010/1/5 4425 4425 A1403.XDCE 2014/3/14 2010/1/4 4891 4891 A1403.XDCE 2014/3/14 2010/1/5 4891 4891 A1405.XDCE 2014/5/16 2010/1/4 4776 4776 A1405.XDCE 2014/5/16 2010/1/5 4776 4776 A1407.XDCE 2014/7/14 2010/1/4 4573 4573 A1407.XDCE 2014/7/14 2010/1/5 4573 4573 A1409.XDCE 2014/9/15 2010/1/4 4512 4512 A1409.XDCE 2014/9/15 2010/1/5 4512 4512 A1411.XDCE 2014/11/14 2010/1/4 4515 4515 A1411.XDCE 2014/11/14 2010/1/5 4515 4515 A1501.XDCE 2015/1/16 2010/1/4 4599 4599 A1501.XDCE 2015/1/16 2010/1/5 4599 4599 A1503.XDCE 2015/3/13 2010/1/4 4076 4076 A1503.XDCE 2015/3/13 2010/1/5 4076 4076 A1505.XDCE 2015/5/15 2010/1/4 4162 4162 A1505.XDCE 2015/5/15 2010/1/5 4162 4162 A1507.XDCE 2015/7/14 2010/1/4 4081 4081 A1507.XDCE 2015/7/14 2010/1/5 4081 4081 A1509.XDCE 2015/9/16 2010/1/4 4050 4050 A1509.XDCE 2015/9/16 2010/1/5 4050 4050 A1511.XDCE 2015/11/13 2010/1/4 3846 3846 A1511.XDCE 2015/11/13 2010/1/5 3846 3846 A1601.XDCE 2016/1/15 2010/1/4 3900 3900 A1601.XDCE 2016/1/15 2010/1/5 3900 3900 A1603.XDCE 2016/3/14 2010/1/4 3377 3377 A1603.XDCE 2016/3/14 2010/1/5 3377 3377 A1605.XDCE 2016/5/16 2010/1/4 3700 3700 A1605.XDCE 2016/5/16 2010/1/5 3700 3700 A1607.XDCE 2016/7/14 2010/1/4 3900 3900 A1607.XDCE 2016/7/14 2010/1/5 3900 3900 A1609.XDCE 2016/9/14 2010/1/4 3530 3620 A1609.XDCE 2016/9/14 2010/1/5 3530 3620 A1611.XDCE 2016/11/14 2010/1/4 3760 3760 A1611.XDCE 2016/11/14 2010/1/5 3760 3760 A1701.XDCE 2017/1/16 2010/1/4 4210 4200 A1701.XDCE 2017/1/16 2010/1/5 4210 4200 A1703.XDCE 2017/3/14 2010/1/4 4000 4000 A1703.XDCE 2017/3/14 2010/1/5 4000 4000 A1705.XDCE 2017/5/15 2010/1/4 3710 3713 A1705.XDCE 2017/5/15 2010/1/5 3710 3713 A1707.XDCE 2017/7/14 2010/1/4 4004 4004 A1707.XDCE 2017/7/14 2010/1/5 4004 4004 A1709.XDCE 2017/9/14 2010/1/4 3738 3738 A1709.XDCE 2017/9/14 2010/1/5 3738 3738 A1711.XDCE 2017/11/14 2010/1/4 3511 3511 A1711.XDCE 2017/11/14 2010/1/5 3511 3511 A1801.XDCE 2018/1/15 2010/1/4 3087 3148 A1801.XDCE 2018/1/15 2010/1/5 3087 3148 A1803.XDCE 2018/3/14 2010/1/4 3298 3250 A1803.XDCE 2018/3/14 2010/1/5 3298 3250 A1805.XDCE 2018/5/15 2010/1/4 3704 3703 A1805.XDCE 2018/5/15 2010/1/5 3704 3703 A1807.XDCE 2018/7/13 2010/1/4 3565 3565 A1807.XDCE 2018/7/13 2010/1/5 3565 3565 A1809.XDCE 2018/9/14 2010/1/4 3568 3595 A1809.XDCE 2018/9/14 2010/1/5 3568 3595 A1811.XDCE 2018/11/14 2010/1/4 3430 3396 A1811.XDCE 2018/11/14 2010/1/5 3430 3396 A1901.XDCE 2019/1/15 2010/1/4 3206 3206 A1901.XDCE 2019/1/15 2010/1/5 3206 3206 A1903.XDCE 2019/3/14 2010/1/4 3277 3277 A1903.XDCE 2019/3/14 2010/1/5 3277 3277 A1905.XDCE 2019/5/17 2010/1/4 3460 3460 A1905.XDCE 2019/5/17 2010/1/5 3460 3460 A1907.XDCE 2019/7/12 2010/1/4 3380 3380 A1907.XDCE 2019/7/12 2010/1/5 3380 3380 A1909.XDCE 2019/9/16 2010/1/4 3410 3410 A1909.XDCE 2019/9/16 2010/1/5 3410 3410 A1911.XDCE 2019/11/14 2010/1/4 3276 3276 A1911.XDCE 2019/11/14 2010/1/5 3276 3276 A2001.XDCE 2019/11/14 2010/1/4 3404 3403 A2001.XDCE 2019/11/14 2010/1/5 3404 3403 A2003.XDCE 2019/11/14 2010/1/4 3439 3441 A2003.XDCE 2019/11/14 2010/1/5 3439 3441 A2005.XDCE 2019/11/14 2010/1/4 3710 3699 A2005.XDCE 2019/11/14 2010/1/5 3710 3699 A2007.XDCE 2019/11/14 2010/1/4 3763 3747 A2007.XDCE 2019/11/14 2010/1/5 3763 3747 A2009.XDCE 2019/11/14 2010/1/4 3778 3769 A2009.XDCE 2019/11/14 2010/1/5 3778 3769
-
Hi @tianjixuetu, I have some logic in self.next() to chain these contracts. In other words, to know at any point of time what is the front contract (n=1) and all the other n contracts in the future.
-
@lampalork do you meet something strange about your position? for example,something your position cannot close successfully if you use many actual individual contract?
according my logic,during some trading days,my position will be closed for a period,just like 2010-2014,but unfortunately,after 2014,I got a steady positon?
I close position use this code:
for data in self.datas: if data in self.broker.positions: position=self.getposition(data) if position.size!=0: self.close(data)
do you meet this similar situation?
Thank you very much.
-
@tianjixuetu hi, as always, if you expect people to help you, you have to prepare a mini reproduceable example with code and data that highlights the problem you are facing. You can't expect people to try to guess what you are doing. This is not intended to be mean but rather to help you get the support you need
-
@tianjixuetu You need to make sure to close the contracts that are expiring. In this case it seems a contract might have expired with a position.
I have a table which gives me LTD and FND (if applicable) for all futures. -
@Laurent-Michelizza Thank you very much! when I log the position,I also find this question and use almost your way to solve it.
But in the future,I guess,we do a auto-close function when the future expires, it is more easy to use. After this four times reading the source code,I maybe try this function.