handle missing data

hobart-liu

Many thanks to author, this is a great tool that really release me from writing my own backtest framework!

I am new to this tool. I am not sure whether this is stupid question, but I have been searching for a while, and finally decide come here for help.

My problem is, I have two data feeds, one of them doesn't have trade data during, say 2015-11-02 ~ 2015-11-11. So when I try to run my strategy from 2015-11-1 to 2015-12-1, the system only feed my strategy with data start from 2015-11-12. I guess this is because one of my dataset has missing value.

I know we can use forward fill to fill the missing value, but my neural network will probably not able to handle this way of filling. I shall give a clear signal to my strategy that between 2015-11-02 and 2015-11-11, the data is missing, so that I can skip this stock in my NN.

What I expected is, the system shall feed my strategy from 2015-11-1, because one stock do has trading data...
Thanks in advance. (If needed, I can post my code and result. )

ab_trader

Put trading logic in the prenext() function as well as in the next(). This should help.

hobart-liu

@ab_trader Does that mean, I shall use pretext() instead of next() in my strategy?

ab_trader

@hobart-liu nope, prenext() is called before next(). You need to call trading logic in both functions. But in the prenext() you'll need to check the availability of data.

Backtrader goes thru all datafeeds bar by bar. Until all datafeeds have values and all indicators have values it calls prenext(). After it calls next().

spyamine

but how can we disable this behaviour. if the data is available run if not wait until the data is available.

Also how can we handle survivor ship bias. since there is stock that will come to live in a period then other that will die in the meantime.

backtrader

@spyamine said in handle missing data:

but how can we disable this behaviour. if the data is available run if not wait until the data is available.

You have access to prenext. That's the behavior.

@spyamine said in handle missing data:

Also how can we handle survivor ship bia

You cannot. It's not the platform to do that.

spyamine

How can we imagine adding a support for survivor ship bias? is there any trick to use. Because it seems that backtesting do not start until the latest stock has the data available for it.

backtrader

@spyamine said in handle missing data:

Because it seems that backtesting do not start until the latest stock has the data available for it.

backtesting starts when all data feeds and indicators have gone through its minimum warm-up period. Which in the trivial case of a data feed is 1.

@ab_trader said in handle missing data:

@hobart-liu nope, prenext() is called before next(). You need to call trading logic in both functions. But in the prenext() you'll need to check the availability of data.
Backtrader goes thru all datafeeds bar by bar. Until all datafeeds have values and all indicators have values it calls prenext(). After it calls next().

spyamine

ok how to check if the data is available ?

backtrader

See: Docs - Platform Concepts specifically the section about the length of Lines objects.

This has been discussed several times.

spyamine

Many thanks and thank you for your patience