How can I check data for missing values?

alexgiul

Hi all,

I have bought some intraday data but I would use pandas to check if something is missing in case of 5-minutes bar.

How can I check it and fill any missing data?

Brandon Johnson

1. Determine if your timestamps are spaced equally (no gaps that are not equal to 5 minutes)

Assuming that your timestamps are stored in the index column of the pandas dataframe df:

timestamps = list(df.index.values)
delta_t = list(set(timestamps[1:])-set(timestamps[:-1]))

And check (for in-market timestamps) whether or not the any element in delta_t is not equal to 5 minutes.

2. Filling Values in Pandas

Pandas has extensive data gap filling capabilities - which you can learn more about here. But for stock/forex data I have used the backfill method, which uses the previous value to fill in the gap. This results in "flats" where the data forms a horizontal line on the chart where data is missing.

alexgiul

@Brandon-Johnson thanks a lot. I will try