How can I check data for missing values?
-
Hi all,
I have bought some intraday data but I would use pandas to check if something is missing in case of 5-minutes bar.
How can I check it and fill any missing data?
-
1. Determine if your timestamps are spaced equally (no gaps that are not equal to 5 minutes)
Assuming that your timestamps are stored in the index column of the pandas dataframe
df
:timestamps = list(df.index.values) delta_t = list(set(timestamps[1:])-set(timestamps[:-1]))
And check (for in-market timestamps) whether or not the any element in
delta_t
is not equal to 5 minutes.2. Filling Values in Pandas
Pandas has extensive data gap filling capabilities - which you can learn more about here. But for stock/forex data I have used the
backfill
method, which uses the previous value to fill in the gap. This results in "flats" where the data forms a horizontal line on the chart where data is missing. -
@Brandon-Johnson thanks a lot. I will try