How can I check data for missing values?
I have bought some intraday data but I would use pandas to check if something is missing in case of 5-minutes bar.
How can I check it and fill any missing data?
Brandon Johnson last edited by Brandon Johnson
1. Determine if your timestamps are spaced equally (no gaps that are not equal to 5 minutes)
Assuming that your timestamps are stored in the index column of the pandas dataframe
timestamps = list(df.index.values) delta_t = list(set(timestamps[1:])-set(timestamps[:-1]))
And check (for in-market timestamps) whether or not the any element in
delta_tis not equal to 5 minutes.
2. Filling Values in Pandas
Pandas has extensive data gap filling capabilities - which you can learn more about here. But for stock/forex data I have used the
backfillmethod, which uses the previous value to fill in the gap. This results in "flats" where the data forms a horizontal line on the chart where data is missing.
@Brandon-Johnson thanks a lot. I will try