For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See:
2019-10-02: The community is currently in read-only mode

How can I check data for missing values?

  • Hi all,

    I have bought some intraday data but I would use pandas to check if something is missing in case of 5-minutes bar.

    How can I check it and fill any missing data?

  • 1. Determine if your timestamps are spaced equally (no gaps that are not equal to 5 minutes)

    Assuming that your timestamps are stored in the index column of the pandas dataframe df:

    timestamps = list(df.index.values)
    delta_t = list(set(timestamps[1:])-set(timestamps[:-1]))

    And check (for in-market timestamps) whether or not the any element in delta_t is not equal to 5 minutes.

    2. Filling Values in Pandas

    Pandas has extensive data gap filling capabilities - which you can learn more about here. But for stock/forex data I have used the backfill method, which uses the previous value to fill in the gap. This results in "flats" where the data forms a horizontal line on the chart where data is missing.

  • @Brandon-Johnson thanks a lot. I will try

Log in to reply