Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    How can I check data for missing values?

    General Discussion
    2
    3
    304
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      alexgiul last edited by

      Hi all,

      I have bought some intraday data but I would use pandas to check if something is missing in case of 5-minutes bar.

      How can I check it and fill any missing data?

      1 Reply Last reply Reply Quote 0
      • Brandon Johnson
        Brandon Johnson last edited by Brandon Johnson

        1. Determine if your timestamps are spaced equally (no gaps that are not equal to 5 minutes)

        Assuming that your timestamps are stored in the index column of the pandas dataframe df:

        timestamps = list(df.index.values)
        delta_t = list(set(timestamps[1:])-set(timestamps[:-1]))
        

        And check (for in-market timestamps) whether or not the any element in delta_t is not equal to 5 minutes.

        2. Filling Values in Pandas

        Pandas has extensive data gap filling capabilities - which you can learn more about here. But for stock/forex data I have used the backfill method, which uses the previous value to fill in the gap. This results in "flats" where the data forms a horizontal line on the chart where data is missing.

        A 1 Reply Last reply Reply Quote 0
        • A
          alexgiul @Brandon Johnson last edited by

          @Brandon-Johnson thanks a lot. I will try

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors