Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14139

Disk failure on one node can (eventually) block a whole cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Duplicate
    • None
    • None
    • Replication, Storage
    • None
    • Replication
    • ALL

    Description

      If a disk failure occurs in such a way as to block IO without returning (admittedly a rare occurrence), the affected mongod will never give up waiting for the IO to complete. Heartbeats are returned as normal, so other nodes will continue to trust it despite being permanently dysfunctional.

      A replica-set or a sharded cluster can eventually be locked up until the single faulty node is identified and terminated.

      Attachments

        Issue Links

          duplicates

          Task - A task that needs to be done. SERVER-29947 Implement Storage Node Watchdog

          • Major - P3 - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-15417 Arbiter didn't elect primary if OS is unreachable (except ping)

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-28422 Cluster stuck because replication heartbeat does not detect hanging members

          • Major - P3 - Major loss of function.
          • Closed
          related to

          New Feature - A new feature of the product, which has yet to be developed. SERVER-29980 Built-in hang detection diagnostics and recovery

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-9552 when replica set member has full disk, step down to (sec|rec)?

          • Major - P3 - Major loss of function.
          • Backlog

          Activity

            People

              backlog-server-repl Backlog - Replication Team
              andrew.ryder@mongodb.com Andrew Ryder (Inactive)
              Votes:
              5 Vote for this issue
              Watchers:
              47 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: