Heartbeat Failures to Unreachable Replica Set Hosts Lead to “Too Many Files Open” Error making the container crash

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: 3.12.8
    • Component/s: Monitoring
    • None
    • None
    • Java Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      Heartbeat Failures to Unreachable Replica Set Hosts Lead to “Too Many Files Open” Error making the container crash.

      The MongoDB Client Driver did not throws any exception nor logged anything in the log even after enabling logging on the driver.

      JAR version

      mongo-java-driver-3.12.8.jar
      mongodb-driver-async-3.12.8.jar
      mongodb-driver-core-3.12.8.jar

       

      How to Reproduce

      Just add unreachable invalid hostnames to replica sets.

      Additional Background

      We misconfigured MongoDB replica set connection string where some of the replica set member hostnames in the seed list were outdated and no longer reachable.

      mongodb://node1.example.com:27017,node2.example.com:27017,node3.example.com:27017/?replicaSet=rs0&readPreference=primaryPreferred&retryWrites=true&w=majority

      The MongoDB client driver continued sending heartbeat requests (hello checks) to these unavailable members. Because the hosts were unreachable, the heartbeat attempts failed, and the driver repeatedly opened new sockets without properly closing the failed ones.

      This behavior caused a gradual increase in open file descriptors, eventually leading to a container crash with the error “Too many files open.”

      The MongoDB Client Driver did not throws any exception nor logged anything in the log even after enabling logging on the driver as below.

      ((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.client")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.client.jndi")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.cluster")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.cluster.event")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.codecs.pojo")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.connection")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.connection.tls")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.uri")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.management")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.event")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.getmore")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.command")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.update")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.delete")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.killcursor")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.query")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.protocol.insert")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.operation")).setLevel(org.apache.logging.log4j.Level.TRACE);
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getLogger("org.mongodb.driver.PojoCodec")).setLevel(org.apache.logging.log4j.Level.TRACE);

       

      How to make MongoDB Driver log errors in such cases?

      How to make MongoDB Driver handle invalid hostnames?

            Assignee:
            Ross Lawley
            Reporter:
            Arjun Balla
            None
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: