Java driver creates too many monitoring threads causing resource exhaustion in production

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: Performance
    • None
    • Java Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Problem:
      We are seeing around 1500+ threads for cluster monitoring and connection cleanup in our production environment.

      The application has around 160 mongo clients each connecting to different replica set. Each replica set has 3-4 members (primary/secondary/arbiter/hidden secondary){}

      Root Cause
      {}Every single server that gets monitored gets its own set of 3 dedicated threads:
        - One for health checks (cluster-{id}-{address}) in DefaultServerMonitor
        - One for measuring latency (cluster-rtt-{id}-{address}) in DefaultServerMonitor
        - One for background maintenance in DefaultConnectionPool

      This adds up fast creating 1500+ threads. This contributes to jvm memory/thread management overhead, context switching costs, and can hit OS thread limits in containerized environments.{}

      Proposed Solution
      The current approach of thread-per-server task seems overkill.

      We can replace the dedicated threads with shared ScheduledExecutorService instances. Instead of spinning up a new thread for every server, use a pool of threads that can handle monitoring tasks for all servers.

      This would dramatically cut down the thread count while keeping all the same functionality.

      Further Optimization

      We could also add configuration options to skip monitoring servers that clients never actually connect to, like arbiters and hidden secondaries. They're important for the replica set but not for client operations, so monitoring them is often unnecessary overhead.

      I can pick this up and contribute to the project if we are aligned on the approach.

       

            Assignee:
            Unassigned
            Reporter:
            Manas Joshi
            None
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: