Limit resmoke jobs to avoid output-drain timeouts on PPC64LE variant

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary: sharding_kill_stepdown_terminate_jscore_passthrough is failing because resmoke cannot drain mongod output quickly enough under high concurrency. The mongod processes complete shutdown, but their final log records remain queued for 3-4 minutes. Resmoke waits for those pipes to drain before considering the processes stopped, causing ContinuousStepdown to exceed its 300-second timeout. This is a test-infrastructure capacity issue, not a MongoDB correctness failure or a metadataConsistency bug.

      Details:
      On the affected PPC64LE variants:

      • The host reports 48 CPUs.
      • resmoke_jobs_factor: 0.25 calculates 12 concurrent jobs.
      • Each job launches nine mongods and two mongoses.
      • The task runs 132 MongoDB processes concurrently.
      • Each process produces verbose replication and heartbeat diagnostics.

      During ContinuousStepdown, resmoke terminates a primary and waits for the process to exit. The process exits quickly, but Process.wait() also waits for stdout and stderr to be fully consumed. When output draining falls behind, the stepdown thread remains active even though mongod has already shut down.

      v9.0 job7 evidence: Relevant timestamps, normalized to UTC:

      Server timestamp:  2026-07-26T17:08:46.043Z
      Message:           mongod shutdown complete
      Shutdown duration: 239 ms
      
      Resmoke receipt:   2026-07-26T17:12:51.432Z
      Output delay:      245.389 seconds
      

      Mongod completed its shutdown work in 239 ms, but resmoke did not emit the completion record for another 245 seconds.

      v9.0 job2 evidence

      Server timestamp:  2026-07-26T17:09:47.973Z
      Message:           mongod shutdown complete
      Shutdown duration: 33.085 seconds
      
      Resmoke receipt:   2026-07-26T17:14:06.672Z
      Output delay:      258.699 seconds
      

      Even after accounting for the 33-second shutdown, 259 additional seconds were spent waiting for output delivery.

      v8.3 job4 evidence

      [2026/05/24 03:45:06.330] [j4:s0:n2] {"t":{"$date":"2026-05-24T06:41:33.237-04:00"}, "msg":"mongod shutdown complete", "attr":{"Summary of time elapsed":{"Statistics":{ "shutdownTaskTotalMillis":222 }}}}
      

      After normalizing server and host time zones:

      Server timestamp:  2026-05-24T10:41:33.237Z
      Resmoke receipt:   2026-05-24T10:45:06.330Z
      Output delay:      213.093 seconds
      

      Same failure occurred on both:

      • MongoDB 8.3 with RHEL 9 PPC64LE.
      • MongoDB 9.0 with RHEL 10 PPC64LE.

      Resulting failure sequence
      Once output draining consumes most of the hook timeout:
      1. ContinuousStepdown cannot become idle within 300 seconds.
      2. Resmoke reports:

      Timed out waiting for stepdown thread to become idle;
      state=running, phase=step_down_all, timeout=300s
      

      3. Resmoke aborts the fixture.
      4. The metadata hook attempts another iteration against stopped mongoses.
      5. CheckMetadataConsistencyInBackground, fixture setup, teardown, and whichever foreground JS test happened to be running are reported as failures. Those later results are cascades, not independent root causes.

      Here is an example of passing ARM64 comparison
      ARM64 execution ran eight jobs:

      09:23:31.016  mongod shutdown complete
      09:23:31.032  resmoke begins restarting the process
      

      Output delay was 16 ms, and stepdown cycles completed in 1.8-3.8 seconds. ARM64 produced more output per job than PPC64LE but drained it at the lower concurrency.

      Fix:
      Limit this generated task at 2 resmoke jobs.

            Assignee:
            Aditya Deshpande
            Reporter:
            Aditya Deshpande
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: