-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Integration
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary: sharding_kill_stepdown_terminate_jscore_passthrough is failing because resmoke cannot drain mongod output quickly enough under high concurrency. The mongod processes complete shutdown, but their final log records remain queued for 3-4 minutes. Resmoke waits for those pipes to drain before considering the processes stopped, causing ContinuousStepdown to exceed its 300-second timeout. This is a test-infrastructure capacity issue, not a MongoDB correctness failure or a metadataConsistency bug.
Details:
On the affected PPC64LE variants:
- The host reports 48 CPUs.
- resmoke_jobs_factor: 0.25 calculates 12 concurrent jobs.
- Each job launches nine mongods and two mongoses.
- The task runs 132 MongoDB processes concurrently.
- Each process produces verbose replication and heartbeat diagnostics.
During ContinuousStepdown, resmoke terminates a primary and waits for the process to exit. The process exits quickly, but Process.wait() also waits for stdout and stderr to be fully consumed. When output draining falls behind, the stepdown thread remains active even though mongod has already shut down.
v9.0 job7 evidence: Relevant timestamps, normalized to UTC:
Server timestamp: 2026-07-26T17:08:46.043Z Message: mongod shutdown complete Shutdown duration: 239 ms Resmoke receipt: 2026-07-26T17:12:51.432Z Output delay: 245.389 seconds
Mongod completed its shutdown work in 239 ms, but resmoke did not emit the completion record for another 245 seconds.
v9.0 job2 evidence
Server timestamp: 2026-07-26T17:09:47.973Z Message: mongod shutdown complete Shutdown duration: 33.085 seconds Resmoke receipt: 2026-07-26T17:14:06.672Z Output delay: 258.699 seconds
Even after accounting for the 33-second shutdown, 259 additional seconds were spent waiting for output delivery.
v8.3 job4 evidence
[2026/05/24 03:45:06.330] [j4:s0:n2] {"t":{"$date":"2026-05-24T06:41:33.237-04:00"}, "msg":"mongod shutdown complete", "attr":{"Summary of time elapsed":{"Statistics":{ "shutdownTaskTotalMillis":222 }}}}
After normalizing server and host time zones:
Server timestamp: 2026-05-24T10:41:33.237Z Resmoke receipt: 2026-05-24T10:45:06.330Z Output delay: 213.093 seconds
Same failure occurred on both:
- MongoDB 8.3 with RHEL 9 PPC64LE.
- MongoDB 9.0 with RHEL 10 PPC64LE.
Resulting failure sequence
Once output draining consumes most of the hook timeout:
1. ContinuousStepdown cannot become idle within 300 seconds.
2. Resmoke reports:
Timed out waiting for stepdown thread to become idle;
state=running, phase=step_down_all, timeout=300s
3. Resmoke aborts the fixture.
4. The metadata hook attempts another iteration against stopped mongoses.
5. CheckMetadataConsistencyInBackground, fixture setup, teardown, and whichever foreground JS test happened to be running are reported as failures. Those later results are cascades, not independent root causes.
Here is an example of passing ARM64 comparison
ARM64 execution ran eight jobs:
09:23:31.016 mongod shutdown complete 09:23:31.032 resmoke begins restarting the process
Output delay was 16 ms, and stepdown cycles completed in 1.8-3.8 seconds. ARM64 produced more output per job than PPC64LE but drained it at the lower concurrency.
Fix:
Limit this generated task at 2 resmoke jobs.