Fixes a mongod hang during storage engine shutdown under TSAN, seen as `task-timed-out` on the TSAN fuzzer variants (BF-45518, and the same signature as BF-38299 which was closed on a suspected-TSAN theory without a root cause).
-
- The deadlock
Read directly off a core dump of a wedged mongod. Two threads deadlock over glibc's dynamic loader lock:
*`SignalHandler`* holds the global lock for shutdown and is running the WiredTiger shutdown checkpoint:
```
__wt_txn_global_shutdown -> __wt_checkpoint_db -> __checkpoint_tree -> __wt_reconcile
-> __wt_blkcache_compress -> zstd_compress -> ZSTD_resetCCtx_internal
-> ___interceptor_memcpy -> ReportRace -> SymbolizeStack -> RefreshModules
> dl_iterate_phdr -> __lll_lock_wait < wants the loader lock
```
*`ftdc`* is collecting `serverStatus`, blocks on the global lock that shutdown holds, times out and throws:
```
EncryptionServerStatusSection::generateSection -> Lock::GlobalLock -> iassertFailed
-> __cxa_throw -> _Unwind_RaiseException -> _Unwind_Find_FDE
> dl_iterate_phdr < holds the loader lock
> ___interceptor_strlen -> TracePartAlloc -> FutexWait < wants a TSAN internal lock
```
Neither can proceed. The process consumes no CPU and cannot service SIGABRT, because it is inside the TSAN runtime holding its locks — which is why these failures have historically produced zero-byte stack trace files.
-
- Changes
*`etc/tsan.suppressions`* — add `called_from_lib:libzstd.so`. The existing `race:ZSTD_` cannot prevent this, because TSAN symbolizes a race *before it can match a suppression, so the reporting path (and the loader lock acquisition) still runs. `called_from_lib` mutes analysis of accesses made from within zstd, so the race is never detected and `ReportRace` never runs. This is the same mechanism already used for `libwiredtiger.so` three lines above.
*`src/mongo/db/mongod_main.cpp`* — move the `stopMongoDFTDC()` block in `shutdownTask()` from after storage engine shutdown to before `lockGlobal(opCtx, MODE_X)`. A `serverStatus` collection then cannot block on the global lock for the duration of storage shutdown, so it cannot time out and throw, so unwinding never takes the loader lock. The existing constraint documented at that call site — that FTDC be joined before `FlowControl::shutdown()` resets its decoration — is still satisfied, since the new position is earlier still.
*`buildscripts/resmokelib/hang_analyzer/dumper.py`* — under a sanitizer the analyzer sends SIGABRT instead of attaching a debugger, then waits its entire 12 minute budget. For a process blocked in its own signal handling path that signal is never serviced, so the task reports with no diagnostics at all. `SigabrtDumper` now bounds that wait (`SIGABRT_GRACE`, 2 minutes) and falls back to `GDBDumper.dump_live_backtraces()` for any survivors. That attaches, runs `thread apply all bt` and detaches, streaming to the task log — no core dump, which is the cost the sanitizer exemption exists to avoid. Uses `--nx` so it does not depend on `.gdbinit`, passes `check=False` so a failing gdb cannot raise out of the teardown path, and bounds each process by `BACKTRACE_TIMEOUT_SECONDS` so one slow symbol load cannot consume the whole budget.
-
- Safety
- The suppression only narrows what TSAN analyses inside a third party compression library whose races were already suppressed; it has no effect on non-sanitizer builds.
- The FTDC move preserves the ordering constraint its comment documents, and shutdown ordering was verified on a local build.
- The hang analyzer change is inert whenever SIGABRT works: the new path runs only for processes that survive it, which today produce no output at all.
-
- Reviewing
The three parts are independent and can be reviewed separately. They span two areas — the suppression and hang analyzer are DevProd Build, the shutdown ordering is Storage Execution — so reviewers from both would be useful. The shutdown ordering is the highest risk change since it affects every mongod shutdown.
- related to
-
SERVER-133979 Anchor the libzstd TSAN suppression
-
- Closed
-