ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.0.0-rc2
    • Affects Version/s: None
    • Component/s: None
    • DB Integration & Observability
    • Fully Compatible
    • v9.0
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      Add a patch to our bazel/wasmtime/ patch set applying the fix proposed in wasmtime#13857: in Stack::drop (lazy_per_thread_init(), crates/wasmtime/src/runtime/vm/sys/unix/signals.rs), call sigaltstack(SS_DISABLE) before munmapping the alternate signal stack, so no dangling registration is left in the kernel.

      Background

      BF-44398 (SIGSEGV in __sanitizer::LargeMmapAllocator::Deallocate on the FTDC thread, rhel8-arm64-debug-aubsan) is caused by the interaction described in wasmtime#13857 (independently root-caused, who hit the identical corruption in their ASan CI):

      1. On first WASM execution on a thread, wasmtime mmaps a 256KB (+1 guard page) sigaltstack and registers it — 0x50000 bytes total on RHEL8 aarch64's 64K pages, matching the corrupted hole in our core dump exactly.
      2. At thread exit, wasmtime's Rust TLS destructor (Stack::drop, signals.rs#L563-L571) munmaps the stack without SS_DISABLE, leaving the kernel registration dangling at the freed range. Both syscalls are raw (rustix), invisible to ASan.
      3. ASan's MmapOrDie recycles the freed range for large (>=128KB) heap allocations (WiredTiger buffers in our core).
      4. Later in the same thread's teardown, ASan's own cleanup (AsanThread::Destroy -> UnsetAlternateSignalStack in sanitizer_posix_libcdep.cpp) queries the registered altstack and unconditionally UnmapOrDie's it — unmapping the dangling wasmtime range, silently destroying its own live heap pages. The crash surfaces much later when the quarantine touches a destroyed chunk header (our fault address 0xfffe574e0000 is exactly stack-base + guard page — the start of ASan's unmap range).

      Notes:

      • LLVM main has since hardened UnsetAlternateSignalStack to only unmap the stack ASan itself installed; our toolchain's runtime (LLVM <= 19 era) unmaps unconditionally. The wasmtime-side SS_DISABLE fix is correct regardless and protects older runtimes.
      • The dangling registration is also a (much smaller) hazard on non-ASan builds: an SA_ONSTACK signal delivered between Stack::drop and thread death would write a signal frame into freed/reused memory. The fix removes that too, so it should apply to all build variants, not just sanitizer ones.
      • The code is unchanged in wasmtime 46, so a version upgrade alone does not fix this.

      Scope of Work

      1. Patch wasmtime

      Files to modify:

      • bazel/wasmtime/ — new patch for the wasmtime crate (same mechanism as the existing val_bytes_*.patch files) modifying Stack::drop in src/runtime/vm/sys/unix/signals.rs to disable the altstack before munmap, per the proposed fix in wasmtime#13857:
        impl Drop for Stack {
            fn drop(&mut self) {
                unsafe {
                    let disable = libc::stack_t {
                        ss_sp: ptr::null_mut(),
                        ss_flags: libc::SS_DISABLE,
                        ss_size: MIN_STACK_SIZE,
                    };
                    let r = libc::sigaltstack(&disable, ptr::null_mut());
                    debug_assert_eq!(r, 0, "sigaltstack(SS_DISABLE) failed during thread shutdown");
                    let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
                    debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
                }
            }
        }
        
      • MODULE.bazel / bazel/crates.lock — register the new patch for the wasmtime crate annotation.

      2. Test Coverage

      • Re-run WASM scope unit tests and a JS-heavy aggregation suite (e.g. aggregation_dependency_graph_validation_passthrough) on rhel8-arm64 aubsan and rhel8 asan variants.
      • Optional: verify via strace on an asan build that each sigaltstack munmap is preceded by an SS_DISABLE and that ASan's teardown no longer unmaps foreign ranges.

      Acceptance Criteria

      • Stack::drop disables the altstack registration before munmapping on all variants.
      • Sanitizer variants pass WASM scope unit tests and JS-heavy aggregation suites.
      • Patch is tracked for upstreaming (see SERVER-127394; wasmtime#13857 has no merged fix yet — we should offer this patch as the PR).

      Technical Notes

      • This supersedes the earlier proposal to gate wasmtime's cfg!(asan) skip-guard on our sanitizer builds: the SS_DISABLE fix keeps wasmtime's 256KB signal stack (no fallback to mongod's 64KB alt stack) and fixes the root defect for every embedder rather than skipping the code.
      • Changing kMongoMinSignalStackSize remains explicitly out of scope.
      • Building Rust with -Zsanitizer=address (separate ticket) remains valuable for ASan coverage of Rust code, but is no longer required to fix this BF.

            Assignee:
            Lee Maguire
            Reporter:
            Lee Maguire
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: