mongod crashes with SEGV in mongo::admission::ThroughputProbing::_run exactly 60 seconds after startup

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: 8.2.5
    • Component/s: None
    • Workload Resilience
    • ALL
    • Hide

      1. Install MongoDB server.
      2. Start the service: `sudo systemctl start mongodb`
      3. Verify it is running: `sudo systemctl status mongodb` (Shows Active/Running).
      4. Wait exactly 60 seconds.
      5. Check the status again: `sudo systemctl status mongodb`. The process is now in a failed state due to a core-dump (SEGV).

      Show
      1. Install MongoDB server. 2. Start the service: `sudo systemctl start mongodb` 3. Verify it is running: `sudo systemctl status mongodb` (Shows Active/Running). 4. Wait exactly 60 seconds. 5. Check the status again: `sudo systemctl status mongodb`. The process is now in a failed state due to a core-dump (SEGV).
    • WR Prioritized list
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      I am experiencing a consistent Segmentation Fault (SEGV) crash with MongoDB exactly 60 seconds after a successful startup. 

      The mongod process initializes perfectly, logs "mongod startup complete", and starts listening on port 27017. However, precisely 60 seconds later (likely when a background task or checkpoint triggers), the process crashes and dumps core. 

      Looking at the stack trace from `systemd-coredump`, the crash consistently originates in the `mongo::admission::ThroughputProbing::_run` function. 

      My environment:

      OS: CachyOS x86_64
      Kernel: Linux 6.19.6-2-cachyos
      CPU: AMD Ryzen 5 7600 (12) @ 5.17 GHz (Supports AVX2 / AVX-512)
      Memory: 30.45 GiB

      Mongodb package: https://aur.archlinux.org/packages/mongodb-bin

      Here is the systemd-coredump stack trace:
      Process 9783 (mongod) of user 954 dumped core.
                                                     
                                                    Stack trace of thread 9810:
                                                    #0  0x00005634c1894b5a _ZN5boost3log12v2s_mt_posix13attribute_setC1Ev (/usr/bin/mongod + 0x9ff5b5a)
                                                    #1  0x00005634bbcbc64b _ZN5mongo5logv29LogSourceC1EPKNS0_9LogDomain8InternalEb (/usr/bin/mongod + 0x441d64b)
                                                    #2  0x00005634bbca317a _ZN5mongo5logv215LogDomainGlobal6sourceEv (/usr/bin/mongod + 0x440417a)
                                                    #3  0x00005634bbcde2f3 _ZN5mongo5logv26detail10_doLogImplEiRKNS0_11LogSeverityERKNS0_10LogOptionsENS_10StringDataERKNS0_26TypeErasedAttribut>
                                                    #4  0x00005634bbb3512b _ZN5mongo5logv26detail9doLogImplEiRKNS0_11LogSeverityERKNS0_10LogOptionsENS_10StringDataERKNS0_26TypeErasedAttributeS>
                                                    #5  0x00005634c17b03ca n/a (/usr/bin/mongod + 0x9f113ca)
                                                    #6  0x00005634c17b0ad0 abruptQuitWithAddrSignal (/usr/bin/mongod + 0x9f11ad0)
                                                    #7  0x00007f542d444a40 n/a (libc.so.6 + 0x44a40)
                                                    #8  0x00007f542d5a7209 n/a (libc.so.6 + 0x1a7209)
                                                    #9  0x00005634c1b90f4a TCMallocInternalCalloc (/usr/bin/mongod + 0xa2f1f4a)
                                                    #10 0x00005634bbbbc36b __wt_calloc (/usr/bin/mongod + 0x431d36b)
                                                    #11 0x00005634bbbe0361 __wt_curhs_open (/usr/bin/mongod + 0x4341361)
                                                    #12 0x00005634bbc7a22b __wt_hs_get_btree (/usr/bin/mongod + 0x43db22b)
                                                    #13 0x00005634bc703883 n/a (/usr/bin/mongod + 0x4e64883)
                                                    #14 0x00005634bc7ae9e9 n/a (/usr/bin/mongod + 0x4f0f9e9)
                                                    #15 0x00007f542d4a9ad3 n/a (libc.so.6 + 0xa9ad3)
                                                    #16 0x00007f542d54067c n/a (libc.so.6 + 0x14067c)
                                                     
                                                    Stack trace of thread 9783:
                                                    #0  0x00007f542d4b2422 n/a (libc.so.6 + 0xb2422)
                                                    #1  0x00007f542d4a5e4f n/a (libc.so.6 + 0xa5e4f)
                                                    #2  0x00007f542d4a6247 n/a (libc.so.6 + 0xa6247)
                                                    #3  0x00007f542d4a8e3a pthread_cond_wait (libc.so.6 + 0xa8e3a)
                                                    #4  0x00005634bc532f79 ZNSt3_V222condition_variable_any4waitISt11unique_lockISt5mutexEEEvRT (/usr/bin/mongod + 0x4c93f79)
                                                    #5  0x00005634c17a81ef _ZN5mongo15waitForShutdownEv (/usr/bin/mongod + 0x9f091ef)
                                                    #6  0x00005634bbdd16d5 n/a (/usr/bin/mongod + 0x45326d5)
                                                    #7  0x00005634bbdd2312 n/a (/usr/bin/mongod + 0x4533312)
                                                    #8  0x00005634bbdd7a3e _ZN5mongo11mongod_mainEiPPc (/usr/bin/mongod + 0x4538a3e)
                                                    #9  0x00005634bbdc02a7 main (/usr/bin/mongod + 0x45212a7)
                                                    #10 0x00007f542d427c0e n/a (libc.so.6 + 0x27c0e)
                                                    #11 0x00007f542d427d4b __libc_start_main (libc.so.6 + 0x27d4b)
                                                    #12 0x00005634bbdc0185 _start (/usr/bin/mongod + 0x4521185)
                                                     
                                                    Stack trace of thread 9791:
                                                    #0  0x00007f542d4b2422 n/a (libc.so.6 + 0xb2422)
                                                    #1  0x00007f542d4a5e4f n/a (libc.so.6 + 0xa5e4f)
                                                    #2  0x00007f542d4fa2de clock_nanosleep (libc.so.6 + 0xfa2de)
                                                    #3  0x00007f542d50a4f7 __nanosleep (libc.so.6 + 0x10a4f7)
                                                    #4  0x00005634bbcbf894 AbslInternalSleepFor_lts_20250512 (/usr/bin/mongod + 0x4420894)
                                                    #5  0x00005634bbdbe416 MallocExtension_Internal_ProcessBackgroundActions (/usr/bin/mongod + 0x451f416)
                                                    #6  0x00005634bbde80bd _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IPFvvEJELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv (/u>
                                                    #7  0x00005634c1b55714 n/a (/usr/bin/mongod + 0xa2b6714)
                                                    #8  0x00007f542d4a9ad3 n/a (libc.so.6 + 0xa9ad3)
                                                    #9  0x00007f542d54067c n/a (libc.so.6 + 0x14067c)
                                                     
                                                    Stack trace of thread 9790:
                                                    #0  0x00007f542d4b2422 n/a (libc.so.6 + 0xb2422)
                                                    #1  0x00007f542d4a5e4f n/a (libc.so.6 + 0xa5e4f)
                                                    #2  0x00007f542d4a6247 n/a (libc.so.6 + 0xa6247)
                                                    #3  0x00007f542d4a8e3a pthread_cond_wait (libc.so.6 + 0xa8e3a)
      lines 1-68

      And here is the tail of the mongod.log showing successful startup right before the crash:
      {"t":\{"$date":"2026-03-11T12:03:29.125+03:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
      {"t":\{"$date":"2026-03-11T12:03:29.125+03:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":
      "/var/lib/mongodb/diagnostic.data"}}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDi
      sabled","oldState":"ConfigPreStart"}}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"STORAGE",  "id":7333401, "ctx":"initandlisten","msg":"Starting the DiskSpaceMonitor"}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"127.0.0.1:27017"}}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
      {"t":\{"$date":"2026-03-11T12:03:29.126+03:00"},"s":"I",  "c":"CONTROL",  "id":8423403, "ctx":"initandlisten","msg":"mongod startup complete","attr":{"Summary of time elapsed":{"Startup fro
      m clean shutdown?":false,"Statistics":{"setUpPeriodicRunnerMillis":0,"setUpOCSPMillis":0,"setUpTransportLayerMillis":0,"initSyncCrashRecoveryMillis":0,"createLockFileMillis":0,"getStorageE
      ngineMetadataMillis":0,"validateMetadataMillis":0,"createStorageEngineMillis":232,"writePIDMillis":0,"initializeFCVForIndexMillis":1,"dropAbandonedIdentsMillis":0,"standaloneClusterParamsM
      illis":0,"userAndRolesGraphMillis":0,"waitForMajorityServiceMillis":0,"startUpReplCoordMillis":0,"recoverChangeStreamMillis":0,"logStartupOptionsMillis":0,"startUpTransportLayerMillis":0,"
      initAndListenTotalMillis":237}}}}
      {"t":\{"$date":"2026-03-11T12:03:30.010+03:00"},"s":"I",  "c":"FTDC",     "id":20631,   "ctx":"ftdc","msg":"Unclean full-time diagnostic data capture shutdown detected, found interim file,
      some metrics may have been lost","attr":{"error":{"code":0,"codeName":"OK"}}}

            Assignee:
            Unassigned
            Reporter:
            Mert N/A
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: