Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89667

NULL ClusterServerParameterRefresher SEGV

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • Service Arch
    • ALL
    • Service Arch 2024-04-29, Service Arch 2024-05-13
    • 166

      While validating another patch, hit a core with a SEGV. Panic stack was

       

      #0  0x0000ffff9eadd5f4 in raise () from /lib64/libpthread.so.0
      #1  0x0000ffffa004f3cc in mongo::(anonymous namespace)::endProcessWithSignal (signalNum=signalNum@entry=11) at src/mongo/util/signal_handlers_synchronous.cpp:136
      #2  0x0000ffffa0050814 in mongo::(anonymous namespace)::abruptQuitWithAddrSignal (signalNum=11, siginfo=0x7075bf622da0, ucontext_erased=<optimized out>) at src/mongo/util/signal_handlers_synchronous.cpp:351
      #3  <signal handler called>
      #4  0x0000ffff9ead40a8 in pthread_mutex_trylock () from /lib64/libpthread.so.0
      #5  0x0000ffffa003d6ec in __gthread_mutex_trylock (__mutex=0x30) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/aarch64-mongodb-linux/bits/gthr-default.h:758
      #6  std::mutex::try_lock (this=0x30) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/std_mutex.h:111
      #7  mongo::latch_detail::Mutex::lock (this=this@entry=0x18) at src/mongo/platform/mutex.cpp:70
      #8  0x0000ffff9ddd3e04 in std::unique_lock<mongo::latch_detail::Latch>::lock (this=0xffff814b3890) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/unique_lock.h:131
      #9  std::unique_lock<mongo::latch_detail::Latch>::unique_lock (__m=..., this=0xffff814b3890) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/unique_lock.h:69
      #10 mongo::ClusterServerParameterRefresher::refreshParameters (this=0x0, opCtx=opCtx@entry=0x7075bfbe4800) at src/mongo/idl/cluster_server_parameter_refresher.cpp:227
      #11 0x0000ffff9dff3788 in mongo::(anonymous namespace)::GetClusterParameterCmd::Invocation::typedRun (opCtx=0x7075bfbe4800, this=0x7075bf8d4cc0) at src/mongo/s/commands/cluster_get_cluster_parameter_cmd.cpp:85
      #12 mongo::TypedCommand<mongo::(anonymous namespace)::GetClusterParameterCmd>::InvocationBase::_callTypedRun (opCtx=0x7075bfbe4800, this=0x7075bf8d4cc0) at src/mongo/db/commands.h:1437
      #13 mongo::TypedCommand<mongo::(anonymous namespace)::GetClusterParameterCmd>::InvocationBase::_runImpl (reply=0xffff814b3b68, opCtx=0x7075bfbe4800, this=0x7075bf8d4cc0) at src/mongo/db/commands.h:1443
      #14 mongo::TypedCommand<mongo::(anonymous namespace)::GetClusterParameterCmd>::InvocationBase::run (this=0x7075bf8d4cc0, opCtx=0x7075bfbe4800, reply=0xffff814b3b68) at src/mongo/db/commands.h:1448
      #15 0x0000ffff9932e834 in mongo::CommandHelpers::runCommandDirectly (opCtx=0x7075bfbe4800, request=...) at src/mongo/db/commands.cpp:162
      #16 0x0000ffff92be1ab8 in mongo::FTDCSimpleInternalCommandCollector::collect (this=<optimized out>, opCtx=<optimized out>, builder=...) at src/mongo/db/ftdc/ftdc_server.cpp:205
      #17 0x0000ffff92b7cd5c in mongo::FTDCCollectorCollection::collect (this=this@entry=0x7075bfbdcb68, client=client@entry=0x7075bf8bcbb0, multiServiceSchema=(unknown: 0x1)) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/unique_ptr.h:173
      #18 0x0000ffff92b83464 in mongo::FTDCController::doLoop (this=0x7075bfbdca00, service=<optimized out>) at src/mongo/db/ftdc/controller.cpp:298
      #19 0x0000ffff92b83734 in operator() (__closure=0x7075bf8608d0) at src/mongo/db/ftdc/controller.cpp:166
      #20 std::__invoke_impl<void, mongo::FTDCController::start(mongo::Service*)::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/invoke.h:61
      #21 std::__invoke<mongo::FTDCController::start(mongo::Service*)::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/invoke.h:96
      #22 std::__apply_impl<mongo::FTDCController::start(mongo::Service*)::<lambda()>, std::tuple<> > (__t=..., __f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/tuple:1858
      #23 std::apply<mongo::FTDCController::start(mongo::Service*)::<lambda()>, std::tuple<> > (__t=..., __f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/tuple:1869
      #24 operator() (__closure=0x7075bf8608c8) at src/mongo/stdx/thread.h:192
      #25 std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::FTDCController::start(mongo::Service*)::<lambda()> >(mongo::FTDCController::start(mongo::Service*)::<lambda()>)::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/invoke.h:61
      #26 std::__invoke<mongo::stdx::thread::thread<mongo::FTDCController::start(mongo::Service*)::<lambda()> >(mongo::FTDCController::start(mongo::Service*)::<lambda()>)::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/invoke.h:96
      #27 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::FTDCController::start(mongo::Service*)::<lambda()> >(mongo::FTDCController::start(mongo::Service*)::<lambda()>)::<lambda()> > >::_M_invoke<0> (this=0x7075bf8608c8) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/std_thread.h:253
      #28 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::FTDCController::start(mongo::Service*)::<lambda()> >(mongo::FTDCController::start(mongo::Service*)::<lambda()>)::<lambda()> > >::operator() (this=0x7075bf8608c8) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/std_thread.h:260
      #29 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::FTDCController::start(mongo::Service*)::<lambda()> >(mongo::FTDCController::start(mongo::Service*)::<lambda()>)::<lambda()> > > >::_M_run(void) (this=0x7075bf8608c0) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.6zb/include/c++/11.3.0/bits/std_thread.h:211
      #30 0x0000ffff9f0daaec in execute_native_thread_routine () from /data/debug/lib/libabsl_base.so
      #31 0x0000ffff9ead1230 in start_thread () from /lib64/libpthread.so.0
      #32 0x0000ffff9ea1f7dc in thread_start () from /lib64/libc.so.6 

      The culprit is the NULL ClusterServerParameterRefresher pointer in #10. gdb confirmed that the NULL+offset mutex pointer we eventually panicked on was at the correct offset:

      $12 = (mongo::ClusterServerParameterRefresher *) 0x0
      $13 = (mongo::Mutex *) 0x18
      $14 = (std::mutex *) 0x30 

            Assignee:
            ronald.steinke@mongodb.com Ronald Steinke
            Reporter:
            ronald.steinke@mongodb.com Ronald Steinke
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: