Loading...

Type: Bug
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None

Assigned Teams:

Storage Execution
Operating System:
ALL
Steps To Reproduce:
Hide

1. Change config in https://github.com/10gen/mongo/pull/48060/changes on file buildscripts/resmokeconfig/suites/query_golden_disagg.yml

to

fixture: class: DisaggReplicaSetFixture mongod_options: set_parameters: enableTestCommands: 1 num_nodes: 2 <--- This has to be minimum 2 not 1

2. run the following evergreen command:

evergreen patch --project=mongodb-mongo-master --variants=enterprise-amazon-linux2023-arm64-all-feature-flags-extra-system-deps --tasks=query_golden_disagg_gen -u
Show
1. Change config in https://github.com/10gen/mongo/pull/48060/changes on file buildscripts/resmokeconfig/suites/query_golden_disagg.yml to fixture: class: DisaggReplicaSetFixture mongod_options: set_parameters: enableTestCommands: 1 num_nodes: 2 <--- This has to be minimum 2 not 1 2. run the following evergreen command: evergreen patch --project=mongodb-mongo-master --variants=enterprise-amazon-linux2023-arm64-all-feature-flags-extra-system-deps --tasks=query_golden_disagg_gen -u
Sprint:
Storage Execution 2026-03-02, Storage Execution 2026-03-16
CAR Domain/s:
None

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name(s):
None
Goal Link:
None

by enabling 2 nodes of replication on disagg enabled query golden test set, The follow stack trace can be found:

Thread 1 (Thread 0xffff59f550c0 (LWP 220753)):
#0  0x0000ffff918b97b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff918703a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000aaaab6b16fbc [PAC] in mongo::endProcessWithSignal (signalNum=signalNum@entry=6) at ./src/mongo/util/signal_handlers_synchronous.cpp:405
#3  0x0000aaaab6b16ac4 in abruptQuit (signalNum=6) at ./src/mongo/util/signal_handlers_synchronous.cpp:261
#4  <signal handler called>
#5  0x0000ffff918b97b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#6  0x0000ffff918703a0 [PAC] in raise () from /lib64/libc.so.6
#7  0x0000ffff9185c264 [PAC] in abort () from /lib64/libc.so.6
#8  0x0000aaaab6b0ad08 [PAC] in mongo::(anonymous namespace)::callAbort () at ./src/mongo/util/assert_util.cpp:94
#9  0x0000aaaab6b0b020 in mongo::fassert_detail::failed (msgid=..., loc=...) at ./src/mongo/util/assert_util.cpp:231
#10 0x0000aaaab470e680 in mongo::ReplIndexBuildState::tryCommit (this=0x30e07f501660, opCtx=0x30e0541dac80) at ./src/mongo/db/index_builds/repl_index_build_state.cpp:460
#11 0x0000aaaab46e066c in mongo::IndexBuildsCoordinator::_tryCommit (opCtx=0x30e0541dac80, replState=std::shared_ptr<mongo::ReplIndexBuildState> (use count 4, weak count 0) = {...}, this=<optimized out>) at ./src/mongo/db/index_builds/index_builds_coordinator.cpp:1376
#12 mongo::IndexBuildsCoordinator::applyCommitIndexBuild (this=<optimized out>, opCtx=<optimized out>, oplogEntry=...) at ./src/mongo/db/index_builds/index_builds_coordinator.cpp:1341
#13 0x0000aaaab443dbb8 in mongo::repl::(anonymous namespace)::$_13::operator() (opCtx=0x30e0541dac80, mode=<optimized out>, this=<optimized out>, op=...) at ./src/mongo/db/repl/oplog.cpp:1083
#14 std::__invoke_impl<mongo::Status, mongo::repl::(anonymous namespace)::$_13&, mongo::OperationContext*, mongo::repl::ApplierOperation const&, mongo::repl::OplogApplication::Mode> (__args=..., __args=..., __f=..., __args=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:61
#15 std::__invoke_r<mongo::Status, mongo::repl::(anonymous namespace)::$_13&, mongo::OperationContext*, mongo::repl::ApplierOperation const&, mongo::repl::OplogApplication::Mode> (__args=..., __args=..., __fn=..., __args=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:114
#16 std::_Function_handler<mongo::Status(mongo::OperationContext*, mongo::repl::ApplierOperation const&, mongo::repl::OplogApplication::Mode), mongo::repl::(anonymous namespace)::$_13>::_M_invoke (__functor=..., __args=<optimized out>, __args=<optimized out>, __args=<optimized out>) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_function.h:290
#17 0x0000aaaab44320f4 in std::function<mongo::Status(mongo::OperationContext*, mongo::repl::ApplierOperation const&, mongo::repl::OplogApplication::Mode)>::operator() (__args=mongo::repl::OplogApplication::Mode::kSecondary, __args=mongo::repl::OplogApplication::Mode::kSecondary, __args=mongo::repl::OplogApplication::Mode::kSecondary, this=<optimized out>) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_function.h:591
#18 mongo::repl::applyCommand_inlock(mongo::OperationContext*, mongo::repl::ApplierOperation const&, mongo::repl::OplogApplication::Mode)::$_2::operator()() const (this=<optimized out>) at ./src/mongo/db/repl/oplog.cpp:2959
#19 mongo::repl::applyCommand_inlock (opCtx=<optimized out>, op=..., mode=<optimized out>) at ./src/mongo/db/repl/oplog.cpp:2954
#20 0x0000aaaab38b5a84 in mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool, std::function<void ()>, mongo::OpCounters*)::$_2::operator()() const (this=0xffff59f520d0) at ./src/mongo/db/repl/oplog_applier_utils.cpp:613
#21 mongo::WriteConflictRetryAlgorithm::operator()<mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool, std::function<void ()>, mongo::OpCounters*)::$_2>(mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool, std::function<void ()>, mongo::OpCounters*)::$_2&&) (this=0xffff59f521d0, f=...) at src/mongo/db/shard_role/lock_manager/exception_util.h:189
#22 mongo::writeConflictRetry<mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool, std::function<void ()>, mongo::OpCounters*)::$_2>(mongo::OperationContext*, mongo::StringData, mongo::NamespaceStringOrUUID const&, mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool, std::function<void ()>, mongo::OpCounters*)::$_2&&, boost::optional<unsigned long>, int) (opCtx=<optimized out>, nssOrUUID=..., f=..., dumpStateRetryCount=0, opStr=..., retryLimit=...) at src/mongo/db/shard_role/lock_manager/exception_util.h:278
#23 mongo::repl::OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon (opCtx=<optimized out>, entryOrGroupedInserts=..., oplogApplicationMode=<optimized out>, isDataConsistent=<optimized out>, incrementOpsAppliedStats=..., opCounters=<optimized out>) at ./src/mongo/db/repl/oplog_applier_utils.cpp:611
#24 0x0000aaaab38ad14c in mongo::repl::applyOplogEntryOrGroupedInserts (opCtx=0x30e0541dac80, entryOrGroupedInserts=..., oplogApplicationMode=mongo::repl::OplogApplication::Mode::kSecondary, isDataConsistent=true) at ./src/mongo/db/repl/oplog_applier_impl.cpp:1114
#25 0x0000aaaab38b649c in std::function<mongo::Status(mongo::OperationContext*, mongo::repl::OplogEntryOrGroupedInserts const&, mongo::repl::OplogApplication::Mode, bool)>::operator() (this=0xffff59f543c0, __args=true, __args=true, __args=true, __args=true) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_function.h:591
#26 mongo::repl::OplogApplierUtils::applyOplogBatchCommon (opCtx=0x30e0541dac80, ops=0x30e0b8685c98, oplogApplicationMode=mongo::repl::OplogApplication::Mode::kSecondary, allowNamespaceNotFoundErrorsOnCrudOps=false, isDataConsistent=<optimized out>, applyOplogEntryOrGroupedInserts=...) at ./src/mongo/db/repl/oplog_applier_utils.cpp:667
#27 0x0000aaaab38ad8c4 in mongo::repl::OplogApplierImpl::applyOplogBatchPerWorker (this=0x30e07fbe8b40, opCtx=0x30e0541dac80, ops=0x30e0b8685c98, workerMultikeyPathInfo=0x30e0b86cd518, isDataConsistent=true) at ./src/mongo/db/repl/oplog_applier_impl.cpp:1165
#28 0x0000aaaab38b1d74 in mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2::operator()(mongo::Status) const::{lambda()#1}::operator()() const (this=<optimized out>) at ./src/mongo/db/repl/oplog_applier_impl.cpp:767
#29 mongo::OperationContext::runWithoutInterruptionExceptAtGlobalShutdown<mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2::operator()(mongo::Status) const::{lambda()#1}>(mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2::operator()(mongo::Status) const::{lambda()#1}&&) (this=0x30e0541dac80, cb=...) at src/mongo/db/operation_context.h:866
#30 mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2::operator()(mongo::Status) const (this=0x30e04070bdc8, scheduleStatus=Status::OK()) at ./src/mongo/db/repl/oplog_applier_impl.cpp:766
#31 mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2>(mongo::repl::OplogApplierImpl::_applyOplogBatch(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry, std::allocator<mongo::repl::OplogEntry> >)::$_2&&)::SpecificImpl::call(mongo::Status&&) (this=0x30e04070bdc0, args=...) at src/mongo/util/functional.h:264
#32 0x0000aaaab1887e7c in mongo::unique_function<void(mongo::Status)>::operator() (args=Status::OK(), this=<optimized out>) at src/mongo/util/functional.h:222
#33 mongo::ThreadPool::Impl::_doOneTask (this=this@entry=0x30e07f6c7880, lk=lk@entry=0xffff59f54700) at ./src/mongo/util/concurrency/thread_pool.cpp:576
#34 0x0000aaaab69af564 in mongo::ThreadPool::Impl::_consumeTasks (this=0x30e07f6c7880, threadName="ReplWriterWorker-8") at ./src/mongo/util/concurrency/thread_pool.cpp:469
#35 mongo::ThreadPool::Impl::_workerThreadBody (this=0x30e07f6c7880, threadName=...) at ./src/mongo/util/concurrency/thread_pool.cpp:442
#36 mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0::operator()() const (this=<optimized out>) at ./src/mongo/util/concurrency/thread_pool.cpp:621
#37 std::__invoke_impl<void, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0>(std::__invoke_other, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0&&) (__f=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:61
#38 std::__invoke<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0&&) (__fn=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:96
#39 __apply_impl<(lambda at src/mongo/util/concurrency/thread_pool.cpp:621:31), std::tuple<> > (__f=..., __t=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/tuple:2921
#40 std::apply<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0, std::tuple<> >(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_0&&, std::tuple<>&&) (__f=..., __t=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/tuple:2936
#41 operator() (this=<optimized out>) at src/mongo/stdx/thread.h:197
#42 __invoke_impl<void, (lambda at src/mongo/stdx/thread.h:185:15)> (__f=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:61
#43 __invoke<(lambda at src/mongo/stdx/thread.h:185:15)> (__fn=...) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/invoke.h:96
#44 _M_invoke<0ul> (this=<optimized out>) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_thread.h:301
#45 operator() (this=<optimized out>) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_thread.h:308
#46 _M_run (this=<optimized out>) at external/mongo_toolchain_v5/stow/gcc-v5/include/c++/14.2.0/bits/std_thread.h:253
#47 0x0000aaaab6c99e5c in execute_native_thread_routine ()
#48 0x0000ffff918b7b78 in start_thread () from /lib64/libc.so.6
#49 0x0000ffff9192521c [PAC] in thread_start () from /lib64/libc.so.6

As seen in evergreen run: https://parsley.corp.mongodb.com/test/mongodb_mongo_master_enterprise_amazon_linux2023_arm64_all_feature_flags_extra_system_deps_query_golden_disagg_2_linux_enterprise_patch_1b440f5d6535aafda20145932ff66438423ceb1a_699edb2c3dd010000899b27b_26_02_25_11_21_46/0/458b305fa27806fbd9bfd5d763e98619?bookmarks=0,379&shareLine=0

while developing: https://github.com/10gen/mongo/pull/48060

is related to

SERVER-120518 fatal WiredTiger abort triggered during a disaggregated checkpoint update (setRecoveryCheckpointMetadata)

Needs Scheduling

Details

Description

Attachments

Issue Links

Activity

People

Dates