[SERVER-53435] Fix SBE bug with replset init where checkCanServeReads fails Created: 18/Dec/20  Updated: 29/Oct/23  Resolved: 15/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Task Priority: Major - P3
Reporter: Alya Berciu Assignee: David Storch
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Related
is related to SERVER-48489 [SBE] Implement a check that the curr... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2021-01-11, Query 2021-01-25
Participants:
Linked BF Score: 42

 Description   

https://github.com/mongodb/mongo/blob/bb3279a62dfc65bdafa5164a7ce85c8322b4757b/src/mongo/db/repl/replication_coordinator_impl.cpp#L2887

When a ShardingTest is being initialized with sbe_enabled, it looks like the above check to see if a node can serve reads fails in the call to IndexScanStage::open():
https://github.com/mongodb/mongo/blob/bb3279a62dfc65bdafa5164a7ce85c8322b4757b/src/mongo/db/exec/sbe/stages/ix_scan.cpp#L179
The failure may be related to SERVER-48489, and should be reproducible by running:

python buildscripts/resmoke.py run --suite=no_passthrough --mongodSetParameters="{featureFlagSBE: true}" jstests/noPassthrough/currentop_query.js

This work should make it possible to remove the sbe_incompatible tag from at least these noPassthrough tests (it is worth checking if a few more can have the tag removed as well once this is fixed):

  • jstests/noPassthrough/currentop_query.js
  • jstests/noPassthrough/kill_operations.js
  • jstests/noPassthrough/killop.js

 



 Comments   
Comment by Githook User [ 19/Jan/21 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-53435 Fix use-after-move
Branch: master
https://github.com/mongodb/mongo/commit/f49f705a741b80e48db54050cbb50173b22d44f2

Comment by Githook User [ 15/Jan/21 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-53435 Make queries run with DBHelpers plus SBE circumvent 'checkCanServeReadsFor()'

Such queries may be run when a node is neither primary nor
secondary, e.g. as part of executing an initial sync. This
affects only the SBE engine, since SBE's internal lock
policy requires it to have its own calls to
'checkCanServeReadsFor()'.
Branch: master
https://github.com/mongodb/mongo/commit/8ccc099b1821a18a120a9ce483bf64dac1899ad1

Comment by David Storch [ 07/Jan/21 ]

Using gdbserver I was able to get the following backtrace for the point at which the NotPrimaryOrSecondary error occurs:

(gdb) bt
#0  0x00007f23bb605f47 in raise () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555e0291ab6f in mongo::breakpoint () at src/mongo/util/debugger.cpp:72
#2  0x0000555dfdd505e7 in mongo::repl::ReplicationCoordinatorImpl::checkCanServeReadsFor_UNSAFE (
    this=0x7f23b571b020, opCtx=0x7f23b5690fe0, ns=..., secondaryOk=true)
    at src/mongo/db/repl/replication_coordinator_impl.cpp:2901
#3  0x0000555dfdd50172 in mongo::repl::ReplicationCoordinatorImpl::checkCanServeReadsFor (
    this=0x7f23b571b020, opCtx=0x7f23b5690fe0, ns=..., secondaryOk=true)
    at src/mongo/db/repl/replication_coordinator_impl.cpp:2857
#4  0x0000555dffdff1c7 in mongo::sbe::IndexScanStage::open (this=0x7f239aba3620, reOpen=false)
    at src/mongo/db/exec/sbe/stages/ix_scan.cpp:179
#5  0x0000555dffe70d77 in mongo::sbe::LoopJoinStage::openInner (this=0x7f23ad5943a0)
    at src/mongo/db/exec/sbe/stages/loop_join.cpp:100
#6  0x0000555dffe70e20 in mongo::sbe::LoopJoinStage::getNext (this=0x7f23ad5943a0)
    at src/mongo/db/exec/sbe/stages/loop_join.cpp:112
#7  0x0000555dffe70df3 in mongo::sbe::LoopJoinStage::getNext (this=0x7f23b56958a0)
    at src/mongo/db/exec/sbe/stages/loop_join.cpp:107
#8  0x0000555dffac5661 in mongo::fetchNext (root=0x7f23b56958a0, resultSlot=0x7f239ac050c0,
    recordIdSlot=0x7f239ac06600, out=0x7f2388d4a360, dlOut=0x7f2388d4a358)
    at src/mongo/db/query/plan_executor_sbe.cpp:291
#9  0x0000555dffac53d4 in mongo::PlanExecutorSBE::getNext (this=0x7f23b568fc20, out=0x7f2388d4a360,
    dlOut=0x7f2388d4a358) at src/mongo/db/query/plan_executor_sbe.cpp:229
#10 0x0000555dff9613e9 in mongo::Helpers::findOne (opCtx=0x7f23b5690fe0, collection=..., qr=...,
    requireIndex=false) at src/mongo/db/dbhelpers.cpp:116
#11 0x0000555dff961064 in mongo::Helpers::findOne (opCtx=0x7f23b5690fe0, collection=..., query=...,
    requireIndex=false) at src/mongo/db/dbhelpers.cpp:86
#12 0x0000555dff960e39 in mongo::Helpers::findOne (opCtx=0x7f23b5690fe0, collection=..., query=...,
    result=..., requireIndex=false) at src/mongo/db/dbhelpers.cpp:67
--Type <RET> for more, q to quit, c to continue without paging--
#13 0x0000555dfdc70bf2 in mongo::repl::StorageInterfaceImpl::isAdminDbValid (this=0x7f23b54b3400,
    opCtx=0x7f23b5690fe0) at src/mongo/db/repl/storage_interface_impl.cpp:1364
#14 0x0000555dfec359cd in mongo::repl::AllDatabaseCloner::postStage (this=0x7f23aebacf20)
    at src/mongo/db/repl/all_database_cloner.cpp:250
#15 0x0000555dfec9baaf in mongo::repl::BaseCloner::run (this=0x7f23aebacf20)
    at src/mongo/db/repl/base_cloner.cpp:76
#16 0x0000555dfeca2abb in mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}::operator()() const (
    this=0x7f2388d4b8f0) at src/mongo/db/repl/base_cloner.cpp:241
#17 0x0000555dfeca2a6c in mongo::future_details::call<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&, mongo::future_details::FakeVoid) (func=...)
    at src/mongo/util/future_impl.h:255
#18 0x0000555dfeca2890 in mongo::future_details::statusCall<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&, mongo::future_details::FakeVoid>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&, mongo::future_details::FakeVoid&&) (func=..., args=...) at src/mongo/util/future_impl.h:276
#19 0x0000555dfeca2616 in mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) &&::{lambda(mongo::future_details::FakeVoid&&)#1}::operator()(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) const (this=0x7f2388d4b7d0, val=...) at src/mongo/util/future_impl.h:897
#20 0x0000555dfeca237f in mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::generalImpl<m--Type <RET> for more, q to quit, c to continue without paging--
ongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) &&::{lambda(mongo::future_details::FakeVoid&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<{lambda()#1}>({lambda()#1}) &&::{lambda(mongo::Status&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<{lambda()#1}>({lambda()#1}) &&::{lambda()#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) &&::{lambda(mongo::future_details::FakeVoid&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<{lambda()#1}>({lambda()#1}) &&::{lambda(mongo::Status&&)#1}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<{lambda()#1}>({lambda()#1}) &&::{lambda()#1}&&) (this=0x7f2388d4b898, success=..., fail=..., notReady=...)
    at src/mongo/util/future_impl.h:1184
#21 0x0000555dfeca22b8 in mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) && (
    this=0x7f2388d4b898, func=...) at src/mongo/util/future_impl.h:894
#22 0x0000555dfeca21a8 in mongo::Future<void>::then<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}, 0>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) && (this=0x7f2388d4b898, func=...)
    at src/mongo/util/future.h:403
#23 0x0000555dfeca20fa in mongo::Promise<void>::setWith<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1--Type <RET> for more, q to quit, c to continue without paging--
}, 0>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&)::{lambda()#1}&&) (this=0x7f23aebacff0, func=...)
    at src/mongo/util/future.h:811
#24 0x0000555dfeca2070 in mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) (this=0x7f23a97af6e8, args=...)
    at src/mongo/db/repl/base_cloner.cpp:240
#25 0x0000555dfeca1fa0 in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::callRegularVoid<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10>(std::integral_constant<bool, true>, mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10&, mongo::executor::TaskExecutor::CallbackArgs const&) (isVoid=..., f=..., args=...)
    at src/mongo/util/functional.h:145
#26 0x0000555dfeca1f61 in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::makeImpl<mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10&>(mongo::repl::BaseCloner::runOnExecutorEvent(mongo::executor::TaskExecutor*)::$_10&)::SpecificImpl::call(mongo::executor::TaskExecutor::CallbackArgs const&) (this=0x7f23a97af6e0, args=...) at src/mongo/util/functional.h:159
#27 0x0000555dfdcc2fbd in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const (this=0x7f239abfd750, args=...)
    at src/mongo/util/functional.h:109
#28 0x0000555dffd27433 in mongo::executor::ScopedTaskExecutor::Impl::_wrapCallback<mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> >(mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}&&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1 const&)#1}::operator()<mongo::executor::TaskExecutor::CallbackArgs>(mongo::executor::TaskExecutor::CallbackArgs const) const::{lambda(mongo::executor::TaskExecutor::CallbackArgs const&)#1}::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const (
--Type <RET> for more, q to quit, c to continue without paging--
    this=0x7f2388d4ba98, x=...) at src/mongo/executor/scoped_task_executor.cpp:274
#29 0x0000555dffd272de in mongo::executor::ScopedTaskExecutor::Impl::_wrapCallback<mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> >(mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}&&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1 const&)#1}::operator()<mongo::executor::TaskExecutor::CallbackArgs>(mongo::executor::TaskExecutor::CallbackArgs const) const (this=0x7f239abfd748, cargs=...)
    at src/mongo/executor/scoped_task_executor.cpp:283
#30 0x0000555dffd27230 in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::callRegularVoid<mongo::executor::ScopedTaskExecutor::Impl::_wrapCallback<mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> >(mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}&&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1 const&)#1}>(std::integral_constant<bool, true>, mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}&, mongo::executor::TaskExecutor::CallbackArgs const&) (isVoid=..., f=..., args=...) at src/mongo/util/functional.h:145
#31 0x0000555dffd27141 in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::makeImpl<mongo::executor::ScopedTaskExecutor::Impl::_wrapCallback<mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> >(mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArg--Type <RET> for more, q to quit, c to continue without paging--
s const&)>&&)::{lambda(auto:1&&)#1}&&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1 const&)#1}>(mongo::executor::ScopedTaskExecutor::Impl::onEvent(mongo::executor::TaskExecutor::EventHandle const&, mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>&&)::{lambda(auto:1&&)#1}&&)::SpecificImpl::call(mongo::executor::TaskExecutor::CallbackArgs const&) (this=0x7f239abfd740, args=...) at src/mongo/util/functional.h:159
#32 0x0000555dfdcc2fbd in mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const (this=0x7f2388d4bc28, args=...)
    at src/mongo/util/functional.h:109
#33 0x0000555e01c4ce57 in mongo::executor::ThreadPoolTaskExecutor::runCallback (this=0x7f23b568fa60,
    cbStateArg=...) at src/mongo/executor/thread_pool_task_executor.cpp:650
#34 0x0000555e01c559ee in mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__cxx11::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<mongo::latch_detail::Latch>)::$_3::operator()<mongo::Status>(mongo::Status) const (this=0x7f239ac06068, status=...)
    at src/mongo/executor/thread_pool_task_executor.cpp:629
#35 0x0000555e01c558b1 in mongo::unique_function<void (mongo::Status)>::callRegularVoid<mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__cxx11::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<mongo::latch_detail::Latch>)::$_3>(std::integral_constant<bool, true>, mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__cxx11::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackSta--Type <RET> for more, q to quit, c to continue without paging--
te> > const&, std::unique_lock<mongo::latch_detail::Latch>)::$_3&, mongo::Status&&) (isVoid=..., f=...,
    args=...) at src/mongo/util/functional.h:145
#36 0x0000555e01c55851 in mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__cxx11::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<mongo::latch_detail::Latch>)::$_3>(mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__cxx11::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<mongo::latch_detail::Latch>)::$_3&&)::SpecificImpl::call(mongo::Status&&) (this=0x7f239ac06060, args=...)
    at src/mongo/util/functional.h:159
#37 0x0000555dfe164ced in mongo::unique_function<void (mongo::Status)>::operator()(mongo::Status) const (
    this=0x7f2388d4bec8, args=...) at src/mongo/util/functional.h:109
#38 0x0000555e02575860 in mongo::ThreadPool::Impl::_doOneTask (this=0x7f23ad88c520, lk=0x7f2388d4c4b0)
    at src/mongo/util/concurrency/thread_pool.cpp:522
#39 0x0000555e02574568 in mongo::ThreadPool::Impl::_consumeTasks (this=0x7f23ad88c520)
    at src/mongo/util/concurrency/thread_pool.cpp:410
#40 0x0000555e02574116 in mongo::ThreadPool::Impl::_workerThreadBody (this=0x7f23ad88c520, threadName=...)
    at src/mongo/util/concurrency/thread_pool.cpp:397
#41 0x0000555e0257af97 in mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21::operator()() const (
    this=0x7f23b3b97530) at src/mongo/util/concurrency/thread_pool.cpp:572
#42 0x0000555e0257af5d in std::__invoke_impl<void, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21>(std::__invoke_other, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21&&) (__f=...)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86--Type <RET> for more, q to quit, c to continue without paging--
_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:60
#43 0x0000555e0257af2d in std::__invoke<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21&&) (__fn=...)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:95
#44 0x0000555e0257af01 in std::__apply_impl<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, std::tuple<>>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21&&, std::tuple<>&&, std::integer_sequence<unsigned long>) (__f=..., __t=...)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/tuple:1678
#45 0x0000555e0257aed2 in std::apply<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, std::tuple<> >(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21&&, std::tuple<>&&) (__f=..., __t=...)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/tuple:1687
#46 0x0000555e0257ae3d in mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}::operator()() (
    this=0x7f23b3b97528) at src/mongo/stdx/thread.h:186
#47 0x0000555e0257adbd in std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}>(std::__invoke_other, mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}&&) (__f=...)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:60
#48 0x0000555e0257ad3d in std::__invoke<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}>(mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}&&) (__fn=...)
--Type <RET> for more, q to quit, c to continue without paging--
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:95
#49 0x0000555e0257ad15 in std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}> >::_M_invoke<0ul> (this=0x7f23b3b97528)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:234
#50 0x0000555e0257ace5 in std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}> >::operator()() (this=0x7f23b3b97528)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:243
#51 0x0000555e0257ab89 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21, , 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::$_21)::{lambda()#1}> > >::_M_run() (this=0x7f23b3b97520)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:186
#52 0x0000555e02ccb7af in std::execute_native_thread_routine (__p=0x7f23b3b97520)
    at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#53 0x00007f23bb1a76db in start_thread () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#54 0x00007f23bb6e8a3f in clone () from target:/lib/x86_64-linux-gnu/libc.so.6

If I'm understanding this stack correctly, the syncing node is checking the validity of its admin database. In particular, it is looking for the document with {_id: "authSchema"} in the "admin.system.version" collection. This find-by-_id query fails when the SBE engine notices that the node is not primary or secondary.

Comment by David Storch [ 07/Jan/21 ]

I was able to reproduce this bug against the HEAD of master (351af2efd3a0ddcdfc284350ff81d98fedd0a001) in a simpler fashion than described above using the following:

➜ cat repro.js
const st = new ShardingTest({shards: 2, rs: {nodes: 1}});
st.stop();
➜ python3 buildscripts/resmoke.py run --suite=no_passthrough --mongodSetParameters="{internalQueryEnableSlotBasedExecutionEngine: true}" repro.js | tee repro.log

The symptom is that the resulting log file shows failed initial syncs due to a NotPrimaryOrSecondary error:

:21200,   "ctx":"ReplCoordExtern-0","msg":"Initial sync attempt failed","attr":{"attemptsLeft":9,"error":"NotPrimaryOrSecondary: error cloning databases :: caused by :: not master or secondary; cannot currently read from this replSet member"}}

Generated at Thu Feb 08 05:30:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.