[SERVER-41933] secondary block during background index build - MongoDB Jira

Details

Type: Bug
Status: Closed
Priority: Major - P3
Resolution: Duplicate
Affects Version/s: 4.0.10
Fix Version/s: None
Component/s: Replication
Labels:
None

Assigned Teams:

Triage
Operating System:
ALL
Steps To Reproduce:
Hide

setup a replica set

create a collection and import 10,000,000 records

create an index on this collection

drop the index created in step2 or some other indexes in this collection (some other DDL ops like drop collection, drop database have the some behavior)

when the secondary fetched the oplog for step3 and step4, the secondary will block all the request (read, write, serverStatus) until the background index building finish.
Show
setup a replica set create a collection and import 10,000,000 records create an index on this collection drop the index created in step2 or some other indexes in this collection (some other DDL ops like drop collection, drop database have the some behavior) when the secondary fetched the oplog for step3 and step4, the secondary will block all the request (read, write, serverStatus) until the background index building finish.

Description

When apply a DDL operation like drop/dropIndex/dropDatabase on secondary, if there is a background index building running, the secondary will block all operations(read, write, serverStatus).

background index building

Thread 70 (Thread 0x7f6a3eade700 (LWP 89525)):

#0  0x00007f6a5a5aaa5d in __wt_btcur_reset ()

#1  0x00007f6a5a5538a9 in ?? ()

#2  0x00007f6a5a471952 in mongo::WiredTigerCursor::reset() ()

#3  0x00007f6a5a45f83d in mongo::WiredTigerRecordStoreCursorBase::save() ()

#4  0x00007f6a5ac212b9 in mongo::PlanExecutor::saveState() ()

#5  0x00007f6a5aac724c in mongo::MultiIndexBlockImpl::insertAllDocumentsInCollection(std::set<mongo::RecordId, std::less<mongo::RecordId>, std::allocator<mongo::RecordId> >*) ()

#6  0x00007f6a5ab226ab in mongo::IndexBuilder::_build(mongo::OperationContext*, mongo::Database*, bool, mongo::Lock::DBLock*) const ()

#7  0x00007f6a5ab23bfa in mongo::IndexBuilder::run() ()

#8  0x00007f6a5b96c811 in mongo::BackgroundJob::jobBody() ()

#9  0x00007f6a5bfda350 in ?? ()

#10 0x00007f6a57e7baa1 in start_thread () from /lib64/libpthread.so.0

dropIndex is pending, awaitNoBgOpInProgForNs (this operation call TempRelease to release the lock)

Thread 41 (Thread 0x7f6a39ad6700 (LWP 70411)):

#0  0x00007f6a57e7f68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007f6a5bf6f54c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()

#2  0x00007f6a5b2478c3 in ?? ()

#3  0x00007f6a5b247b2a in mongo::BackgroundOperation::awaitNoBgOpInProgForNs(mongo::StringData) ()

#4  0x00007f6a5ab02669 in mongo::repl::applyCommand_inlock(mongo::OperationContext*, mongo::BSONObj const&, mongo::repl::OplogApplication::Mode) ()

#5  0x00007f6a5a74462a in ?? ()

#6  0x00007f6a5a748fbe in mongo::repl::SyncTail::syncApply(mongo::OperationContext*, mongo::BSONObj const&, mongo::repl::OplogApplication::Mode) ()

#7  0x00007f6a5a74afbc in mongo::repl::multiSyncApply(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*) ()#8  0x00007f6a5a732288 in std::_Function_handler<mongo::Status (mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*), mongo::Status (*)(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*&&, mongo::repl::SyncTail*&&, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*&&) ()

#9  0x00007f6a5a7452f5 in ?? ()

#10 0x00007f6a5b5f012c in mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*) ()

#11 0x00007f6a5b5f062c in mongo::ThreadPool::_consumeTasks() ()

#12 0x00007f6a5b5f1016 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()

#13 0x00007f6a5bfda350 in ?? ()

#14 0x00007f6a57e7baa1 in start_thread () from /lib64/libpthread.so.0

#15 0x00007f6a57bc893d in clone () from /lib64/libc.so.6

serverStatus, and other request is blocking, the secondary will be out of service during the whole index building process.

Thread 71 (Thread 0x7f6a3f5e0700 (LWP 89728)):

#0  0x00007f6a57e7fa5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007f6a5b57ae78 in mongo::CondVarLockGrantNotification::wait(mongo::Duration<std::ratio<1l, 1000l> >) ()

#2  0x00007f6a5b5819cd in mongo::LockerImpl<false>::lockComplete(mongo::OperationContext*, mongo::ResourceId, mongo::LockMode, mongo::Date_t, bool) ()

#3  0x00007f6a5b56e3c1 in mongo::Lock::ResourceLock::lock(mongo::LockMode) ()

#4  0x00007f6a5b56e491 in mongo::Lock::GlobalLock::_enqueue(mongo::LockMode, mongo::Date_t) ()

#5  0x00007f6a5b56e552 in mongo::Lock::GlobalLock::GlobalLock(mongo::OperationContext*, mongo::LockMode, mongo::Date_t, mongo::Lock::InterruptBehavior, mongo::Lock::GlobalLock::EnqueueOnly) ()

#6  0x00007f6a5b56e598 in mongo::Lock::GlobalLock::GlobalLock(mongo::OperationContext*, mongo::LockMode, mongo::Date_t, mongo::Lock::InterruptBehavior) ()

#7  0x00007f6a5a440bf8 in mongo::WiredTigerServerStatusSection::generateSection(mongo::OperationContext*, mongo::BSONElement const&) const ()

#8  0x00007f6a5b7b4fea in mongo::CmdServerStatus::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&) ()

#9  0x00007f6a5b85fdb9 in mongo::BasicCommand::Invocation::run(mongo::OperationContext*, mongo::CommandReplyBuilder*) ()

#10 0x00007f6a5a42c79f in ?? ()

#11 0x00007f6a5a42cbef in ?? ()

#12 0x00007f6a5a430df3 in ?? ()

#13 0x00007f6a5a432c8f in ?? ()

#14 0x00007f6a5a433f9d in mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) ()

#15 0x00007f6a5a41e58a in mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&) ()

#16 0x00007f6a5a42a0ca in mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard) ()

#17 0x00007f6a5a424d87 in mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard) ()

#18 0x00007f6a5a428591 in ?? ()

#19 0x00007f6a5b6ab222 in mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName) ()

#20 0x00007f6a5a422faf in mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership) ()

#21 0x00007f6a5a426135 in mongo::ServiceStateMachine::_sourceCallback(mongo::Status) ()

#22 0x00007f6a5a4244c7 in mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard) ()

#23 0x00007f6a5a424e0d in mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard) ()

Attachments

Issue Links

duplicates

SERVER-21307 Replicated DDL (catalog) operation during background index build blocks reads/writes

Closed

secondary block during background index build

Details

Description

Attachments

Issue Links

Activity

People

Dates