[SERVER-41009] TimestampMonitor and getOldestActiveTimestamp shouldn't conflict with PBWM lock Created: 03/May/19  Updated: 29/Oct/23  Resolved: 20/May/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.1.12

Type: Bug Priority: Major - P3
Reporter: Siyuan Zhou Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-41411 OplogTruncaterThread can self-deadlock Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Storage NYC 2019-05-20
Participants:
Linked BF Score: 33

 Description   

On secondary, TimestampMonitor and getOldestActiveTimestamp can block after PBWM lock.

Thread 15: "WTCheckpointThread" (Thread 0x7e984f92ee60 (LWP 16641))
#0  0x00007e9857b82b94 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0
#1  0x00000b5b64fa1e68 in __gthread_cond_timedwait (__abs_timeout=0x7e984f92d998, __mutex=<optimized out>, __cond=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/ppc64le-mongodb-linux/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:178
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:106
#4  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:129
#5  std::condition_variable::wait_for<long int, std::ratio<1, 1000000000>, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __rtime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:156
#6  mongo::CondVarLockGrantNotification::wait (this=this@entry=0xb5b93e490a0, timeout=..., timeout@entry=...) at src/mongo/db/concurrency/lock_state.cpp:213
#7  0x00000b5b64fa3eec in mongo::LockerImpl::lockComplete (this=0xb5b93e49000, opCtx=0x0, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:867
#8  0x00000b5b64fa67a0 in mongo::LockerImpl::lock (this=<optimized out>, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.h:173
#9  0x00000b5b64f9155c in mongo::Lock::ResourceLock::lock (this=0x7e984f92ddc8, mode=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:343
#10 0x00000b5b64f9172c in mongo::Lock::GlobalLock::_enqueue (this=this@entry=0x7e984f92ddb8, lockMode=lockMode@entry=mongo::MODE_IS, deadline=...) at src/mongo/db/concurrency/d_concurrency.cpp:181
#11 0x00000b5b64f918e8 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f92ddb8, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>, enqueueOnly=...) at src/mongo/db/concurrency/d_concurrency.cpp:158
#12 0x00000b5b64f91974 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f92ddb8, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:140
#13 0x00000b5b64f91a74 in mongo::Lock::DBLock::DBLock (this=0x7e984f92dda0, opCtx=0xb5bad92b400, db="config", mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_manager_defs.h:101
#14 0x00000b5b637a7650 in mongo::TransactionParticipant::getOldestActiveTimestamp (stableTimestamp=...) at src/mongo/base/string_data.h:95
#15 0x00000b5b62c7e5b8 in std::_Function_handler<mongo::StatusWith<boost::optional<mongo::Timestamp> > (mongo::Timestamp), mongo::StatusWith<boost::optional<mongo::Timestamp> > (*)(mongo::Timestamp)>::_M_invoke(std::_Any_data const&, mongo::Timestamp&&) (__functor=..., __args#0=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:88
#16 0x00000b5b62fd5dd0 in std::function<mongo::StatusWith<boost::optional<mongo::Timestamp> > (mongo::Timestamp)>::operator()(mongo::Timestamp) const (__args#0=..., this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:682
#17 mongo::WiredTigerKVEngine::getOplogNeededForRollback (this=0xb5b8b364500) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:1876
#18 0x00000b5b62fe236c in mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run (this=0xb5b8b613500) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:351
#19 0x00000b5b64fb0110 in mongo::BackgroundJob::jobBody (this=0xb5b8b613500) at src/mongo/util/background.cpp:148
#20 0x00000b5b65377694 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#21 0x00007e9857b7885c in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0
#22 0x00007e9857a99028 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6
 
 
Thread 16: "TimestampMonitor" (Thread 0x7e984f11ee60 (LWP 16654))
#0  0x00007e9857b82b94 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0
#1  0x00000b5b64fa1e68 in __gthread_cond_timedwait (__abs_timeout=0x7e984f11df18, __mutex=<optimized out>, __cond=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/ppc64le-mongodb-linux/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:178
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:106
#4  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:129
#5  std::condition_variable::wait_for<long int, std::ratio<1, 1000000000>, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __rtime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:156
#6  mongo::CondVarLockGrantNotification::wait (this=this@entry=0xb5b8b34aea0, timeout=..., timeout@entry=...) at src/mongo/db/concurrency/lock_state.cpp:213
#7  0x00000b5b64fa3eec in mongo::LockerImpl::lockComplete (this=0xb5b8b34ae00, opCtx=0x0, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:867
#8  0x00000b5b64fa67a0 in mongo::LockerImpl::lock (this=<optimized out>, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.h:173
#9  0x00000b5b64f9155c in mongo::Lock::ResourceLock::lock (this=0x7e984f11e268, mode=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:343
#10 0x00000b5b64f9172c in mongo::Lock::GlobalLock::_enqueue (this=this@entry=0x7e984f11e258, lockMode=lockMode@entry=mongo::MODE_IS, deadline=...) at src/mongo/db/concurrency/d_concurrency.cpp:181
#11 0x00000b5b64f918e8 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f11e258, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>, enqueueOnly=...) at src/mongo/db/concurrency/d_concurrency.cpp:158
#12 0x00000b5b64f91974 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f11e258, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:140
#13 0x00000b5b62f72bcc in mongo::Lock::GlobalLock::GlobalLock (lockMode=mongo::MODE_IS, opCtx=<optimized out>, this=0x7e984f11e258) at src/mongo/db/concurrency/d_concurrency.h:197
#14 mongo::KVStorageEngine::TimestampMonitor::<lambda(mongo::Client*)>::operator() (__closure=0xb5b8b2c5f30, client=<optimized out>) at src/mongo/db/storage/kv/kv_storage_engine.cpp:910
#15 std::_Function_handler<void(mongo::Client*), mongo::KVStorageEngine::TimestampMonitor::startup()::<lambda(mongo::Client*)> >::_M_invoke(const std::_Any_data &, mongo::Client *&&) (__functor=..., __args#0=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:297
#16 0x00000b5b6345b468 in std::function<void (mongo::Client*)>::operator()(mongo::Client*) const (__args#0=0xb5b90451500, this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:682
#17 mongo::PeriodicRunnerImpl::PeriodicJobImpl::<lambda()>::operator() (__closure=0xb5b8b0f30f8) at src/mongo/util/periodic_runner_impl.cpp:119
#18 std::__invoke_impl<void, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/invoke.h:60
#19 std::__invoke<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/invoke.h:95
#20 std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > >::_M_invoke<0> (this=0xb5b8b0f30f8) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:234
#21 std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > >::operator() (this=0xb5b8b0f30f8) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:243
#22 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > > >::_M_run(void) (this=0xb5b8b0f30f0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:186
#23 0x00000b5b65377694 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#24 0x00007e9857b7885c in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0
#25 0x00007e9857a99028 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6



 Comments   
Comment by Githook User [ 20/May/19 ]

Author:

{'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek', 'username': 'GWlodarek'}

Message: SERVER-41009 TimestampMonitor and getOldestActiveTimestamp shouldn't conflict with PBWM lock
Branch: master
https://github.com/mongodb/mongo/commit/ad2b5ce357ad424355f364c4c4cb6c2acae84be1

Generated at Thu Feb 08 04:56:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.