Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41009

TimestampMonitor and getOldestActiveTimestamp shouldn't conflict with PBWM lock

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.1.12
    • Affects Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Storage NYC 2019-05-20
    • 33

      On secondary, TimestampMonitor and getOldestActiveTimestamp can block after PBWM lock.

      Thread 15: "WTCheckpointThread" (Thread 0x7e984f92ee60 (LWP 16641))
      #0  0x00007e9857b82b94 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0
      #1  0x00000b5b64fa1e68 in __gthread_cond_timedwait (__abs_timeout=0x7e984f92d998, __mutex=<optimized out>, __cond=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/ppc64le-mongodb-linux/bits/gthr-default.h:871
      #2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:178
      #3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:106
      #4  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:129
      #5  std::condition_variable::wait_for<long int, std::ratio<1, 1000000000>, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __rtime=..., __lock=<synthetic pointer>..., this=0xb5b93e490d0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:156
      #6  mongo::CondVarLockGrantNotification::wait (this=this@entry=0xb5b93e490a0, timeout=..., timeout@entry=...) at src/mongo/db/concurrency/lock_state.cpp:213
      #7  0x00000b5b64fa3eec in mongo::LockerImpl::lockComplete (this=0xb5b93e49000, opCtx=0x0, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:867
      #8  0x00000b5b64fa67a0 in mongo::LockerImpl::lock (this=<optimized out>, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.h:173
      #9  0x00000b5b64f9155c in mongo::Lock::ResourceLock::lock (this=0x7e984f92ddc8, mode=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:343
      #10 0x00000b5b64f9172c in mongo::Lock::GlobalLock::_enqueue (this=this@entry=0x7e984f92ddb8, lockMode=lockMode@entry=mongo::MODE_IS, deadline=...) at src/mongo/db/concurrency/d_concurrency.cpp:181
      #11 0x00000b5b64f918e8 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f92ddb8, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>, enqueueOnly=...) at src/mongo/db/concurrency/d_concurrency.cpp:158
      #12 0x00000b5b64f91974 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f92ddb8, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:140
      #13 0x00000b5b64f91a74 in mongo::Lock::DBLock::DBLock (this=0x7e984f92dda0, opCtx=0xb5bad92b400, db="config", mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_manager_defs.h:101
      #14 0x00000b5b637a7650 in mongo::TransactionParticipant::getOldestActiveTimestamp (stableTimestamp=...) at src/mongo/base/string_data.h:95
      #15 0x00000b5b62c7e5b8 in std::_Function_handler<mongo::StatusWith<boost::optional<mongo::Timestamp> > (mongo::Timestamp), mongo::StatusWith<boost::optional<mongo::Timestamp> > (*)(mongo::Timestamp)>::_M_invoke(std::_Any_data const&, mongo::Timestamp&&) (__functor=..., __args#0=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:88
      #16 0x00000b5b62fd5dd0 in std::function<mongo::StatusWith<boost::optional<mongo::Timestamp> > (mongo::Timestamp)>::operator()(mongo::Timestamp) const (__args#0=..., this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:682
      #17 mongo::WiredTigerKVEngine::getOplogNeededForRollback (this=0xb5b8b364500) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:1876
      #18 0x00000b5b62fe236c in mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run (this=0xb5b8b613500) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:351
      #19 0x00000b5b64fb0110 in mongo::BackgroundJob::jobBody (this=0xb5b8b613500) at src/mongo/util/background.cpp:148
      #20 0x00000b5b65377694 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #21 0x00007e9857b7885c in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0
      #22 0x00007e9857a99028 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6
       
       
      Thread 16: "TimestampMonitor" (Thread 0x7e984f11ee60 (LWP 16654))
      #0  0x00007e9857b82b94 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0
      #1  0x00000b5b64fa1e68 in __gthread_cond_timedwait (__abs_timeout=0x7e984f11df18, __mutex=<optimized out>, __cond=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/ppc64le-mongodb-linux/bits/gthr-default.h:871
      #2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:178
      #3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:106
      #4  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:129
      #5  std::condition_variable::wait_for<long int, std::ratio<1, 1000000000>, mongo::CondVarLockGrantNotification::wait(mongo::Milliseconds)::<lambda()> > (__p=..., __rtime=..., __lock=<synthetic pointer>..., this=0xb5b8b34aed0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/condition_variable:156
      #6  mongo::CondVarLockGrantNotification::wait (this=this@entry=0xb5b8b34aea0, timeout=..., timeout@entry=...) at src/mongo/db/concurrency/lock_state.cpp:213
      #7  0x00000b5b64fa3eec in mongo::LockerImpl::lockComplete (this=0xb5b8b34ae00, opCtx=0x0, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:867
      #8  0x00000b5b64fa67a0 in mongo::LockerImpl::lock (this=<optimized out>, resId=..., mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.h:173
      #9  0x00000b5b64f9155c in mongo::Lock::ResourceLock::lock (this=0x7e984f11e268, mode=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:343
      #10 0x00000b5b64f9172c in mongo::Lock::GlobalLock::_enqueue (this=this@entry=0x7e984f11e258, lockMode=lockMode@entry=mongo::MODE_IS, deadline=...) at src/mongo/db/concurrency/d_concurrency.cpp:181
      #11 0x00000b5b64f918e8 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f11e258, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>, enqueueOnly=...) at src/mongo/db/concurrency/d_concurrency.cpp:158
      #12 0x00000b5b64f91974 in mongo::Lock::GlobalLock::GlobalLock (this=0x7e984f11e258, opCtx=<optimized out>, lockMode=<optimized out>, deadline=..., behavior=<optimized out>) at src/mongo/db/concurrency/d_concurrency.cpp:140
      #13 0x00000b5b62f72bcc in mongo::Lock::GlobalLock::GlobalLock (lockMode=mongo::MODE_IS, opCtx=<optimized out>, this=0x7e984f11e258) at src/mongo/db/concurrency/d_concurrency.h:197
      #14 mongo::KVStorageEngine::TimestampMonitor::<lambda(mongo::Client*)>::operator() (__closure=0xb5b8b2c5f30, client=<optimized out>) at src/mongo/db/storage/kv/kv_storage_engine.cpp:910
      #15 std::_Function_handler<void(mongo::Client*), mongo::KVStorageEngine::TimestampMonitor::startup()::<lambda(mongo::Client*)> >::_M_invoke(const std::_Any_data &, mongo::Client *&&) (__functor=..., __args#0=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:297
      #16 0x00000b5b6345b468 in std::function<void (mongo::Client*)>::operator()(mongo::Client*) const (__args#0=0xb5b90451500, this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/std_function.h:682
      #17 mongo::PeriodicRunnerImpl::PeriodicJobImpl::<lambda()>::operator() (__closure=0xb5b8b0f30f8) at src/mongo/util/periodic_runner_impl.cpp:119
      #18 std::__invoke_impl<void, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/invoke.h:60
      #19 std::__invoke<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/bits/invoke.h:95
      #20 std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > >::_M_invoke<0> (this=0xb5b8b0f30f8) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:234
      #21 std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > >::operator() (this=0xb5b8b0f30f8) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:243
      #22 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > > >::_M_run(void) (this=0xb5b8b0f30f0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.tmq/include/c++/8.2.0/thread:186
      #23 0x00000b5b65377694 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #24 0x00007e9857b7885c in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0
      #25 0x00007e9857a99028 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6
      

            Assignee:
            gregory.wlodarek@mongodb.com Gregory Wlodarek
            Reporter:
            siyuan.zhou@mongodb.com Siyuan Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: