Details
-
Bug
-
Resolution: Fixed
-
Critical - P2
-
4.0.0
-
None
-
Fully Compatible
-
ALL
-
v4.0
-
Repl 2018-07-16, Repl 2018-07-30, Repl 2018-08-13
-
62
Description
Unstashing a transaction's lock state also involves reacquiring a WiredTiger write ticket. The call to Locker::reacquireTicket() has no lock timeout and occurs while holding the Session::_mutex. Since the killAllExpiredTransactions() function calls Session::abortArbitraryTransactionIfExpired() which attempts to acquire the Session::_mutex, it isn't possible to reap expired transactions while a transaction's lock state is being unstashed and no WiredTiger write tickets are available.
Thread 36: "conn1" (Thread 0x7f8e579ef700 (LWP 14971))
|
#0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
|
#1 0x00007f8e595b861e in mongo::TicketHolder::waitForTicketUntil (this=<optimized out>, opCtx=<optimized out>, until=...) at src/mongo/util/concurrency/ticketholder.cpp:104
|
#2 0x00007f8e595b8673 in mongo::TicketHolder::waitForTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/util/concurrency/ticketholder.cpp:90
|
#3 0x00007f8e5951eacd in mongo::LockerImpl<false>::_acquireTicket (this=<optimized out>, opCtx=<optimized out>, mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:338
|
#4 0x00007f8e5951eb1d in mongo::LockerImpl<false>::reacquireTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/concurrency/lock_state.cpp:322
|
#5 0x00007f8e592fd06f in mongo::Session::TxnResources::release (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/session.cpp:658
|
#6 0x00007f8e593021e3 in mongo::Session::unstashTransactionResources (this=<optimized out>, opCtx=<optimized out>, cmdName=...) at src/mongo/db/session.cpp:775
|
#7 0x00007f8e584adc80 in mongo::(anonymous namespace)::invokeInTransaction (opCtx=<optimized out>, invocation=<optimized out>, replyBuilder=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:466
|
#8 0x00007f8e584b0dd9 in runCommandImpl (sessionOptions=..., extraFieldsBuilder=<optimized out>, behaviors=..., startOperationTime=..., replyBuilder=<optimized out>, request=..., invocation=<optimized out>, opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:537
|
#9 mongo::(anonymous namespace)::execCommandDatabase (opCtx=<optimized out>, command=<optimized out>, request=..., replyBuilder=<optimized out>, behaviors=...) at src/mongo/db/service_entry_point_common.cpp:873
|
#10 0x00007f8e584b2291 in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1018
|
#11 0x00007f8e584b2291 in mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...)
|
#12 0x00007f8e584b31d1 in runCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1033
|
#13 mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1307
|
#14 0x00007f8e584a09da in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_mongod.cpp:123
|
#15 0x00007f8e584ab76a in mongo::ServiceStateMachine::_processMessage (this=<optimized out>, guard=...) at src/mongo/transport/service_state_machine.cpp:378
|
...
|
#36 mongo::(anonymous namespace)::runFunc (ctx=<optimized out>) at src/mongo/transport/service_entry_point_utils.cpp:55
|
#37 0x00007f8e56844184 in start_thread (arg=0x7f8e579ef700) at pthread_create.c:312
|
#38 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
Thread 39: "startPe.actions" (Thread 0x7f8e4551d700 (LWP 14968))
|
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
|
#1 0x00007f8e56846649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#2 0x00007f8e56846470 in __GI___pthread_mutex_lock (mutex=0x7f8e5f6f8080) at ../nptl/pthread_mutex_lock.c:79
|
#3 0x00007f8e592fdcc7 in __gthread_mutex_lock (__mutex=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/x86_64-mongodb-linux/bits/gthr-default.h:748
|
#4 lock (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:135
|
#5 lock_guard (__m=..., this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:386
|
#6 mongo::Session::abortArbitraryTransactionIfExpired (this=<optimized out>) at src/mongo/db/session.cpp:858
|
#7 0x00007f8e58a90d7b in mongo::<lambda(mongo::OperationContext*, mongo::Session*)>::operator()(mongo::Session *, mongo::OperationContext *) (session=<optimized out>, opCtx=<optimized out>, __closure=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:76
|
#8 0x00007f8e59306504 in operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267
|
#9 mongo::SessionCatalog::scanSessions(mongo::OperationContext*, mongo::SessionKiller::Matcher const&, std::function<void (mongo::OperationContext*, mongo::Session*)>) (this=<optimized out>, opCtx=<optimized out>, matcher=..., workerFn=...) at src/mongo/db/session_catalog.cpp:208
|
#10 0x00007f8e58a911b8 in mongo::killAllExpiredTransactions (opCtx=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:96
|
#11 0x00007f8e5881041f in operator() (__closure=<optimized out>, client=<optimized out>) at src/mongo/db/periodic_runner_job_abort_expired_transactions.cpp:91
|
#12 std::_Function_handler<void(mongo::Client*), mongo::startPeriodicThreadToAbortExpiredTransactions(mongo::ServiceContext*)::<lambda(mongo::Client*)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/maxh/debugging/mongo/mongod, CU 0x0, DIE 0x316c7>) (__functor=..., __args#0=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871
|
#13 0x00007f8e5923d47c in operator() (__args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267
|
#14 operator() (__closure=<optimized out>) at src/mongo/util/periodic_runner_impl.cpp:108
|
#15 _M_invoke<> (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531
|
#16 operator() (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520
|
#17 std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::run()::<lambda()>()> >::_M_run(void) (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115
|
#18 0x00007f8e59e680d0 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84
|
#19 0x00007f8e56844184 in start_thread (arg=0x7f8e4551d700) at pthread_create.c:312
|
#20 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
Attachments
Issue Links
- is related to
-
SERVER-35217 killSessions command attempts to kill a transaction while holding SessionCatalog::_mutex, which leads to deadlock
-
- Closed
-
- related to
-
SERVER-43868 Session::TxnResources::release() can throw exceptions in destructors (4.0.x only)
-
- Closed
-
-
SERVER-39320 lower wiredTigerConcurrentWriteTransactions to 64 in concurrency_simultaneous suites
-
- Closed
-