Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35770

Running a multi-statement transaction when all WiredTiger write tickets are exhausted may lead to deadlock

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 4.0.0
    • Fix Version/s: 4.0.2, 4.1.2
    • Component/s: Replication
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.0
    • Sprint:
      Repl 2018-07-16, Repl 2018-07-30, Repl 2018-08-13
    • Linked BF Score:
      62

      Description

      Unstashing a transaction's lock state also involves reacquiring a WiredTiger write ticket. The call to Locker::reacquireTicket() has no lock timeout and occurs while holding the Session::_mutex. Since the killAllExpiredTransactions() function calls Session::abortArbitraryTransactionIfExpired() which attempts to acquire the Session::_mutex, it isn't possible to reap expired transactions while a transaction's lock state is being unstashed and no WiredTiger write tickets are available.

      Thread 36: "conn1" (Thread 0x7f8e579ef700 (LWP 14971))
      #0  sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
      #1  0x00007f8e595b861e in mongo::TicketHolder::waitForTicketUntil (this=<optimized out>, opCtx=<optimized out>, until=...) at src/mongo/util/concurrency/ticketholder.cpp:104
      #2  0x00007f8e595b8673 in mongo::TicketHolder::waitForTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/util/concurrency/ticketholder.cpp:90
      #3  0x00007f8e5951eacd in mongo::LockerImpl<false>::_acquireTicket (this=<optimized out>, opCtx=<optimized out>, mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:338
      #4  0x00007f8e5951eb1d in mongo::LockerImpl<false>::reacquireTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/concurrency/lock_state.cpp:322
      #5  0x00007f8e592fd06f in mongo::Session::TxnResources::release (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/session.cpp:658
      #6  0x00007f8e593021e3 in mongo::Session::unstashTransactionResources (this=<optimized out>, opCtx=<optimized out>, cmdName=...) at src/mongo/db/session.cpp:775
      #7  0x00007f8e584adc80 in mongo::(anonymous namespace)::invokeInTransaction (opCtx=<optimized out>, invocation=<optimized out>, replyBuilder=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:466
      #8  0x00007f8e584b0dd9 in runCommandImpl (sessionOptions=..., extraFieldsBuilder=<optimized out>, behaviors=..., startOperationTime=..., replyBuilder=<optimized out>, request=..., invocation=<optimized out>, opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:537
      #9  mongo::(anonymous namespace)::execCommandDatabase (opCtx=<optimized out>, command=<optimized out>, request=..., replyBuilder=<optimized out>, behaviors=...) at src/mongo/db/service_entry_point_common.cpp:873
      #10 0x00007f8e584b2291 in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1018
      #11 0x00007f8e584b2291 in mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...)
      #12 0x00007f8e584b31d1 in runCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1033
      #13 mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1307
      #14 0x00007f8e584a09da in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_mongod.cpp:123
      #15 0x00007f8e584ab76a in mongo::ServiceStateMachine::_processMessage (this=<optimized out>, guard=...) at src/mongo/transport/service_state_machine.cpp:378
      ...
      #36 mongo::(anonymous namespace)::runFunc (ctx=<optimized out>) at src/mongo/transport/service_entry_point_utils.cpp:55
      #37 0x00007f8e56844184 in start_thread (arg=0x7f8e579ef700) at pthread_create.c:312
      #38 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      

      Thread 39: "startPe.actions" (Thread 0x7f8e4551d700 (LWP 14968))
      #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1  0x00007f8e56846649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2  0x00007f8e56846470 in __GI___pthread_mutex_lock (mutex=0x7f8e5f6f8080) at ../nptl/pthread_mutex_lock.c:79
      #3  0x00007f8e592fdcc7 in __gthread_mutex_lock (__mutex=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/x86_64-mongodb-linux/bits/gthr-default.h:748
      #4  lock (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:135
      #5  lock_guard (__m=..., this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:386
      #6  mongo::Session::abortArbitraryTransactionIfExpired (this=<optimized out>) at src/mongo/db/session.cpp:858
      #7  0x00007f8e58a90d7b in mongo::<lambda(mongo::OperationContext*, mongo::Session*)>::operator()(mongo::Session *, mongo::OperationContext *) (session=<optimized out>, opCtx=<optimized out>, __closure=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:76
      #8  0x00007f8e59306504 in operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267
      #9  mongo::SessionCatalog::scanSessions(mongo::OperationContext*, mongo::SessionKiller::Matcher const&, std::function<void (mongo::OperationContext*, mongo::Session*)>) (this=<optimized out>, opCtx=<optimized out>, matcher=..., workerFn=...) at src/mongo/db/session_catalog.cpp:208
      #10 0x00007f8e58a911b8 in mongo::killAllExpiredTransactions (opCtx=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:96
      #11 0x00007f8e5881041f in operator() (__closure=<optimized out>, client=<optimized out>) at src/mongo/db/periodic_runner_job_abort_expired_transactions.cpp:91
      #12 std::_Function_handler<void(mongo::Client*), mongo::startPeriodicThreadToAbortExpiredTransactions(mongo::ServiceContext*)::<lambda(mongo::Client*)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/maxh/debugging/mongo/mongod, CU 0x0, DIE 0x316c7>) (__functor=..., __args#0=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871
      #13 0x00007f8e5923d47c in operator() (__args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267
      #14 operator() (__closure=<optimized out>) at src/mongo/util/periodic_runner_impl.cpp:108
      #15 _M_invoke<> (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531
      #16 operator() (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520
      #17 std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::run()::<lambda()>()> >::_M_run(void) (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115
      #18 0x00007f8e59e680d0 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84
      #19 0x00007f8e56844184 in start_thread (arg=0x7f8e4551d700) at pthread_create.c:312
      #20 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              matthew.russotto Matthew Russotto
              Reporter:
              max.hirschhorn Max Hirschhorn
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: