Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26018

Inconsistency between the LockManager grantedList and grantedCount

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Community Answered
    • 3.0.12, 3.2.10, 3.4.0-rc0
    • None
    • Stability
    • None
    • ALL
    • Hide

      unpredictable, seems to be produce on secondary
      (not sure, 3 times on secondary replica in two week)

      Show
      unpredictable, seems to be produce on secondary (not sure, 3 times on secondary replica in two week)
    • Sharding 2016-09-19, Sharding 2016-10-10, Sharding 2016-10-31

    Description

      All threads hang on waiting locks. The state of the LockHead indicates that there is inconsistency between the grantedList and grantedCount where there are not granted requests, but the granted counts are non-zero:

      (gdb) p $18
      $38 = {
        resourceId = {
          _fullHash = 2305843009213693953
        },
        grantedList = {
          _front = 0x0,
          _back = 0x0
        },
        grantedCounts = {0, 1, 0, 0, 0},
        grantedModes = 2,
        conflictList = {
          _front = 0x7f08028,
          _back = 0x5902cf628
        },
        conflictCounts = {0, 1490, 0, 0, 1},
        conflictModes = 18,
        partitions = {
          <std::_Vector_base<mongo::LockManager::Partition*, std::allocator<mongo::LockManager::Partition*> >> = {
            _M_impl = {
              <std::allocator<mongo::LockManager::Partition*>> = {
                <__gnu_cxx::new_allocator<mongo::LockManager::Partition*>> = {<No data fields>}, <No data fields>},
              members of std::_Vector_base<mongo::LockManager::Partition*, std::allocator<mongo::LockManager::Partition*> >::_Vector_impl:
              _M_start = 0x7470640,
              _M_finish = 0x7470640,
              _M_end_of_storage = 0x7470680
            }
          }, <No data fields>},
        conversionsCount = 0,
        compatibleFirstCount = 0
      }
      

      Attachment shows stacks of all threads and the output of db.currentOp().

      Here is our cluster info:

      • 3 shards * 3 replica
      • using both range & hash based sharding
      • collection size from 50GB to 500GB

      Attachments

        1. currentOp.out
          910 kB
        2. gdb.withsymbols.out
          6.60 MB
        3. LockMgrInvariants.diff
          14 kB
        4. server_status.txt
          24 kB
        5. stacks.txt
          6.14 MB

        Issue Links

          Activity

            People

              kaloian.manassiev@mongodb.com Kaloian Manassiev
              xiaost xiaost
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: