Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3266

mongos consistently locks up distributing parallel updates to multiple shards -- cluster unusable

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Blocker - P1 Blocker - P1
    • 1.8.3
    • Affects Version/s: 1.8.1, 1.8.2, 1.8.3
    • Component/s: Concurrency, Sharding, Stability
    • Labels:
      None
    • Environment:
      centos, two machines, four shards each, 30 simultaneous updateres
    • Linux

      a data set of 120m docs in one collection is being updated to add data to each document.

      every few million records processed (mixed reads and writes, many fewer writes), mongos becomes unresponsive. this number has some down from 20m gradually until it now locks after only a few million.

      queries against each of the shard and config mongod instance show them to be responsive to requests

      queries against mongos hang indefinitely as does db.stats().

      gdb stack trace (attached) shows many threads in mongo::ChunkManager::getShardsForQuery waiting to obtain mongo::rwlock::rwlock

            Assignee:
            greg_10gen Greg Studer
            Reporter:
            rgiar Rob Giardina
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: