[SERVER-22114] Mongos can accumulate multiple copies of ChunkManager when a shard restarts Created: 09/Jan/16 Updated: 25/Jan/17 Resolved: 15/Jan/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.0.9, 3.2.3, 3.3.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Randolph Tan | Assignee: | Randolph Tan |
| Resolution: | Done | Votes: | 0 |
| Labels: | code-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Completed: | |||||||||||||||||
| Sprint: | Sharding F (01/29/16) | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
When a shard restarts, it loses all its sharding metadata information. So if a mongos that has sharding information sends a write command to the restarted shard, it will get a stale config error from the shard because the shard contains no shard version (see note 1). The mongos will see the stale error with shard zero version from the response and will decide to perform a full reload (see note 2). In setups with very huge number of chunks (in millions), it takes time for the entire chunk metadata to be loaded and this can cause issues because this is done under a mutex, and will cause other threads which got the same response from the shard to decide to perform a full reload as well. This is exacerbated by the fact that the thread will have to acquire and release the same contentious mutex multiple times until it finishes, so it can hold on to the newly fully loaded data for even longer periods of time. Also note that for every full reload, it will create a new instance of ChunkManager and will try to atomically replace the old one with the new one when the reload finishes. In a mongos with multiple threads trying to execute a write command, it can create a situation where several threads will queue up trying to perform a full reload and some threads have loaded their own copy of the chunk metadata but are blocked waiting for the same mutex the other threads are waiting for the full reload. In certain cases with large enough chunks and simultaneous write command operations, it can spiral out of control, consume too much memory and ultimately get killed by the OOM killer in the operating system. Note 1: sets the error in the response, and then performs a refresh afterwards: Note 2: and ultimately, causing it to flush the entire chunk manager: |
| Comments |
| Comment by Githook User [ 16/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: (cherry picked from commit 0f85256eefa78b1b74b9e6ae0d1c646853f6f22f) |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier. (cherry picked from commit 65f2da2c49c2b22d2b80e6562b9b61f242cb9a18) |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: (cherry picked from commit 0f85256eefa78b1b74b9e6ae0d1c646853f6f22f) Conflicts: |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier. (cherry picked from commit 65f2da2c49c2b22d2b80e6562b9b61f242cb9a18) Conflicts: |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}Message: In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier. |