[SERVER-22114] Mongos can accumulate multiple copies of ChunkManager when a shard restarts Created: 09/Jan/16  Updated: 25/Jan/17  Resolved: 15/Jan/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.0.9, 3.2.3, 3.3.1

Type: Bug Priority: Major - P3
Reporter: Randolph Tan Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: code-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-23958 DBConfig::_loadIfNeeded will not do a... Closed
related to SERVER-23965 DBConfig::getChunkManager should not ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Sharding F (01/29/16)
Participants:

 Description   

When a shard restarts, it loses all its sharding metadata information. So if a mongos that has sharding information sends a write command to the restarted shard, it will get a stale config error from the shard because the shard contains no shard version (see note 1). The mongos will see the stale error with shard zero version from the response and will decide to perform a full reload (see note 2). In setups with very huge number of chunks (in millions), it takes time for the entire chunk metadata to be loaded and this can cause issues because this is done under a mutex, and will cause other threads which got the same response from the shard to decide to perform a full reload as well. This is exacerbated by the fact that the thread will have to acquire and release the same contentious mutex multiple times until it finishes, so it can hold on to the newly fully loaded data for even longer periods of time. Also note that for every full reload, it will create a new instance of ChunkManager and will try to atomically replace the old one with the new one when the reload finishes. In a mongos with multiple threads trying to execute a write command, it can create a situation where several threads will queue up trying to perform a full reload and some threads have loaded their own copy of the chunk metadata but are blocked waiting for the same mutex the other threads are waiting for the full reload. In certain cases with large enough chunks and simultaneous write command operations, it can spiral out of control, consume too much memory and ultimately get killed by the OOM killer in the operating system.

Note 1:
In mongod write command execution path, version is checked first inside here:
https://github.com/mongodb/mongo/blob/r3.2.0/src/mongo/db/commands/write_commands/batch_executor.cpp#L321

sets the error in the response, and then performs a refresh afterwards:
https://github.com/mongodb/mongo/blob/r3.2.0/src/mongo/db/commands/write_commands/batch_executor.cpp#L361

Note 2:
Shard response with zero version will result to unknown comparison result:
https://github.com/mongodb/mongo/blob/r3.2.0/src/mongo/s/chunk_manager_targeter.cpp#L152

and ultimately, causing it to flush the entire chunk manager:
https://github.com/mongodb/mongo/blob/r3.2.0/src/mongo/s/chunk_manager_targeter.cpp#L674



 Comments   
Comment by Githook User [ 16/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 fix code format
Branch: v3.0
https://github.com/mongodb/mongo/commit/4946e2dd15dc97d0c884d04f63a8f9470839786f

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Avoid doing unnecessary repeated calls to DBConfig::_load

(cherry picked from commit 0f85256eefa78b1b74b9e6ae0d1c646853f6f22f)
Branch: v3.2
https://github.com/mongodb/mongo/commit/d74c7ae40f82bd5bf6334c8114c0cf4948f60002

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Remove unnecessary refresh in ChunkManagerTargeter::refreshNow

In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier.

(cherry picked from commit 65f2da2c49c2b22d2b80e6562b9b61f242cb9a18)
Branch: v3.2
https://github.com/mongodb/mongo/commit/927caf6e06d77adda6caa23120fc08b20059151b

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Avoid doing unnecessary repeated calls to DBConfig::_load

(cherry picked from commit 0f85256eefa78b1b74b9e6ae0d1c646853f6f22f)

Conflicts:
src/mongo/s/config.cpp
src/mongo/s/config.h
Branch: v3.0
https://github.com/mongodb/mongo/commit/aaa56a5d37ee1ad78326a2e22e097bdf91bedeed

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Remove unnecessary refresh in ChunkManagerTargeter::refreshNow

In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier.

(cherry picked from commit 65f2da2c49c2b22d2b80e6562b9b61f242cb9a18)

Conflicts:
src/mongo/s/chunk_manager_targeter.cpp
src/mongo/s/chunk_manager_targeter.h
src/mongo/s/cluster_write.cpp
src/mongo/s/cluster_write.h
src/mongo/s/commands/cluster_write_cmd.cpp
src/mongo/s/write_ops/batch_write_exec.cpp
src/mongo/s/write_ops/batch_write_exec.h
src/mongo/s/write_ops/batch_write_exec_test.cpp
Branch: v3.0
https://github.com/mongodb/mongo/commit/56eae69cf882d92e502340c9ce5dcc4a7058db9a

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Fix comment
Branch: master
https://github.com/mongodb/mongo/commit/7407700466ea8f2f281c3daad50e7a2b42d4f460

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Avoid doing unnecessary repeated calls to DBConfig::_load
Branch: master
https://github.com/mongodb/mongo/commit/0f85256eefa78b1b74b9e6ae0d1c646853f6f22f

Comment by Githook User [ 15/Jan/16 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-22114 Remove unnecessary refresh in ChunkManagerTargeter::refreshNow

In addition, limit the lifetime of ChunkManager instances so it can free references to ChunkManager earlier.
Branch: master
https://github.com/mongodb/mongo/commit/65f2da2c49c2b22d2b80e6562b9b61f242cb9a18

Generated at Thu Feb 08 03:59:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.