[SERVER-14041] enhance secondaryThrottle parameter Created: 23/May/14  Updated: 27/Oct/15  Resolved: 15/Jul/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 2.7.4

Type: Improvement Priority: Major - P3
Reporter: Greg Studer Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-14708 A mixed single node & replicaset set ... Closed
Related
related to SERVER-15591 enforce secondaryThrottle during move... Closed
is related to SERVER-16842 moveChunk with replication is slow wi... Closed
is related to SERVER-14708 A mixed single node & replicaset set ... Closed
is related to SERVER-16548 Allow enforcement of additional custo... Closed
is related to DOCS-4716 moveChunk _secondaryThrottle argument... Closed
is related to DOCS-4318 Document secondaryThrottle improvements Closed
is related to SERVER-14567 Implement checkIfWriteConcernCanBeSat... Closed
is related to SERVER-16357 Chunk migration pre-commit write conc... Closed
Tested
Participants:

 Description   

For clusters with significant load on secondary nodes of replica set shards, the default secondary throttle mode (w:1) may not ensure that chunk migration doesn't interfere with all secondaries. Support other write concern "w" modes (1, 2, "majority") for this parameter to allow configurable throttling. Also the default is changed to { _secondaryThrottle: true } for chunk migration.



 Comments   
Comment by Randolph Tan [ 16/Jul/14 ]

Tested by hand.

Scenarios tested:
1. Migration -
a. no _secondaryThrottle
2. secondaryThrottle: true
3. secondaryThrottle: false
4. 2. secondaryThrottle: BSON

2. MoveChunk command -
a. no secondaryThrottle, no writeConcern.
b. no secondaryThrottle, with writeConcern.
c. secondaryThrottle: true, no writeConcern.
d. secondaryThrottle: true, with writeConcern.
e. secondaryThrottle: false, no writeConcern.
f. secondaryThrottle: false, with writeConcern.

3. cleanupOrphaned - same as #2.

Note that for test 2 & 3, make sure that mongos sends the right parameter when sending the right parameters for moveChunk and make sure the shard does the same way for _recvChunkStart (must have at least vebose level 1). To make sure that the write concern was passed down correctly by checking the logs for "begin removal of" (requires log level 1).

Comment by Randolph Tan [ 16/Jul/14 ]

This patch allows the user to specify a more detailed write concern mode compared to the on/off switch before. This currently only supports the "w" and "wtimeout" parameters (fsync and j are ignored). The places where this are supported are in the balancer config, moveChunk command (as well as the moveChunk shard version and _recvChunkStart) and cleanupOrphaned.

Below are a list of examples on how to pass the setting. Note: that the balancer config document 'overloads' the _secondaryThrottle field, whereas for commands, the new writeConcern field should be used.

{ _id: 'balancer', _secondaryThrottle: { w: 2, wtimeout: 1000 }}
{ moveChunk: 'test.user', ..., secondaryThrottle: true, writeConcern: { w: 2, wtimeout: 1000 }} // secondaryThrottle is optional
{ cleanupOrphaned: 'test.user', ..., secondaryThrottle: true, writeConcern: { w: 2, wtimeout: 1000 }} // secondaryThrottle is optional

This patch also make _secondaryThrottle and secondaryThrottle interchangable for commands (again, not for the config doc). In the case when both of them are specified, secondaryThrottle overrides _secondaryThrottle.

The migration process involves a lot of write operations and here's a list of all the places where mongod waits for replication during migration:

From side:
1. RangeDeleter deletes during migration cleanup.

To side:
1. Individual deletes on the incoming range at the start of migration.
2. RangeDeleter deletes when a migration aborts.
3. Individual writes when cloning documents.
4. Individual writes during transferMod. { w: majority, wtimeout: -1 }
5. After transferMod phase is completed. { w: majority } -> loops up to 10 hours.

RangeDeleter waits for replication on:
1. Individual deletes.
2. After the delete completes. { w: majority }

If the user explicitly passes a custom write concern, all items not marked with { w: majority } will use it. In other words, the cases where the server waits for majority is not affected by the secondaryThrottle setting.

Notes:

  • It is not legal to specify secondaryThrottle = false and a writeConcern.
  • writeConcern of w: 1 are treated as no secondaryThrottle.
  • The default writeConcern is 2 if the replica set has enough nodes.
Comment by Githook User [ 16/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14041 enhance secondaryThrottle parameter

This reverts commit 37f2a1e3b724dbd9e1f8eafd4ac87c5bf613c048 (undo revert).
This reverts commit d60fd22dec1c0bd104622eab463cdbba18bf11a9 (undo revert).
Fix Windows compile failure.
Add more defensive checks.
Branch: master
https://github.com/mongodb/mongo/commit/2fd2eebb2656ae65bd6eb63b39f8faeb84e0db4e

Comment by Githook User [ 15/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: Revert "SERVER-14041 enhance secondaryThrottle parameter"

This reverts commit fa1233fbe4a48ef0675820f381987f1df4f42f75.
Branch: master
https://github.com/mongodb/mongo/commit/37f2a1e3b724dbd9e1f8eafd4ac87c5bf613c048

Comment by Githook User [ 15/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: Revert "SERVER-14041 enhance secondaryThrottle parameter"

This reverts commit d85f99a8a150a1932d72bd393511be2b85a409c7.
Branch: master
https://github.com/mongodb/mongo/commit/d60fd22dec1c0bd104622eab463cdbba18bf11a9

Comment by Githook User [ 15/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14041 enhance secondaryThrottle parameter

Fix debug build compile failure
Branch: master
https://github.com/mongodb/mongo/commit/d85f99a8a150a1932d72bd393511be2b85a409c7

Comment by Githook User [ 15/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14041 enhance secondaryThrottle parameter
Branch: master
https://github.com/mongodb/mongo/commit/fa1233fbe4a48ef0675820f381987f1df4f42f75

Comment by Githook User [ 15/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14041 enhance secondaryThrottle parameter
Branch: master
https://github.com/mongodb/mongo/commit/fa1233fbe4a48ef0675820f381987f1df4f42f75

Generated at Thu Feb 08 03:33:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.