[SERVER-24242] Improve replication performance Created: 20/May/16  Updated: 16/Mar/17  Resolved: 11/Aug/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.3.11

Type: Improvement Priority: Major - P3
Reporter: Mathias Stearn Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-24575 Don't block writing command replies t... Backlog
Related
is related to SERVER-28315 replBatchLimitBytes should be 100MB i... Closed
is related to SERVER-24187 OplogFetcher should optimistically is... Backlog
is related to SERVER-24418 Make it possible to get owned BSONObj... Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 17 (07/15/16), Repl 18 (08/05/16), Repl 2016-08-29, Integrate+Tuning 15 (06/03/16), Integrate+Tuning 16 (06/24/16)
Participants:
Linked BF Score: 0

 Description   

Recent improvements have increased the rate that primary nodes can accept writes. We need to increase secondary performance to ensure that they can keep up.



 Comments   
Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Raise replication op batch limit to 50,000

This improves performance by ~10% in the small document cases.

Also lowering the byte limit down to its originally intended 100MB. When we
raised MMAPv1's maximum journal commit size, that unintentionally raised the
repl byte limiti to 512MB. However, it makes no sense for the batch limit to
be larger than the default BGSync blocking queue size of 256MB.
Branch: master
https://github.com/mongodb/mongo/commit/b53a93270c8ea8ccc9f43a0cf0592353b0edd012

Comment by Githook User [ 01/Aug/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Make replica set apply batch size tunable
Branch: master
https://github.com/mongodb/mongo/commit/b417c39b6ed549a02f2629c6175122eb279e2899

Comment by Githook User [ 01/Aug/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Lock::ParallelBatchWriterMode sets isBatchWriter while in scope
Branch: master
https://github.com/mongodb/mongo/commit/af042cfeda17768fa217207069880fbbcb0346ff

Comment by Githook User [ 01/Aug/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 MultiApplier should pass a clean txn to _multiApply
Branch: master
https://github.com/mongodb/mongo/commit/b0f766dfaf65fddf341e425b5a842ec27a5eff8f

Comment by Ian Whalen (Inactive) [ 17/Jul/16 ]

re-opening since one of the commits was reverted - assuming you'll want to fix it and re-push.

Comment by Githook User [ 16/Jul/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: Revert "SERVER-24242 Lock::ParallelBatchWriterMode sets isBatchWriter while in scope"

This reverts commit 0673f9531be7cbaa8e5477ee28431a28191fe7ed.
Branch: master
https://github.com/mongodb/mongo/commit/3b371e48fab6cd5b022415f5352080e6e268b9e0

Comment by Mathias Stearn [ 16/Jul/16 ]

I'm done with apply-side improvements so I'm closing out this ticket. Some additional network-side improvements will continue under the linked tickets.

Comment by Githook User [ 16/Jul/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Parallelize oplog writing on secondaries
Branch: master
https://github.com/mongodb/mongo/commit/dcb0d3fe6429db310307122e640196d7bd88b5eb

Comment by Githook User [ 16/Jul/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Start writing the Oplog on secondaries while divvying up work

This means we hold the PBW lock while in fillWriterVectors, but it also lets
us start doing useful work sooner.
Branch: master
https://github.com/mongodb/mongo/commit/29a2e0352d8ac20a7e92d27a1daa7a439cad58d6

Comment by Githook User [ 16/Jul/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Lock::ParallelBatchWriterMode sets isBatchWriter while in scope
Branch: master
https://github.com/mongodb/mongo/commit/0673f9531be7cbaa8e5477ee28431a28191fe7ed

Comment by Githook User [ 23/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Stop applying the MatchExpression to OplogReplay queries after first match
Branch: master
https://github.com/mongodb/mongo/commit/8a53668a313062adb9b4af48f5d76716e2ca58c8

Comment by Githook User [ 22/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Don't create a vector<BSONElement> just to iterate
Branch: master
https://github.com/mongodb/mongo/commit/27a563f0c7adddebc6354dd6591e0849c86eaf64

Comment by Githook User [ 22/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 OldThreadPool::schedule() should move its task argument
Branch: master
https://github.com/mongodb/mongo/commit/d74b549a87899dbd5fe347a1a08cdab0536daad2

Comment by Githook User [ 22/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Only get slaveDelaySecs from the replCoordinator once per batch
Branch: master
https://github.com/mongodb/mongo/commit/f9f3ea2d4afb835b82fe7e97ef1a1c6e2a29c2b5

Comment by Githook User [ 22/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Use FastClockSource when enforcing replBatchLimitSeconds

It only needs second-level precision anyway.
Branch: master
https://github.com/mongodb/mongo/commit/402b4573153c129263670b9db0ad6a2d7864a9d0

Comment by Githook User [ 22/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Stop copying OplogEntries so much

With this change, we should no longer be copying any OplogEntries during
steady-state replication.
Branch: master
https://github.com/mongodb/mongo/commit/9fe9ddc29177f1929884f00cf9220fbcd5d932f4

Comment by Githook User [ 09/Jun/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-24242 Make BSONObj nothow_move_constructible
Branch: master
https://github.com/mongodb/mongo/commit/54a293b576d22775a57b0809a57b920283c436da

Generated at Thu Feb 08 04:05:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.