[SERVER-32335] Update initial sync to use new Applier interface for oplog application Created: 14/Dec/17  Updated: 30/Oct/23  Resolved: 01/Jun/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.1.1

Type: Task Priority: Major - P3
Reporter: Gregory McKeon (Inactive) Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-32332 Put SyncTail behind new Applier inter... Closed
Related
related to SERVER-34900 initial sync uses different batch lim... Closed
related to SERVER-35343 Coverity analysis defect 103593: Unch... Closed
related to SERVER-33840 extend SyncTail::MultiSyncApplyFunc t... Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 2018-03-26, Repl 2018-04-09, Repl 2018-04-23, Repl 2018-05-07, Repl 2018-05-21, Repl 2018-06-04
Participants:

 Comments   
Comment by Githook User [ 01/Jun/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 remove DataReplicatorExternalState::getNextApplierBatch() and _multiApply()
Branch: master
https://github.com/mongodb/mongo/commit/65bfd4b30ec1e66b84a5e4b5c8d53b130cf38a31

Comment by Githook User [ 01/Jun/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 InitialSyncer uses OplogApplier directly
Branch: master
https://github.com/mongodb/mongo/commit/e8f46c1649fad1f42753b2cbc203d27c5d543065

Comment by Githook User [ 01/Jun/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 add DataReplicatorExternalState::makeOplogApplier()
Branch: master
https://github.com/mongodb/mongo/commit/7d2cd76aa0c766f1564b21a7ebfd10e440ad2db9

Comment by Githook User [ 01/Jun/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 split OplogApplier into interface and implementation

This change moves SyncTail dependent code into the implementation and allows
application code to invoke oplog application logic through the OplogApplier
interface.
Branch: master
https://github.com/mongodb/mongo/commit/b4e4663603da06726762c38f6e4c70702391cb32

Comment by Githook User [ 31/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 move SyncTail::calculateBatchLimitBytes to OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/5eb286fb772a6596cec549842e4f611d1a22b21a

Comment by Githook User [ 31/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 move SyncTail::replBatchLimitBytes to OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/8184571d846ba08f625e0b779620d6ffa2e75da0

Comment by Githook User [ 31/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 move SyncTail::replBatchLimitOperations to OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/c191447505716af0219eb1c6b03b7b052ef84ddb

Comment by Githook User [ 31/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 move SyncTail::makeWriterPool to OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/392a831a351f4a8229fef5efc3d30b94abff379d

Comment by Githook User [ 30/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 remove multiInitialSyncApply

Replaces calls to SyncTail::multiApply with OplogApplier::multiApply
Branch: master
https://github.com/mongodb/mongo/commit/a3fb68c2b88493ee41793cde037636ce3f330058

Comment by Githook User [ 30/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 implement multiInitialSyncApply in terms of multiSyncApply

Extends multiSyncApply to fetch missing documents on failed updates
Branch: master
https://github.com/mongodb/mongo/commit/d2978d6af210356616c07193a29641a41f797d8b

Comment by Githook User [ 29/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 multiInitialSyncApply() groups inserts
Branch: master
https://github.com/mongodb/mongo/commit/a185c9159702d417c7ec50e69e51a2697bf1a9f3

Comment by Githook User [ 29/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 make variable names in multiInitialSyncApply consistent with multiSyncApply
Branch: master
https://github.com/mongodb/mongo/commit/523ae427bd3e1ec9c3400720cb1bdd9585f631b5

Comment by Githook User [ 25/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 idempotency test ops should be applied individually by SyncTailTest::runOpsInitialSync()
Branch: master
https://github.com/mongodb/mongo/commit/1c8738bf9ba8ffe274191fe0633f8a7498e6a42d

Comment by Githook User [ 25/May/18 ]

Author:

{'username': 'benety', 'name': 'Benety Goh', 'email': 'benety@mongodb.com'}

Message: SERVER-32335 replace SyncTail::setHostname with OplogApplier option
Branch: master
https://github.com/mongodb/mongo/commit/dea326f41fbca28ca83f881bff1591b0f95ed645

Comment by Githook User [ 16/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'username': 'benety', 'name': 'Benety Goh'}

Message: SERVER-32335 remove unused function ReplicationCoordinatorExternalState::getInitialSyncBatchLimits()
Branch: master
https://github.com/mongodb/mongo/commit/833be196169816ed1859510234e886b4320365d4

Comment by Githook User [ 16/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'username': 'benety', 'name': 'Benety Goh'}

Message: SERVER-32335 remove unused InitialSyncerOptions::batchLimits
Branch: master
https://github.com/mongodb/mongo/commit/63d721812d0d72c31d1144bbe7e46f53041d3733

Comment by Githook User [ 16/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'username': 'benety', 'name': 'Benety Goh'}

Message: SERVER-32335 remove unused batchLimits argument from DataReplicatorExternalState::getNextApplierBatch()
Branch: master
https://github.com/mongodb/mongo/commit/cf339b8a8d8708e8b28747fe0cafee7cc79fe9a6

Comment by Githook User [ 15/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'username': 'benety', 'name': 'Benety Goh'}

Message: SERVER-32335 make initial sync batch limits internal to getNextApplierBatch()

Stop using batch limits provided by the caller.
Branch: master
https://github.com/mongodb/mongo/commit/43d15d7bf913939321905ae0439fa8741a1deb9f

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 make SyncTail::OpQueue accept the maximum number of ops in a batch.

This makes it unnecessary to reference the static variable replBatchLimitOperations
at construction.
Branch: master
https://github.com/mongodb/mongo/commit/a9ff8390522ee74d41141235b977c41c166aa38e

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 initial sync calls common batching logic in OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/e31a40ab59fb5c784ff8d15f07c0b811bd54a516

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 add DataReplicatorExternalState::getNextApplierBatch()

This is a passthrough for OplogApplier::getNextApplierBatch().
Branch: master
https://github.com/mongodb/mongo/commit/89f2f0843b58ea5ac67e7edc3bf2fc0275f76513

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 add OplogApplier::getNextApplierBatch()
Branch: master
https://github.com/mongodb/mongo/commit/8ae5bbab804a525d445c052be962cbc957a0330c

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 move BatchLimits from SyncTail to OplogApplier
Branch: master
https://github.com/mongodb/mongo/commit/07ae6a93bf517643adcf4098e369f49e28f0bf99

Comment by Githook User [ 09/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 initial sync includes applyOps commands with other CRUD ops when reading from oplog buffer
Branch: master
https://github.com/mongodb/mongo/commit/117e1911afaa799071b5e02ce363f793645d5654

Comment by Githook User [ 03/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 remove unnecessary applyOperationFn constructor argument from MultiApplier

DataReplicator::multiApply() stopped requiring this function since commit d6470f917762315ab93b2f7719da78cafdb8d8bf
Branch: master
https://github.com/mongodb/mongo/commit/feadc837b1e94efaa769313641f37ce2c98bea25

Comment by Githook User [ 01/May/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 InitialSyncer::_getNextApplierBatch_inlock() no longer needs to handle oplog entries on system.indexes
Branch: master
https://github.com/mongodb/mongo/commit/e5f4331b710b0c933699e26e8c65f702231038cc

Comment by Githook User [ 23/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 remove unused initial sync functions from ReplicationCoordinatorExternalState
Branch: master
https://github.com/mongodb/mongo/commit/207bcfacd612dc1ddbe98c524624658cb54e4346

Comment by Githook User [ 23/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 move initial sync oplog buffer settings from ReplicationCoordinatorExternalStateImpl to DataReplicatorExternalStateImpl
Branch: master
https://github.com/mongodb/mongo/commit/c991b4cd27e3cde813bfe863494c531278264e27

Comment by Githook User [ 22/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 DataReplicatorExternalStateImpl::_multiApply calls SyncTail directly
Branch: master
https://github.com/mongodb/mongo/commit/ac11c78003b63696ac949995827ed099c8be654d

Comment by Githook User [ 22/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 remove unused initial sync functions from DataReplicatorExternalState and ReplicationCoordinatorExternalState

remove MultiApplier::ApplyOperationFn argument from ReplicationCoordinatorExternalState::multiApply
remove unused ApplyOperationFn argument from DataReplicatorExternalState::multiApply
remove unused function ReplicationCoordinatorExternalState::multiInitialSyncApply
remove unused function DataReplicatorExternalState::_multiInitialSyncApply
Branch: master
https://github.com/mongodb/mongo/commit/d6470f917762315ab93b2f7719da78cafdb8d8bf

Comment by Githook User [ 22/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 ReplicationCoordinatorExternalStateImpl::multiApply always uses repl::multiInitialSyncApply for writer thread function
Branch: master
https://github.com/mongodb/mongo/commit/b90e530039593357d49f8f5bc7e431a667804000

Comment by Githook User [ 22/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 extend InitialSyncer external state function multiApply function to accept ThreadPool, sync source and OplogApplier::Observer

This commit has no functional changes.
Branch: master
https://github.com/mongodb/mongo/commit/8c99a68fa8bd95048e5c635539a0cacd565ce419

Comment by Githook User [ 20/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 SyncTail increments fetched document counter only when a document is successfully fetched
Branch: master
https://github.com/mongodb/mongo/commit/356d294b7afa0c665cdc580250df6859cb28d6fc

Comment by Githook User [ 20/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 multiInitialSyncApply stops accepting fetchCount

Callers wishing to track the documents fetched during initial sync should provide
an OplogApplier::Observer.

This allows us to pass this function to SyncTail without wrapping it in a lambda.
Branch: master
https://github.com/mongodb/mongo/commit/1345fc4e0f7915d6904e3cb7484c681e551b185a

Comment by Githook User [ 19/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 ReplicationCoordinatorExternalStateImpl uses OplogApplier::Observer to increment fetched document count
Branch: master
https://github.com/mongodb/mongo/commit/33da87d9b0f5dd6f1b2c8d454b22345a1d0f1d4e

Comment by Githook User [ 19/Mar/18 ]

Author:

{'email': 'benety@mongodb.com', 'name': 'Benety Goh', 'username': 'benety'}

Message: SERVER-32335 SyncTail notifies OplogApplier::Observer on fetching and inserting document successfully
Branch: master
https://github.com/mongodb/mongo/commit/d9e69b569f4d5e46973640cc658001435e1685c1

Generated at Thu Feb 08 04:29:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.