[SERVER-39239] Two-phase index builds on secondaries will wait for the commitIndexBuild oplog entry before committing. Created: 28/Jan/19  Updated: 29/Oct/23  Resolved: 23/Sep/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.3.1

Type: Task Priority: Major - P3
Reporter: Dianna Hohensee (Inactive) Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-43192 fix checks for missing abortIndexBuil... Closed
depends on SERVER-38605 IndexBuildsCoordinator writes commitI... Closed
depends on SERVER-39068 Replication of simultaneous index bui... Closed
depends on SERVER-39533 Hook up abortIndexBuild such that pri... Closed
depends on SERVER-42760 emit no-op commitIndexBuild oplog ent... Closed
depends on SERVER-42762 recover to timestamp should ignore no... Closed
depends on SERVER-42763 rollback via refetch should ignore no... Closed
depends on SERVER-42764 improve test assertion message in two... Closed
depends on SERVER-43023 emit no-op startIndexBuild oplog entr... Closed
depends on SERVER-43024 secondary oplog application should ig... Closed
depends on SERVER-43025 rollback should ignore no-op startInd... Closed
depends on SERVER-43026 AuthenticationManager should handle s... Closed
depends on SERVER-43190 rollback should ignore no-op abortInd... Closed
depends on SERVER-43191 AuthenticationManager should handle a... Closed
depends on SERVER-43295 Secondaries should start index build ... Closed
is depended on by TOOLS-2374 Support 4.4 index build oplog entries Closed
is depended on by SERVER-39458 Add continuous draining on secondary'... Closed
is depended on by SERVER-39462 Enhance index build logic on secondar... Closed
is depended on by SERVER-43301 add no_passthrough to two phase index... Closed
Problem/Incident
Related
related to SERVER-43612 secondary fails to start index build ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Storage NYC 2019-03-11, Execution Team 2019-09-23
Participants:
Linked BF Score: 31
Story Points: 5

 Description   

The secondary's index build thread should spin in a loop before finishing up and committing the index.

The secondary will do nothing while spinning in a loop. It should drop all locks. SERVER-39458 will add functionality to the loop, of periodically reacquiring the lock and running side table draining.

On receipt of the commitIndexBuild oplog entry, the secondary should signal the index build to commit with a timestamp passed in the oplog entry. The oplog applier thread will need to drop locks to allow the index build to take the X lock for index commit – I believe this should be safe, since 'c' oplog entries are applied serially, and so dropping the lock is OK. IndexBuildsCoordinator::commitIndexBuild will need to fetch a Future from the index build to return to the oplog applier thread to wait upon for commit completion.

SERVER-39533 will be done after this ticket and will hook up the abortIndexBuild oplog entry so that alternatively the index build on the secondary can be aborted while it's spinning in that loop waiting for commit. In case that's a design consideration.

Do not try to act on abort/commit signals earlier in the index build on secondaries than that spinning loop that waits for commit/abort. This is the simple first iteration implementation, we'll make it fancier in a subsequent ticket.



 Comments   
Comment by Githook User [ 22/Sep/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39239 secondary index build waits for commitIndexBuild or abortIndexBuild
Branch: master
https://github.com/mongodb/mongo/commit/1d310fa818fd8b0357e7ac7d0c0edcbe46b37772

Comment by Githook User [ 21/Sep/19 ]

Author:

{'username': 'benety', 'email': 'benety@mongodb.com', 'name': 'Benety Goh'}

Message: SERVER-39239 IndexBuild() IndexBuildsCoordinator::commitIndexBuild() updates commit information in ReplIndexBuildState
Branch: master
https://github.com/mongodb/mongo/commit/56f001f2ec7499531690857f2846d64e6a01dd6f

Comment by Githook User [ 21/Sep/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39239 IndexBuildsCoordinator::abortIndexBuildByBuildUUID() updates abort information in ReplIndexBuildState
Branch: master
https://github.com/mongodb/mongo/commit/e90239daa9ca865e178fa3982b880cf5517e1c23

Comment by Githook User [ 21/Sep/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39239 add js test for commitIndexBuild handling on secondaries
Branch: master
https://github.com/mongodb/mongo/commit/0ab1e3d4a143625ea4fbf5033eb869994d8b1fb0

Comment by Githook User [ 21/Sep/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39239 move IndexBuildsCoordinatorMongod::commitIndexBuild() into base class and make non-virtual
Branch: master
https://github.com/mongodb/mongo/commit/b9474feee8945304cccacd3b18c2d3a46e427176

Comment by Githook User [ 21/Sep/19 ]

Author:

{'username': 'benety', 'email': 'benety@mongodb.com', 'name': 'Benety Goh'}

Message: SERVER-39239 IndexBuildsCoordinator::_buildIndex() accepts indexBuildOptions
Branch: master
https://github.com/mongodb/mongo/commit/0fb8601535e37e5fedbcec4649502a01de80f00a

Comment by Githook User [ 20/Sep/19 ]

Author:

{'username': 'benety', 'email': 'benety@mongodb.com', 'name': 'Benety Goh'}

Message: SERVER-39239 IndexBuildsCoordinator::abortIndexBuildByBuildUUID() accepts opCtx
Branch: master
https://github.com/mongodb/mongo/commit/5e282411b35eae601e9e1d52b3030957528246ae

Comment by Githook User [ 20/Sep/19 ]

Author:

{'username': 'benety', 'email': 'benety@mongodb.com', 'name': 'Benety Goh'}

Message: SERVER-39239 add fields to ReplIndexBuildState to support commitIndexBuild and abortIndexBuild
Branch: master
https://github.com/mongodb/mongo/commit/43b3d71af00922f7ebc1958f54ad39f0622a7c97

Comment by Githook User [ 20/Sep/19 ]

Author:

{'name': 'Benety Goh', 'username': 'benety', 'email': 'benety@mongodb.com'}

Message: SERVER-39239 add IndexBuildsCoordinator::_getIndexBuild()
Branch: master
https://github.com/mongodb/mongo/commit/328d1877ce3ab19538d8f19eee8f8867d90742e8

Generated at Thu Feb 08 04:51:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.