[SERVER-25145] During rollback (or w/minvalid invalid) select sync source based on end OpTime Created: 19/Jul/16  Updated: 20/Sep/17  Resolved: 03/Nov/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.0, 3.3.0
Fix Version/s: 3.2.11, 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Scott Hernandez (Inactive) Assignee: Spencer T Brody
Resolution: Fixed Votes: 0
Labels: code-and-test

Issue Links:
Depends
Duplicate
is duplicated by SERVER-27277 [rsBackgroundSync] Fatal assertion 18... Closed
is duplicated by SERVER-28056 rollback fail Closed
is duplicated by SERVER-25026 Secondary abort immediately following... Closed
is duplicated by SERVER-27547 Adjust the system time . then MongoDB... Closed
is duplicated by SERVER-27676 Mongod is not able to restart after O... Closed
is duplicated by SERVER-25861 Adjust minValid logic Closed
Related
related to SERVER-26415 Make sure to catch up to minValid bef... Closed
related to SERVER-26922 minvalid2 should wait for arbiter to ... Closed
related to SERVER-26972 BackgroundSync should compare minVali... Closed
is related to SERVER-21537 chainingAllowed = false not being enf... Open
is related to SERVER-21988 Rollback does not wait for applier to... Closed
is related to SERVER-23070 rewrite SyncSourceResolver to select ... Closed
is related to SERVER-27050 Ensure upstream node doesn't roll bac... Closed
is related to SERVER-25848 Enforce single sync source during rol... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Epic Link: Improved support for Rollback for 3.4
Sprint: Repl 2016-11-21
Participants:
Case:
CRM plugin field not viewable

 Description   

If we use the lastFetchedOptime, which is also the common point found during rollback, it is possible that a sync source is selected which will cause the rollback to be attempted again (after the first says it succeeds), but while in an inconsistent state (with minvalid showing an invalid state, needing to apply before being consistent).

By using the end of minvalid bounds as a criteria for sync source selection it will not be possible to do another rollback until the first has completed and apply all necessary options to get into a consistent state.



 Comments   
Comment by Eric Milkie [ 24/Aug/16 ]

The proposal for fixing rollback will prohibit changing sync sources at the point where the oplog has been truncated to the common point, so I suspect the work described here will no longer make sense.
We should never change sync sources when minValid is greater than the last applied op.

Comment by Eric Milkie [ 19/Oct/16 ]

However, in lieu of the rollback project, this could be an intermediate fix that could avoid the double-rollback trap.

Comment by Githook User [ 02/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 OplogReader::connectToSyncSource selects sync sources based on min valid if given
Branch: master
https://github.com/mongodb/mongo/commit/cbb4f8ef9028461b682bd4aea5097a95f8cfe631

Comment by Githook User [ 02/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: Revert "SERVER-25145 OplogReader::connectToSyncSource selects sync sources based on min valid if given"

This reverts commit cbb4f8ef9028461b682bd4aea5097a95f8cfe631.
Branch: master
https://github.com/mongodb/mongo/commit/16b2afc48459fbd2670c5e824bd662d6ac7b584f

Comment by Githook User [ 02/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 SyncSourceResolver selects sync sources based on required optime if given
Branch: master
https://github.com/mongodb/mongo/commit/053a2dbc9c503bf55ba52bc3c937bc138cc67b8a

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 BackgroundSync provides minValid to sync source selection in RECOVERING mode
Branch: master
https://github.com/mongodb/mongo/commit/fb1cc32e76910334327c0835e3777ee1ddedf2d8

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 OplogReader::connectToSyncSource selects sync sources based on required optime if given
Branch: master
https://github.com/mongodb/mongo/commit/d16d537fe262cc5f2e18b2fbe413ba80515e8c36

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 OplogReader::connectToSyncSource selects sync sources based on required optime if given

(cherry picked from commit d16d537fe262cc5f2e18b2fbe413ba80515e8c36)
Branch: v3.2
https://github.com/mongodb/mongo/commit/aeca1c4b7eaeacff19a32facee8169662cbf3bd6

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25145 BackgroundSync provides minValid to sync source selection in RECOVERING mode

(cherry picked from commit fb1cc32e76910334327c0835e3777ee1ddedf2d8)
Branch: v3.2
https://github.com/mongodb/mongo/commit/560dc779aff60c78fd28194a1412a944cffe32f5

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-25145 Add jstest to ensure that we can no longer trigger double-rollback.
Branch: master
https://github.com/mongodb/mongo/commit/f5c613f11f170cadd4b698d108b3c7b636913568

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-25145 Add jstest to ensure that we can no longer trigger double-rollback.

(cherry picked from commit 3716f0ab00f8f2f4e5e85a87dc06c125628fe03e)
Branch: v3.2
https://github.com/mongodb/mongo/commit/f3a4d0914fca1ad2fc9a659f4ece42f9dcf0b2c2

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-25145 fix lint
Branch: master
https://github.com/mongodb/mongo/commit/b85157b4a7ccc68fb05efa092f5b057ac117978f

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-25145 fix lint

(cherry picked from commit 28b17c04835e4000be52b44e3b96ceaeac95567a)
Branch: v3.2
https://github.com/mongodb/mongo/commit/e0e6faf67235d5eb9a758d7b0dd140bd939682c3

Generated at Tue Nov 21 21:59:47 UTC 2017 using JIRA 7.2.10#72012-sha1:2651463a07e52d81c0fcf01da710ca333fcb42bc.