[SERVER-22287] Merging replica sets with replication protocol version 1 may result in two primaries Created: 25/Jan/16  Updated: 25/Jan/17  Resolved: 10/Feb/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.1
Fix Version/s: 3.2.4, 3.3.2

Type: Bug Priority: Major - P3
Reporter: Kevin Adistambha Assignee: Benety Goh
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File two_primaries.js    
Issue Links:
Duplicate
is duplicated by SERVER-7366 Create a random seed on rs.initiate()... Closed
Related
is related to DOCS-1089 Replica Set deployment instructions n... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

1. Create a replica set with at least two members, using rs.initiate().
2. Create another replica set with a single member with the same name as (1), using rs.initiate().
3. rs.add() the PRIMARY of (2) to the replica set (1).
4. rs.status() on (1) shows two PRIMARY.

Note: two_primaries.js jstest is attached.

Sprint: Repl F (01/29/16), Repl 10 (02/19/16)
Participants:

 Description   

In replication protocol version 1, it is possible to create a replica set with two primaries by creating two replica sets with the same name and adding the primary of one set to the other set.

Any write to the set at this state was done to one of the primaries. However, attempt to restart the set may show data corruption error. Error message is:

[rsBackgroundSync] Fatal assertion 28723 UnrecoverableRollbackError need to rollback, but unable to determine common point between local and remote oplog: NoMatchingDocument RS101 reached beginning of local oplog [2] @ 18752

This issue does not happen with protocol version 0.



 Comments   
Comment by Githook User [ 18/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 added replica set ID validation to heartbeat response processing, replSetReconfigure and replica set quorum check.

(cherry picked from commit 0bc96b1ede73c82d45b6d6c2e0b9a33e613ca7c3)
(cherry picked from commit 8633f9c7290e025182c1ea6a42d1c5c44224506f)
(cherry picked from commit 3e96008ad946a5423806f823e8ee3744c49537be)
(cherry picked from commit 38cd69968a54a3a72b9c83f0bcdbc5acac03785a)
Branch: v3.2
https://github.com/mongodb/mongo/commit/8033364795d3b4453ddbbfedb5edd8a75f18c85b

Comment by Githook User [ 18/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 added replicaSetId field to replica set configuration and metadata.

This field is generated when a replica set configuration is created from the replSetInitiate command

(cherry picked from commit 1ecf7de7e18df42c154c04b1d9310956c91a75b7)
Branch: v3.2
https://github.com/mongodb/mongo/commit/afa3bd494b3a211da1d076641fc72e25dab8f829

Comment by Githook User [ 18/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 build ReplSetTest._alldbpaths using actual dbpath from MongoRunner.runMongod result

(cherry picked from commit c58239c96ee83dab5232084cb69238de2b093c49)
Branch: v3.2
https://github.com/mongodb/mongo/commit/4cfb0c2e72d29fa4514613dc5242da9ab2169c2d

Comment by Githook User [ 08/Feb/16 ]

Author:

{u'username': u'dannenberg', u'name': u'Matt Dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: Make clear that rs.initiate() should not be run against multiple nodes

Users have always been confused about rs.initiate() and have been known to run it against each node of a replica set. When they then go to rs.add() the other members to one node, sad things happen. These sad things are much sadder in 3.2, so we should do everything we can to help them avoid this. See https://jira.mongodb.org/browse/SERVER-22287 for more about the 3.2 sad things.

Signed-off-by: kay <kay.kim@10gen.com>
Branch: master
https://github.com/mongodb/docs/commit/9114da830ea7c56ef98a1ffeec71c2ec8e4a976d

Comment by Githook User [ 08/Feb/16 ]

Author:

{u'username': u'dannenberg', u'name': u'Matt Dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: Make clear that rs.initiate() should not be run against multiple nodes

Users have always been confused about rs.initiate() and have been known to run it against each node of a replica set. When they then go to rs.add() the other members to one node, sad things happen. These sad things are much sadder in 3.2, so we should do everything we can to help them avoid this. See https://jira.mongodb.org/browse/SERVER-22287 for more about the 3.2 sad things.

Signed-off-by: kay <kay.kim@10gen.com>
Branch: v3.0
https://github.com/mongodb/docs/commit/42188bd6d18baea1872f9b7fde6680f983e54080

Comment by Githook User [ 08/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 check heartbeat message instead of logs for inconsistent replica set id error
Branch: master
https://github.com/mongodb/mongo/commit/38cd69968a54a3a72b9c83f0bcdbc5acac03785a

Comment by Githook User [ 05/Feb/16 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-22287: disable test for ephemaral variants
Branch: master
https://github.com/mongodb/mongo/commit/3e96008ad946a5423806f823e8ee3744c49537be

Comment by Githook User [ 05/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 comment out failing test ReplCoordHBV1Test::IgnoreTheContentsOfMetadataWhenItsReplicaSetIdDoesNotMatchOurs
Branch: master
https://github.com/mongodb/mongo/commit/8633f9c7290e025182c1ea6a42d1c5c44224506f

Comment by Githook User [ 05/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 added replica set ID validation to heartbeat response processing, replSetReconfigure and replica set quorum check.
Branch: master
https://github.com/mongodb/mongo/commit/0bc96b1ede73c82d45b6d6c2e0b9a33e613ca7c3

Comment by Githook User [ 05/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 build ReplSetTest._alldbpaths using actual dbpath from MongoRunner.runMongod result
Branch: master
https://github.com/mongodb/mongo/commit/c58239c96ee83dab5232084cb69238de2b093c49

Comment by Githook User [ 05/Feb/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-22287 added replicaSetId field to replica set configuration and metadata.

This field is generated when a replica set configuration is created from the replSetInitiate command
Branch: master
https://github.com/mongodb/mongo/commit/1ecf7de7e18df42c154c04b1d9310956c91a75b7

Comment by Eric Milkie [ 25/Jan/16 ]

I suspect that depending on config version (and term, in pv1), you can make both pv0 and pv1 replica sets merge in this way. SERVER-7366 has been an issue for a while now, and it is something we hope to solve for 3.3.

Generated at Thu Feb 08 03:59:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.