[SERVER-58636] Initial syncing node can miss final oplog entry when calculating stopTimestamp against a secondary sync source Created: 16/Jul/21  Updated: 29/Oct/23  Resolved: 05/Oct/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.7, 5.0.0-rc8
Fix Version/s: 4.4.11, 5.1.0-rc0, 5.0.5

Type: Bug Priority: Major - P3
Reporter: Jason Chan Assignee: Vesselina Ratcheva (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0, v4.4, v4.2, v4.0
Sprint: Repl 2021-08-09, Repl 2021-08-23, Repl 2021-09-06, Repl 2021-09-20, Repl 2021-10-04, Repl 2021-10-18
Participants:
Linked BF Score: 115

 Description   

(10/4/21 discoverability update: We figured out this happens due to the relevant secondary read getting a readSource of lastApplied (per most other use cases). Making that an untimestamped read solves the problem.)

The calls to applyCommand_inlock and scheduleOplogWrites in secondary application are not atomic. So it's possible that when an initial syncing node chooses a secondary as a sync source, it sees that a command like drop has been applied, but misses the oplog entry when calculating the stopTimestamp.

The following can happen:

  1. Initial syncing node sees the drop on collection foo has been applied on a secondary sync source (but no oplog write yet). The collectionCloner will stop with NamespaceNotFound error, expecting us to apply the drop during the initial sync oplog application phase.
  2. Initial syncing node fetches the lastApplied of the sync source, setting the stopTimestamp to T.
  3. The sync source writes the oplog for the drop from (1) at timestamp T + 1.
  4. The initial syncing node reaches stopTimestamp T, transitions to secondary, and applies the drop, and crashes because the collection does not exist.


 Comments   
Comment by Githook User [ 19/Nov/21 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-58636 Allow initial syncing nodes to see entries from uncommitted batches on secondaries

(cherry picked from commit 4a230c070cb187604d07d04598a912b4feca937d)
Branch: v4.4
https://github.com/mongodb/mongo/commit/755aaba320ca2a2a163f198ee7d249e95fdd77e7

Comment by Githook User [ 17/Nov/21 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-58636 Allow initial syncing nodes to see entries from uncommitted batches on secondaries

(cherry picked from commit 4a230c070cb187604d07d04598a912b4feca937d)
Branch: v5.0
https://github.com/mongodb/mongo/commit/2c046c14b8042d025b44142b6d32138a11aaf223

Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 04/Oct/21 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-58636 Allow initial syncing nodes to see entries from uncommitted batches on secondaries
Branch: master
https://github.com/mongodb/mongo/commit/4a230c070cb187604d07d04598a912b4feca937d

Generated at Thu Feb 08 05:45:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.