[SERVER-74024] Ensure Shard Merge supports change stream pre- and post-images Created: 14/Feb/23  Updated: 29/Oct/23  Resolved: 25/Apr/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Christopher Caplinger Assignee: Christopher Caplinger
Resolution: Fixed Votes: 0
Labels: change-streams-support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Server Serverless 2023-03-06, Server Serverless 2023-03-20, Server Serverless 2023-04-03, Server Serverless 2023-04-17, Server Serverless 2023-05-01
Participants:

 Description   

Supporting pre-images in the event of a failover after the completion of a Shard Merge is difficult. Consider the following scenario:

  • TenantOplogApplier applies a write on the recipient primary during the oplog catchup phase of a shard merge with donor timeline TS = 100 and a pre-image is inserted at the same ts via writeChangeStreamPreImage in db/repl/oplog.cpp. The write operation itself is logged on the recipient timeline at TS = 150.
  • ChangeStreamPreImageCollectionManager::insertPreImage is called via op observer as a result of the above applied write, but with an optime on the recipient timeline at TS = 150.
  • On the recipient secondary, replication happens for the write at TS = 150, resulting in writeChangeStreamPreImage being called, this time on the recipient timeline at TS = 150.

at the end of this sequence of events, we have 2 pre-image entries for the same write on the primary, identical except for differing timestamps (TS = 100 and TS = 150). on the secondary, we have one pre-image entry at TS = 150. if we attempt to resume a change stream on the recipient primary with a resume token from before the migration, we can successfully resume because the pre-image entry on the donor timeline exists. resumption fails on the secondary because only the recipient timeline pre-image entry exists.

Ideally, we should also try to fix the "duplicate" pre-image entry issue described above so that the entries on primary and secondaries are consistent.



 Comments   
Comment by Githook User [ 24/Apr/23 ]

Author:

{'name': 'Christopher Caplinger', 'email': 'christopher.caplinger@mongodb.com', 'username': 'UnicodeSnowman'}

Message: SERVER-74024: Shard Merge supports change stream pre-images
Branch: master
https://github.com/mongodb/mongo/commit/5993f36ebb8e7adc26a51262558156a5c3a9e8ed

Generated at Thu Feb 08 06:26:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.