[SERVER-38682] Make writes to documents performed as part of chunk migrations visible in a change stream Created: 18/Dec/18  Updated: 29/Oct/23  Resolved: 22/May/19

Status: Closed
Project: Core Server
Component/s: Querying, Sharding
Affects Version/s: None
Fix Version/s: 4.1.12

Type: Task Priority: Major - P3
Reporter: Evan Nixon Assignee: Evan Nixon
Resolution: Fixed Votes: 0
Labels: FTS
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Sprint: Query 2019-03-25, Query 2019-04-08, Query 2019-04-22, Query 2019-05-06, Query 2019-05-20, Query 2019-06-03
Participants:

 Description   

This task is to allow a client connected directly to a shard server to open a change stream that observes migration-related document events, such as the local insertion or deletion of documents on a shard executed as part of the chunk migration and orphan cleanup processes.



 Comments   
Comment by Githook User [ 22/May/19 ]

Author:

{'name': 'Evan Nixon', 'email': 'evan.nixon@10gen.com', 'username': 'navenoxin'}

Message: SERVER-38682 Add option to show writes to documents performed as part of chunk migrations in change streams
Branch: master
https://github.com/mongodb/mongo/commit/8cf664042e2cb74c12beaf37ef1a3fbe12d90b73

Comment by Evan Nixon [ 21/May/19 ]

A quick summary of the approach we are now discussing for this ticket:

  • Add undocumented option `showMigrationEvents`, false by default. When true, show migration events. This option is not supported for use by applications.
  • Consider it an error when option `showMigrationEvents` is set for change streams opened from mongos.
Comment by Evan Nixon [ 03/Jan/19 ]

Sorry to get back to this slowly and thank you for your clarifying questions and description update.

I do believe that that accurately describe our use case. I think this is a workable issue for maintaining eventually consistent secondary structures (I think secondary structures should resolve to consistency on orphaned document cleanup).

Comment by Andy Schwerin [ 19/Dec/18 ]

OK. Last clarification, below. If you're onboard, I'll update the description and maybe the summary to be a little more precise.

Migration-related document insert events that in the oplog are speculative, and so would be in a simple implementation in changestreams as well. That is, documents in a chunk get inserted into the recipient before we know that the chunk migration will commit, so you'll see inserts sometimes that don't correspond to migrations that complete. In most cases, aborted migrations get cleaned up and so there will be corresponding delete events on the recipient. For maintaining eventually consistent indexes / secondary structures, this should be workable, and I think matches your use case. Does it?

Comment by Evan Nixon [ 19/Dec/18 ]

Sure thing, thanks schwerin. kevin.rosendahl I think this accurately describes what we talked about as well:

What chunk migration event do you wish to represent in the change stream? For example, do you want to see the chunk commit event or the individual document migration events?

Ideally we would see the individual document migration events.

How do you wish to observe these changes? Should it be possible to expose them in the same stream that shows regular crud operations?

Ideally the events would look the same as and expose themselves in the same stream as regular crud operations.

Do you want this feature available for general use, in which case it would be exposed through a router, or for internal use by processes monitoring specific shards, in which case it would be exposed only through the monitored shard?

We would only like this to be exposed through the monitored shard.

Comment by Andy Schwerin [ 18/Dec/18 ]

evan.nixon, could you clarify this ticket, by answering the following questions?

  • What chunk migration event do you wish to represent in the change stream? For example, do you want to see the chunk commit event or the individual document migration events?
  • How do you wish to observe these changes? Should it be possible to expose them in the same stream that shows regular crud operations?
  • Do you want this feature available for general use, in which case it would be exposed through a router, or for internal use by processes monitoring specific shards, in which case it would be exposed only through the monitored shard?
Generated at Thu Feb 08 04:49:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.