[SERVER-38682] Make writes to documents performed as part of chunk migrations visible in a change stream Created: 18/Dec/18 Updated: 29/Oct/23 Resolved: 22/May/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.12 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Evan Nixon | Assignee: | Evan Nixon |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | FTS | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Sprint: | Query 2019-03-25, Query 2019-04-08, Query 2019-04-22, Query 2019-05-06, Query 2019-05-20, Query 2019-06-03 | ||||
| Participants: | |||||
| Description |
|
This task is to allow a client connected directly to a shard server to open a change stream that observes migration-related document events, such as the local insertion or deletion of documents on a shard executed as part of the chunk migration and orphan cleanup processes. |
| Comments |
| Comment by Githook User [ 22/May/19 ] |
|
Author: {'name': 'Evan Nixon', 'email': 'evan.nixon@10gen.com', 'username': 'navenoxin'}Message: |
| Comment by Evan Nixon [ 21/May/19 ] |
|
A quick summary of the approach we are now discussing for this ticket:
|
| Comment by Evan Nixon [ 03/Jan/19 ] |
|
Sorry to get back to this slowly and thank you for your clarifying questions and description update. I do believe that that accurately describe our use case. I think this is a workable issue for maintaining eventually consistent secondary structures (I think secondary structures should resolve to consistency on orphaned document cleanup). |
| Comment by Andy Schwerin [ 19/Dec/18 ] |
|
OK. Last clarification, below. If you're onboard, I'll update the description and maybe the summary to be a little more precise. Migration-related document insert events that in the oplog are speculative, and so would be in a simple implementation in changestreams as well. That is, documents in a chunk get inserted into the recipient before we know that the chunk migration will commit, so you'll see inserts sometimes that don't correspond to migrations that complete. In most cases, aborted migrations get cleaned up and so there will be corresponding delete events on the recipient. For maintaining eventually consistent indexes / secondary structures, this should be workable, and I think matches your use case. Does it? |
| Comment by Evan Nixon [ 19/Dec/18 ] |
|
Sure thing, thanks schwerin. kevin.rosendahl I think this accurately describes what we talked about as well:
Ideally we would see the individual document migration events.
Ideally the events would look the same as and expose themselves in the same stream as regular crud operations.
We would only like this to be exposed through the monitored shard. |
| Comment by Andy Schwerin [ 18/Dec/18 ] |
|
evan.nixon, could you clarify this ticket, by answering the following questions?
|