[SERVER-39495] Shard key is omitted from update and remove oplog entries with multi:true Created: 11/Feb/19 Updated: 29/Oct/23 Resolved: 28/Feb/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Sharding |
| Affects Version/s: | 4.1.6 |
| Fix Version/s: | 4.1.9, 4.0.17 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bernard Gorman | Assignee: | Kaloian Manassiev |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Operating System: | ALL | ||||
| Backport Requested: |
v4.0
|
||||
| Sprint: | Sharding 2019-02-25, Sharding 2019-03-11 | ||||
| Participants: | |||||
| Description |
|
For update and delete operations, a change stream respectively relies upon the kObject2FieldName and kObjectFieldName fields of the oplog entry to provide the documentKey for that event; that is, an object containing the values of each field of the shard key, plus the _id field if it is not already present as part of the shard key. On the 4.0 branch, this is working as expected. On current master, it appears that in at least some scenarios, the update and remove oplog entries incorrectly omit the shard key fields and provide only the _id.
This behaviour persists even when the shards are force-refreshed via _flushRoutingTableCacheUpdates, and even in cases where internally examining the ScopedCollectionMetadata proves that the mongoD is aware of the shard key for this collection. |
| Comments |
| Comment by Githook User [ 26/Feb/20 ] |
|
Author: {'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}Message: ShardingState logically contains answers to questions about whether the This is a partial cherry-pick from b049257fbd1d215388cffaf7544f6741dbce5b45, adapted for the 4.0 branch. Also backports the addition of more testing for multi:true/justOne:false updates and ChangeStreams, which was taken from commit 50f6bd4d6a9428a6f1df22db792d7b55d773762c. |
| Comment by Githook User [ 26/Feb/20 ] |
|
Author: {'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}Message: This is a partial cherry-pick from 851dad7902d6bb8c3ed25f99f565a2e2c8c8bc47, adapted for the 4.0 branch. |
| Comment by Githook User [ 29/Jan/20 ] |
|
Author: {'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}Message: Revert " This reverts commit fe4ced8f98d731883e5a4511d434716629e457a8. |
| Comment by Githook User [ 29/Jan/20 ] |
|
Author: {'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}Message: Revert " This reverts commit 1a01c53df8f7c1e016c0ccbc38b77f6b3508bf65. |
| Comment by Githook User [ 26/Jan/20 ] |
|
Author: {'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}Message: ShardingState logically contains answers to questions about whether the This is a partial cherry-pick from b049257fbd1d215388cffaf7544f6741dbce5b45, adapted for the 4.0 branch. Also backports the addition of more testing for multi:true/justOne:false updates and ChangeStreams, which was taken from commit 50f6bd4d6a9428a6f1df22db792d7b55d773762c. |
| Comment by Githook User [ 26/Jan/20 ] |
|
Author: {'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm', 'name': 'Kaloian Manassiev'}Message: This is a partial cherry-pick from 851dad7902d6bb8c3ed25f99f565a2e2c8c8bc47, adapted for the 4.0 branch. |
| Comment by Githook User [ 06/Mar/19 ] |
|
Author: {'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}Message: |
| Comment by Githook User [ 28/Feb/19 ] |
|
Author: {'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}Message: |
| Comment by Githook User [ 27/Feb/19 ] |
|
Author: {'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}Message: ShardingState logically contains answers to questions about whether the |
| Comment by Bernard Gorman [ 12/Feb/19 ] |
|
Many thanks kaloian.manassiev! I was mostly looking at this from the perspective of fixing change streams, so there may be some additional callsites beyond the three identified above where a similar change is appropriate. |
| Comment by Kaloian Manassiev [ 12/Feb/19 ] |
|
Thank you bernard.gorman for the detailed analysis! Indeed the intention of this change was that operations which were not sent with sharded information on them should be treated as unsharded (or as if the customer wrote directly to the shard). However now I see how this can break multi-writes. I will change it back to use getCurrentMetadata instead. |