[SERVER-81862] Streams: for $merge to sharded collections not sharded on _id, need to include shard key in upsert requests Created: 04/Oct/23  Updated: 04/Jan/24  Resolved: 06/Dec/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: Matthew Normyle Assignee: Sandeep Dhoot
Resolution: Fixed Votes: 0
Labels: init-337-m3, prioritize
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Assigned Teams:
Atlas Streams
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sprint 33, Sprint 34, Sprint 35, Sprint 36, Sprint 37
Participants:
Linked BF Score: 30

 Description   

Create a shard collection with key that isn't _id. $merge writes will send things to the DLQ with:
{
_id: ObjectId("651d8464f0b07ba2ac0ce8a3"),
_stream_meta:

{ timestamp: ISODate("2023-10-04T15:27:32.278Z") }

,
errInfo: {
reason: 'Failed to process an input document in the current batch in MergeOperator with error: Failed to target upsert by query :: could not extract exact shard key: generic server error'
}
}



 Comments   
Comment by Githook User [ 04/Jan/24 ]

Author:

{'name': 'Sandeep Dhoot', 'email': 'sandeep.dhoot@mongodb.com', 'username': 'sandeepgd'}

Message: SERVER-81862: Remove streams_auth test suite.

GitOrigin-RevId: ff6e453f7b78deb1b38c8ed17ea83ee44e1429c5
Branch: master
https://github.com/mongodb/mongo/commit/937c16dac516fe5c0f08daa823096e55501ee169

Comment by Sandeep Dhoot [ 06/Dec/23 ]

yoonsoo.kim@mongodb.com Yeah, this is done. Closing it now. Thanks!

Comment by Yoon Soo Kim [ 06/Dec/23 ]

sandeep.dhoot@mongodb.com, is this supposed to be closed? I have a BF depending on this ticket and wanted to make sure that this is actually closed.

Comment by Githook User [ 29/Nov/23 ]

Author:

{'name': 'Sandeep Dhoot', 'email': 'sandeep.dhoot@mongodb.com', 'username': 'sandeepgd'}

Message: SERVER-81862: [streams] Make content-based routing work for sharded collections.
Branch: master
https://github.com/mongodb/mongo/commit/c465e20138b275940a2e8c1c09fd010fe697b819

Comment by Sandeep Dhoot [ 07/Oct/23 ]

Still need to figure out a proper fix for this. But possibly the error goes away if we specify the shard key in $merge.on. Confirming this. It also seems like server team recently worked on making this work even when the shard key is not specified in the incoming write. But it seems more efficient to always specify shard key for sharded collections.

Generated at Thu Feb 08 06:47:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.