[SERVER-61613] Enhance "Performing an update on the path '_id' would modify the immutable field '_id" error message Created: 19/Nov/21  Updated: 10/Mar/23  Resolved: 16/Dec/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Andre M Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-61612 $merge failed to update the matching ... Closed
Participants:

 Description   

This is a very strange case - two identical collections behave in a completely different way.

Initially I was trying to run $group with a merge and started getting _id errors in the title and I whittled the test case down to two collections and a simple aggregate pipeline. I can't see any difference between these collections to the point that I tried to BSON-dump them and data looks the same.

The collection pg is a result of a $group aggregation saved in a collection to simplify things. Here's its contents:

db.pg.find()
[ { _id: 1, last_id: 84 }, { _id: 2, last_id: 60 } 

The b2 collection is problematic. Here it is:

db.b2.find()
[
  { _id: 1, created: ISODate("2021-11-18T23:16:33.149Z") },
  { _id: 2, created: ISODate("2021-11-18T23:16:33.149Z") }
] 

Here's the working collection b3:

db.b3.find()
[
  { _id: 1, created: ISODate("2021-11-18T22:53:02.113Z") },
  { _id: 2, created: ISODate("2021-11-18T22:53:02.113Z") }
] 

Here's indexes for both (to make sure it's not a collation issue):

db.b2.getIndexes()
[ { v: 2, key: { _id: 1 }, name: '_id_' } ]

db.b3.getIndexes()
[ { v: 2, key: { _id: 1 }, name: '_id_' } ] 

Finally, here's the problem. When I run this command, it succeeds

db.pg.aggregate([{$merge: {into: "b3", whenMatched: "merge", whenNotMatched: "fail"}} ]);
 
db.b3.find()
[
  { _id: 1, created: ISODate("2021-11-18T22:53:02.113Z"), last_id: 84 },
  { _id: 2, created: ISODate("2021-11-18T22:53:02.113Z"), last_id: 60 }
] 

When I run the same command against b2, it generates this error:

db.pg.aggregate([{$merge: {into: "b2", whenMatched: "merge", whenNotMatched: "fail"}} ]);
MongoServerError: $merge failed to update the matching document, did you attempt to modify the _id or the shard key? :: caused by :: Performing an update on the path '_id' would modify the immutable field '_id 

In a real-life case, there's no pg collection and instead it's a $group stage that updates a collection with the grouping output.

This is happening on an Atlas cluster v4.4.10. Stats for both collections are the same, so I'm not adding them to keep the post smaller. Let me know if you need any other information. I cannot see the difference between these collections in any of the outputs I could come up with.



 Comments   
Comment by Andre M [ 16/Dec/21 ]

Eric, it appears that you marked the issue as Done without changing Fix Version or having any other visible changes in the issue. Wouldn't this mess up release notes by listing an issue with no effect on the product? Is it possible that you meant to set resolution to Won't Fix instead, as this is what the outcome really is?

Comment by Eric Sedor [ 16/Dec/21 ]

Thanks cis74633@bell.net, much appreciated!

Comment by Andre M [ 13/Dec/21 ]

Here's the suggestion, just to keep them cross-linked.

 

https://feedback.mongodb.com/forums/924280-database/suggestions/44560227-please-report-mixed-type-numeric-id-fields-in-me

Comment by Andre M [ 10/Dec/21 ]

Thanks for jumping in. Will do some time next week.

Comment by Eric Sedor [ 10/Dec/21 ]

Hi cis74633@bell.net, sorry for the delay and great work figuring out the source of the error in this case.

Can I please ask you to submit this description of what you've been through to feedback.mongodb.com? Through that channel, it will receive the most visibility from our product teams. Improving the log message might be straightforward, but they will be able to consider other aspects of this problem like:

  • The transparency of BSON types to some clients
  • the way we compare numerics of different types
  • the overall behavior of $merge
Comment by Andre M [ 19/Nov/21 ]

I finally figured out what the actual issue is - the merge error is generated because of the mismatching _id for Int32 and Double types.

The original collection, which is represented by b2 in this test case, was imported via mongoimport and has Int32 type for all numeric data, while the b3 collection was created via Mongo Shell to model the collection against which $group was executed, which was modified via Mongo Shell at some point and updated some numeric data as plain values, such as 123, which is interpreted as Double. So when $max was applied in $group against plain numeric data, it yielded Double, which was considered by $merge as if there's a change in type for _id, hence the error.

Here's bsondump output for each collection:

bsondump blogs_master_db_\b2.bson
{"_id":{"$numberInt":"1"},"created":{"$date":{"$numberLong":"1637277393149"}}}
{"_id":{"$numberInt":"2"},"created":{"$date":{"$numberLong":"1637277393149"}}}
 
bsondump blogs_master_db_\b3.bson
{"_id":{"$numberDouble":"1.0"},"created":{"$date":{"$numberLong":"1637275982113"}}}
{"_id":{"$numberDouble":"2.0"},"created":{"$date":{"$numberLong":"1637275982113"}}}

I wish $merge would figure out how to exclude the field used in the on condition, which works for querying fields of similar numeric types, as most of Mongo comparisons do, but if you decide to just conclude that it's a type mismatch and not a bug, I hope you can change the error message from the current ambiguous form to include what field is being modified and the source and destination values and types, so it's clear what to look into.

Generated at Thu Feb 08 05:52:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.