[SERVER-9074] Upsert fails with "cannot modify shard key" when not modifying shard key Created: 21/Mar/13  Updated: 11/Jul/16  Resolved: 14/Nov/13

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.0
Fix Version/s: 2.5.4

Type: Bug Priority: Major - P3
Reporter: Chad Wallace Assignee: Scott Hernandez (Inactive)
Resolution: Done Votes: 6
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian Wheezy amd64


Attachments: File mongo-commands-2.2.3-successful     File mongo-commands-2.4.0-failing    
Issue Links:
Depends
Duplicate
is duplicated by SERVER-8384 Using $set with upsert against sharde... Closed
Gantt Dependency
Related
related to SERVER-7379 Immutable shardkey becomes mutable Closed
related to SERVER-6835 Sharding edge case upserts documents ... Closed
Operating System: ALL
Participants:

 Description   

After upgrading to 2.4.0, updates stopped working. All of my updates are upserts, and they fail with the message "cannot modify shard key for collection tripquest.persistentlistings, found new value for q.md", both through the Perl MongoDB module and in the mongo shell.

I've checked and double-checked, and the shard key is not being modified. It even happens when there is NO existing document. A straight insert does work.

It also works if I run a 2.2.3 mongos and connect to the same configdb, with the same mongod's (all 2.4) behind it. With a 2.4.0 mongos, it doesn't work.

Also, it complains that it found a new value for the first entry in "q". If I put "de" first, it complains that "de" is the new value. Likewise for "gw".

The shard key is a composite:

{ "q.de" : 1, "q.gw" : 1, "q.md" : 1 }

 Comments   
Comment by Githook User [ 12/Dec/13 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@10gen.com'}

Message: SERVER-9074 Fix incorrect test cases in shard key immutability test
Branch: master
https://github.com/mongodb/mongo/commit/6a13bdab4b65807884fe975d669ebf234b4fd6f2

Comment by auto [ 26/Oct/13 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@10gen.com'}

Message: SERVER-9074 Use the query as the shard key value source in upserts with $ mods
Branch: master
https://github.com/mongodb/mongo/commit/4729ca809b99da06ef7b57f9962740089088242d

Comment by Randolph Tan [ 15/Jul/13 ]

Hi,

The okForStorage is a sanity check performed in the server for the validity of the BSON document and it was rejecting the $date field that you used in your object. To get around this, you can use the Date object to replace the $date like this:

{ "expiryDate" : { "$date" : "2013-06-27T11:32:02.238Z"}}

becomes:

{ "expiryDate" : new Date("2013-06-27T11:32:02.238Z") }

You can also get more information about extended json here:
http://docs.mongodb.org/manual/reference/mongodb-extended-json/

Comment by Juan Cervera [ 27/Jun/13 ]

Just for a test, I removed the subdocument and still produced me problems, see below:

db.websession.update({ "key" : "99901_17" , "subKey" : "gl01331m"}, { "$set" : { "data" : "myData" , "expiryDate" : { "$date" : "2013-06-27T11:32:02.238Z"}}}, {upsert: true} )
not okForStorage

What's this "not okForStorage" about?

If I remove the $set bit and put the key back in the update area it seems to work though ... see below:

db.websession.update({ "key" : "99901_17" , "subKey" : "gl01331m"}, { "key" : "99901_17", "subKey" : "gl01331m", "data" : "myData2" , "expiryDate" : "someDate"}, {upsert: true} )

Note though that removing the subdocument is really not a palatable option as we would have to migrate a lot of production data to the new structure, something I would rather avoid.

Comment by Juan Cervera [ 27/Jun/13 ]

Doug, I've tried the approach that you suggest but still gives problems if the shard key is part of a subdocument and other bits of that subdocument are being udpated, see example below:

shard key: { "content.key" : "hashed" }
 
db.websession.update(
{
    "content.key": "99901_17",
    "content.subKey": "gl01331m"
},
{"$set": {
    "content": {
        "data": "myData",
        "expiryDate": {
            "$date": "2013-06-27T11:32:02.238Z"
        }
    }
}
},
{upsert: true}
)
 
Can't modify shard key's value. field: content: { data: "myData", expiryDate: { $date: "2013-06-27T11:32:02.238Z" } } collection: data.websession
 

Any suggestions for this scenario that wouldn't require extracting the key out of "content"?

Thanks,
Juan

Comment by Doug Woos [ 09/May/13 ]

Here is an update where the shard key is in the query but not in the update:

>>> db.test.test_coll.update({'_id': 'a', 'nested.key': 'b'}, {'_id': 'a', 'b': 'c'}, safe=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gc/env/lib/python2.7/site-packages/pymongo/collection.py", line 479, in update
    check_keys, self.__uuid_subtype), safe)
  File "/gc/env/lib/python2.7/site-packages/pymongo/mongo_client.py", line 916, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/gc/env/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
pymongo.errors.OperationFailure: full shard key must be in update object for collection: test.test_coll

This makes sense, since an update does a replace and the shard key is not in the replacement dictionary. How would you modify this query so that it does not error out?

Comment by Scott Hernandez (Inactive) [ 09/May/13 ]

Save is just a (special) update with the query of the _id. The full shard key will always need to be there for save to work, and will always be included since you doing a full document replacement in the update part.

Any save can be manually converted to a regular update w/upsert-flag. You can include the shard key in the query but not the update and it will work just fine. Save is a special case which doesn't work well in a sharded cluster unless you shard by the _id field.

To sum it up, save works just fine on sharded collections since it must include the full shard key.

Comment by Doug Woos [ 09/May/13 ]

Scott,

When doing a save (as opposed to an atomic update) you definitely need to include the shard key in the update for it to work at all--the error text if you don't even says "full shard key must be in update object for collection". And the shard key has to be in the query in order for it to be targeted, right? So as it stands it is impossible to reliably do full saves (as opposed to atomic updates) on sharded collections.

Comment by Scott Hernandez (Inactive) [ 09/May/13 ]

The query is used as the base for the insert, if the document isn't found to update, so as long as you include the shard key in the query it will work out; you don't need to include it in both, and in some cases doing so will cause problems when depending on the form used in the update part.

Comment by Kiril Savino [ 09/May/13 ]

Thanks Scott-

So given an update like:

db.collection.update(

{shard_key: 1, _id: 1}

, {_id: 1, shard_key: 1})

Will that populate shard_key on an insert? (nearly all of our writes are upserts, because we generate keys remotely)

And for compound keys, I assume you'd need all of the values that are part of the key to be removed?

Kiril Savino
Founder, CTO
GameChanger Media
@kirilnyc

Comment by Scott Hernandez (Inactive) [ 09/May/13 ]

Kiril, if you remove the shard key from the update part, and just leave it in the query part then it should work. If you have a new case which is not working, please let us know.

Comment by Kiril Savino [ 09/May/13 ]

Just to chime in, this is a serious breaking change. We just had to roll back our staging environment to 2.2, and will now be delaying our 2.4 rollout to product indefinitely, which really sucks.

Comment by Pierre-Alban DEWITTE [ 06/May/13 ]

Thanks for your quick reply.

Comment by Scott Hernandez (Inactive) [ 25/Apr/13 ]

We have been tightening up the restricts around a few corner case bugs where it was possible to accidentally change the shard key during the insert (if the document wasn't found and upsert was specified). Please remove the shard key from the update statement for now until we can diagnose and fix this on the server.

// old update:
db.persistentlistings.update(
   {'q.de':'44','q.gw':'YVR','q.md':'0505','q.da':'20130505','q.du':7},
   {q:{de:'44',gw:'YVR',md:'0505',da:'20130505',du:7},query_timestamp:1363906380,query_status:1},
   {upsert:1})
 
// new update:
db.persistentlistings.update(
   {'q.de':'44','q.gw':'YVR','q.md':'0505','q.da':'20130505','q.du':7},
   {$set: { query_timestamp: 1363906380, query_status:1}}, //the query criteria will be included if it results in an insert
   {upsert:1})

Comment by Pierre-Alban DEWITTE [ 25/Apr/13 ]

Hi,

We face the same issue with a fresh migration from 2.2 to 2.4.

Best regards

Generated at Thu Feb 08 03:19:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.