Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-12266

Update no longer allows empty modifier objects

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 2.5.4
    • Fix Version/s: None
    • Component/s: Write Ops
    • Labels:
      None
    • Operating System:
      ALL

      Description

      tl;dr:

      There is no consistent way to upsert an identifier-only document between 2.4.x and 2.5.x. Allowing modifiers in a newObj to be an empty BSON object would help.


      I noticed the following while testing Doctrine ODM against 2.5.x. In 2.4.x and earlier versions, I believe there was only a single way to upsert a document that only contained an identifier field:

      > db.foo.update({_id:1}, {$set: {}}, true); db.getLastErrorObj();
      {
      	"updatedExisting" : false,
      	"n" : 1,
      	"connectionId" : 1,
      	"err" : null,
      	"ok" : 1
      }
      > db.foo.update({_id:2}, {}, true); db.getLastErrorObj();
      {
      	"updatedExisting" : false,
      	"upserted" : ObjectId("52cb12553ad84dc22b80c72f"),
      	"n" : 1,
      	"connectionId" : 1,
      	"err" : null,
      	"ok" : 1
      }
      > db.foo.update({_id:3}, {$set: {_id:3}}, true); db.getLastErrorObj();
      {
      	"err" : "Mod on _id not allowed",
      	"code" : 10148,
      	"n" : 0,
      	"connectionId" : 1,
      	"ok" : 1
      }
      > db.foo.find()
      { "_id" : 1 }
      { "_id" : ObjectId("52cb12553ad84dc22b80c72f") }

      Using an empty object for the newObj argument results in the upsert ignoring the client-provided _id. $set cannot be used on _id, even if that would technically be OK for an upsert. $setOnInsert would make more sense, but it also doesn't work – it's also 2.4+ only, so I wouldn't rely on it for essential ODM logic.

      In 2.5.x, the one working method from 2.4.x no longer works. The two methods that didn't work in 2.4.x do work in 2.5.x:

      > db.foo.update({_id:1}, {$set: {}}, true); db.getLastErrorObj();
      {
      	"err" : "'$set' is empty. You must specify a field like so: {$mod: {<field>: ...}}",
      	"code" : 16840,
      	"n" : 0,
      	"connectionId" : 2,
      	"ok" : 1
      }
      > db.foo.update({_id:2}, {}, true); db.getLastErrorObj();
      {
      	"updatedExisting" : false,
      	"upserted" : 2,
      	"n" : 1,
      	"connectionId" : 2,
      	"syncMillis" : 0,
      	"writtenTo" : null,
      	"err" : null,
      	"ok" : 1
      }
      > db.foo.update({_id:3}, {$set: {_id:3}}, true); db.getLastErrorObj();
      {
      	"updatedExisting" : false,
      	"upserted" : 3,
      	"n" : 1,
      	"connectionId" : 2,
      	"syncMillis" : 0,
      	"writtenTo" : null,
      	"err" : null,
      	"ok" : 1
      }
      > db.foo.find()
      { "_id" : 2 }
      { "_id" : 3 }

      The strict validation that makes the 2.4.x solution no longer work looks to have been introduced in this commit for SERVER-7175.

        Issue Links

          Activity

          Hide
          JacobGH111 Jacob Heller added a comment - - edited

          I'm using Pymogo (version 2.7.1). The issue arises when I'm using $set and $unset at the same time. If write (in Python):
          myDB.myTable.update(

          {'_id':12345}

          , {'$set':

          {'setThisField':123}

          , '$unset':{}})

          Then the function returns nothing. There is no indication that setting setThisField failed because the argument to $unset was emtpy. This is what I mean by data being mysteriously lost. Indeed, if I specify "w=1" as an argument to update, then it raises an exception indicating that the save failed, but any code that is designed for older versions of Mongo are potentially going to fall into this pitfall of failing silently.

          As Glenn Maynard suggested, this is a good place to create a warning, but you shouldn't lose data that you're trying to $set because $unset was empty.

          Show
          JacobGH111 Jacob Heller added a comment - - edited I'm using Pymogo (version 2.7.1). The issue arises when I'm using $set and $unset at the same time. If write (in Python): myDB.myTable.update( {'_id':12345} , {'$set': {'setThisField':123} , '$unset':{}}) Then the function returns nothing. There is no indication that setting setThisField failed because the argument to $unset was emtpy. This is what I mean by data being mysteriously lost. Indeed, if I specify "w=1" as an argument to update, then it raises an exception indicating that the save failed, but any code that is designed for older versions of Mongo are potentially going to fall into this pitfall of failing silently. As Glenn Maynard suggested, this is a good place to create a warning, but you shouldn't lose data that you're trying to $set because $unset was empty.
          Hide
          behackett Bernie Hackett added a comment -

          Please make sure you are using pymongo.MongoClient, not pymongo.Connection which has been deprecated since PyMongo 2.4 and is being removed in PyMongo 3.0. With MongoClient there is no need to pass w=1 in the update statement:

          >>> import pymongo
          >>> c = pymongo.MongoClient()
          >>> c.myDB.myTable.update({'_id': 12345}, {'$set': {'setThisField': 123}, '$unset': {}})
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
            File "pymongo/collection.py", line 574, in update
              _check_write_command_response(results)
            File "pymongo/helpers.py", line 206, in _check_write_command_response
              raise OperationFailure(error.get("errmsg"), error.get("code"), error)
          pymongo.errors.OperationFailure: '$unset' is empty. You must specify a field like so: {$mod: {<field>: ...}}

          Also, make sure your application isn't globally setting w: 0 as its default write concern.

          Show
          behackett Bernie Hackett added a comment - Please make sure you are using pymongo.MongoClient, not pymongo.Connection which has been deprecated since PyMongo 2.4 and is being removed in PyMongo 3.0. With MongoClient there is no need to pass w=1 in the update statement: >>> import pymongo >>> c = pymongo.MongoClient() >>> c.myDB.myTable.update({'_id': 12345}, {'$set': {'setThisField': 123}, '$unset': {}}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pymongo/collection.py", line 574, in update _check_write_command_response(results) File "pymongo/helpers.py", line 206, in _check_write_command_response raise OperationFailure(error.get("errmsg"), error.get("code"), error) pymongo.errors.OperationFailure: '$unset' is empty. You must specify a field like so: {$mod: {<field>: ...}} Also, make sure your application isn't globally setting w: 0 as its default write concern.
          Hide
          JacobGH111 Jacob Heller added a comment -

          Ah I did not realize that pymongo.Connection had been replaced. I will replace that right away, but I still think $set and $unset should allow empty arguments. Thanks!

          Show
          JacobGH111 Jacob Heller added a comment - Ah I did not realize that pymongo.Connection had been replaced. I will replace that right away, but I still think $set and $unset should allow empty arguments. Thanks!
          Hide
          hrach Hrachya Mnatsakanyan added a comment -

          I think this restriction is not very important and it would be better to revert this change as I have to use version 2.4.12 instead of the latest to avoid this restriction as I would have to do more changes.

          Show
          hrach Hrachya Mnatsakanyan added a comment - I think this restriction is not very important and it would be better to revert this change as I have to use version 2.4.12 instead of the latest to avoid this restriction as I would have to do more changes.
          Hide
          glenn Glenn Maynard added a comment - - edited

          This is a huge, breaking change and I'm disappointed that it's been ignored. This broken our code in dozens of places and I've had to insert a shim to work around it. This change was harmful, breaking important functionality and making Mongo's update language brittle and illogical. We've stuck with 2.4 for a long time because of this problem.

          All of our code uses find_and_modify as a single, consistent way to handle all basic queries, including finds, updates, find + update, and cases where we may need to do any combination depending on flags. To do a find, we just do {$set: {}}. We fill in the update dynamically if there are changes to be made, and if there aren't we still run the query in order to retrieve the current document. All of these are obvious uses of findAndModify that this change broke.

          For other people bit by this, here's a partial workaround for Python. This is against pymongo.Collection; we have no plans to change APIs for software in maintenance mode. It also only patches find_and_modify, since we never use update() directly.

          def _fix_find_and_modify():
              from pymongo import collection
              orig_find_and_modify = collection.Collection.find_and_modify
              def fixed_find_and_modify(*args, **kwargs):
                  value = None
                  if len(args) >= 3:
                      value = args[2]
                  elif 'update' in kwargs:
                      value = kwargs['update']
           
                  if value:
                      for key, val in value.items():
                          if not val:
                              del value[key]
           
                      if not value:
                          # Prevent the update from being empty.
                          value['$pull'] = {
                              '_dummy_pull': True
                          }
           
                  return orig_find_and_modify(*args, **kwargs)
              collection.Collection.find_and_modify = fixed_find_and_modify
          _fix_find_and_modify()
          

          Show
          glenn Glenn Maynard added a comment - - edited This is a huge, breaking change and I'm disappointed that it's been ignored. This broken our code in dozens of places and I've had to insert a shim to work around it. This change was harmful, breaking important functionality and making Mongo's update language brittle and illogical. We've stuck with 2.4 for a long time because of this problem. All of our code uses find_and_modify as a single, consistent way to handle all basic queries, including finds, updates, find + update, and cases where we may need to do any combination depending on flags. To do a find, we just do {$set: {}}. We fill in the update dynamically if there are changes to be made, and if there aren't we still run the query in order to retrieve the current document. All of these are obvious uses of findAndModify that this change broke. For other people bit by this, here's a partial workaround for Python. This is against pymongo.Collection; we have no plans to change APIs for software in maintenance mode. It also only patches find_and_modify, since we never use update() directly. def _fix_find_and_modify(): from pymongo import collection orig_find_and_modify = collection.Collection.find_and_modify def fixed_find_and_modify(*args, **kwargs): value = None if len(args) >= 3: value = args[2] elif 'update' in kwargs: value = kwargs['update']   if value: for key, val in value.items(): if not val: del value[key]   if not value: # Prevent the update from being empty. value['$pull'] = { '_dummy_pull': True }   return orig_find_and_modify(*args, **kwargs) collection.Collection.find_and_modify = fixed_find_and_modify _fix_find_and_modify()

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 day ago
                Date of 1st Reply: