[SERVER-14024] Update fails when query contains part of a DBRef and results in an insert (upsert:true) Created: 21/May/14  Updated: 09/Nov/16  Resolved: 17/Jul/14

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: 2.6.0
Fix Version/s: 2.6.4, 2.7.4

Type: Bug Priority: Major - P3
Reporter: John Mullanaphy Assignee: Scott Hernandez (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Run the second query, seems and "*.$id" upsert fails.

Participants:

 Description   
Issue Status as of Jul 22, 2014

ISSUE SUMMARY

When the update query contains a partial DBRef ($id / $ref / $db) with upsert:true, and a document is not found which results in an insert then the update will fail.

Example:

db.coll1.drop()
db.coll1.update({a.$id:1}, {$set: {a: DBRef("coll2", 1)}, $inc: {c: 1}}}, {upsert:true})
...
  "code" : 55,
  "errmsg" : "Found $id field without a $ref before it, which is invalid."
...

This failure is due to the query being validated for storage, which will fail due to the incomplete DBRef even though the final updated document will contain the full DBRef.

USER IMPACT
The upsert fails but no other operations are affected.

WORKAROUNDS
You can include the entire DBRef in the query part instead of incomplete DBRef fields.

AFFECTED VERSIONS
Production release versions 2.6.0 to 2.6.3 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
This code change moves the validation of the DBRef until after the full update has been performed.

Original description

When the update query contains a partial DBRef ($id/$ref/$db) with upsert:true, and a document is not found which results in an insert then the update will fail.

> db.coll1.drop()
> db.coll1.update({a.$id:1}, {$set: {a: DBRef("coll2", 1)}, $inc: {c: 1}}}, {upsert:true})
...
  "code" : 55,
  "errmsg" : "Found $id field without a $ref before it, which is invalid."
...

This failure is due to the query being validated for storage, which will fail due to the incomplete DBRef even though the final updated document will contain the full DBRef. This code changes simply moves the validation of the DBRef until after the full update has been performed.

Old Description
We have an index on a DBRef field for it's $id value "config.$id" instead of on "config" for performance reasons. Find works fine, however doing an update since 2.6 and searching for "config.$id" errors out with:

{
  "code" : 55,
  "errmsg" : "Found $id field without a $ref before it, which is invalid."
}

The error was happening in the PHP driver, yet it is testable via the command line as well.

So this query works:

db.alerts.update(
  {
    "config": {
      "$ref": "alert",
      "$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx1"),
      "$db": "Configs"
    },
    "date":ISODate("2014-05-21T00:00:00Z"),
    "server": "someserver.com"
  },
  {
    "$set": {
      "config": {
        "$ref": "alert",
        "$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx1"),
        "$db": "Config"
      },
      "date": ISODate("2014-05-21T04:00:00Z"),
      "server": "someserver.com",
      "user": {
        "$ref": "User",
        "$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx2"),
        "$db": "Users"
      },
      "triggered": ISODate("2014-05-21T16:57:57Z")
    },
    "$inc": {
      "count": 1
    }
  },
  {
    "upsert": true
  }
);

However this one that uses the index doesn't:

db.alerts.update(
  {
    "config.$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx1"),
    "date":ISODate("2014-05-21T00:00:00Z"),
    "server": "someserver.com"
  },
  {
    "$set": {
      "config": {
        "$ref": "alert",
        "$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx1"),
        "$db": "Config"
      },
      "date": ISODate("2014-05-21T04:00:00Z"),
      "server": "someserver.com",
      "user": {
        "$ref": "User",
        "$id": ObjectId("5xxxxxxxxxxxxxxxxxxxxxx2"),
        "$db": "Users"
      },
      "triggered": ISODate("2014-05-21T16:57:57Z")
    },
    "$inc": {
      "count": 1
    }
  },
  {
    "upsert": true
  }
);

A work around for now is I'll add a find, and then an update/insert if I find that document. Which would probably cleaner either way.



 Comments   
Comment by Chris Levy [ 09/Nov/16 ]

My apologies there is no confusion on your part but only mine, thanks for the help.

Comment by Chris Levy [ 09/Nov/16 ]

Hi Dave, thanks for your reply, I think you are confusing the "match" part of the update with the "update" part, however you are correct in that this is working fine, I think I made a mistake in picking my fields, I now have this working ok.

Comment by David Storch [ 08/Nov/16 ]

Hi clevy,

I believe that the behavior you pointed out is working as designed. If the update were to succeed and perform an insert, it would result in a document like the following:

{ "_id" : ObjectId("582212066e4b79985973a834"), "manufacturerref" : { "$id" : ObjectId("52f9f59ae34e246e2c4f961f") }, "partno" : "115000F00000G", "pricecount" : 2 }

This is an illegal DBRef, since the $id field is not preceded by $ref. This is different from the case originally reported here; in that case, the upsert resulted in a perfectly valid DBRef, but was nevertheless rejected by the update subsystem's validation code.

I hope this explanation makes sense. If you believe that my "works as designed" assessment is not correct, please open a new SERVER ticket reporting the bug and we will investigate and triage by our regular process.

Best,
Dave

Comment by Chris Levy [ 01/Nov/16 ]

This issue appears to be back in version 2.6.12 :-

db.part.update(
{"partno":"115000F00000G", "manufacturerref.$id":ObjectId("52f9f59ae34e246e2c4f961f")},
{$set:{"partno":"115000F00000G", "pricecount":2}},
{"upsert":1}
)

results in

WriteResult({
	"nMatched" : 0,
	"nUpserted" : 0,
	"nModified" : 0,
	"writeError" : {
		"code" : 55,
		"errmsg" : "Found $id field without a $ref before it, which is invalid."
	}
})

Comment by Githook User [ 17/Jul/14 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-14024: validate update insert fields at the end
(cherry picked from commit a0bad0a3f1b6562cd39fe66774db6ee486064045)

Conflicts:
jstests/core/update_setOnInsert.js
Branch: v2.6
https://github.com/mongodb/mongo/commit/7e51683298ef8f9f58bb1ef879b6df1182e279a4

Comment by Githook User [ 17/Jul/14 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-14024: validate update insert fields at the end
Branch: master
https://github.com/mongodb/mongo/commit/a0bad0a3f1b6562cd39fe66774db6ee486064045

Comment by Scott Hernandez (Inactive) [ 27/Jun/14 ]

Kiril, it is marked for back-port consideration on the next 2.6.x release, once we get it committed to master of course.

Ben, yes, it covers that case too.

Comment by Kiril Savino [ 27/Jun/14 ]

Any way this will get back-ported to 2.6? It's going to be blocking upgrade for us.

Comment by Ben Yelsey [ 27/Jun/14 ]

Hi,
We're seeing this issue as well (operation failure doing an upsert->insert when the <query> includes an $id).
A example of a query that replicates this, but should work:

db.foo.update({'bar.$id': ObjectId()}, {$set: {$ref: 'bar', 'key': 'value'}}, {upsert: true})

We also get a similar error if we replace $id with $ref.

Any idea when we can expect a fix?

Comment by John Mullanaphy [ 23/May/14 ]

Excellent and good to hear. I'll keep watching it although as of right now it isn't affecting me, since I'm doing a find then update/insert instead of the upsert. Anyone using Doctrine's ODM can possibly be affected by this as they rely heavily on the "field.$id" for references and not even realize it.

So, thanks again and I'll keep an eye on this.

Comment by Thomas Rueckstiess [ 23/May/14 ]

Hi John,

Thanks for reporting the issue. We can reproduce this behavior and are investigating. You can watch this ticket for progress on the issue.

Regards,
Thomas

Comment by John Mullanaphy [ 22/May/14 ]

Seems that on the update if the document doesn't exist it fails because of the find the config.$id makes the update sad and it's not due to the value itself, whether that part of the DBRef is an ObjectId or a string _id.

Generated at Thu Feb 08 03:33:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.