[SERVER-5691] Queries with nested collections fail after MongoDB reorders fields post-update Created: 24/Apr/12  Updated: 15/Aug/12  Resolved: 24/Apr/12

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.0.4
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Arun Bhalla Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux:
Linux 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:58:24 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

OSX (10.6.8):
Darwin 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 i386 MacPro1,1 Darwin

MongoDB 2.0.4
Mongo Java Driver 2.7.3
Casbah 2.1.5-1
Salat 0.0.8-SNAPSHOT


Operating System: ALL
Participants:

 Description   

We are developing a straightforward Scala web application with a MongoDB backend using Salat+Casbah. We've noticed that sometimes our updates via Salat+Casbah fail to succeed. I can now explain why it's happening:

  1. We create documents using Salat which closely correspond to our model (Scala case classes). I can observe that the field ordering of the newly-inserted documents match the model.
  2. After some update, the fields in the document are updated to be in alphanumeric order (as documented).
  3. Salat is still able to deserialize the document as a proper object despite the field re-ordering.
  4. When we try to update the document using code like collection.update(old, new) as a form of optimistic locking – to ensure that the document hasn't been updated under us – this will fail even though the application's representation and the database's representation are isomorphic.

Although we ran into this issue using Salat+Casbah, I was able to reproduce it using the Mongo CLI:

# Insert a document with fields in alphanumeric order:
var o = {
    "_id" : NumberLong(12000009),
    "createdBy" : NumberLong(1185),
    "creationDate" : ISODate("2008-02-06T06:00:00Z"),
    "draftHamsVendor" : {
        "id" : NumberLong(120),
        "name" : "Johnny Appleseed",
    },
    "name" : "Johnny Appleseed",
    "users" : [
        NumberLong(1185)
    ]
}
 
db.test.save(o)
 
# Try to find the document using the model's field ordering:
var p = {
    "_id" : NumberLong(12000009),
    "creationDate" : ISODate("2008-02-06T06:00:00Z"),
    "createdBy" : NumberLong(1185),
    "name" : "Johnny Appleseed",
    "draftHamsVendor" : {
        "name" : "Johnny Appleseed",
        "id" : NumberLong(120),
    },
    "users" : [
        NumberLong(1185)
    ]
}
 
> db.test.findOne(p)
null

For some reason, this issue only surfaces with nested documents, despite claims that key order is significant in queries. For example, the above inserted object will be found by both of the following queries:

db.test.find({"creationDate" : ISODate("2008-02-06T06:00:00Z"), "createdBy" : NumberLong(1185)})
db.test.find({"createdBy" : NumberLong(1185), "creationDate" : ISODate("2008-02-06T06:00:00Z")}) 

I gather that both the significance of key ordering and the potential field re-ordering of documents are both by design, but the interaction between these two aspects seems to be a bad bug; the application runs fine until MongoDB needs to re-order fields in some documents. This is a subtle and unexpected bug, in part because the significance of key ordering in queries and documents is hardly documented at all.



 Comments   
Comment by Arun Bhalla [ 25/Apr/12 ]

To be clear, MongoDB reordered the fields in the embedded documents. My example simulates that in order to reproduce the issue. I think the intersection of these two MongoDB design choices described above surfaces as a bug, but we will pursue one of your suggestions.

Thanks.

Comment by Scott Hernandez (Inactive) [ 24/Apr/12 ]

You should use a version number combined with the id to do this kind of optimistic locking. See this article: http://code.google.com/p/morphia/wiki/MongoNewsletterArticleDec2010

Your document is the not the same as the query since you reordered fields in embedded docs and are comparing them. See the documentation on using docs as keys/for-lookups : http://www.mongodb.org/display/DOCS/Indexes#Indexes-UsingDocumentsasKeys

If you really wanted to make that a query you would need to use dot-notation like this:

db.test.find({
   ..., 
   "draftHamsVendor.name" : "Johnny Appleseed",
   "draftHamsVendor.id" : NumberLong(120),
   ...})

Comment by Arun Bhalla [ 24/Apr/12 ]

Some related issues:

Generated at Thu Feb 08 03:09:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.