[SERVER-5944] modifier failure with empty string field name and nested fields Created: 28/May/12  Updated: 06/Dec/22  Resolved: 20/May/16

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: 2.0.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tasos Bitsios Assignee: Backlog - Query Team (Inactive)
Resolution: Duplicate Votes: 1
Labels: $set, update
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tests with CLI PHP but confirmed most on shell, too. Two environments checked: Single mongod v2.0.3 on OSX, and v2.0.5 on Linux (EC2 micro instance 2.6.35.14).


Attachments: Text File multi-set.log     Text File single-set.log     File test-update-emptystringkey.tgz    
Issue Links:
Depends
depends on SERVER-6852 Disallow empty path components in doc... Backlog
Duplicate
is duplicated by SERVER-6399 Refactor update() code Closed
Related
related to SERVER-2051 Disallow empty string keys Open
Assigned Teams:
Query
Operating System: ALL
Participants:

 Description   

Test:

t.save( { a:{ '':1 } } );
t.update( {}, { $set:{ 'a.b':2 } } );
printjson( db.getLastError() );
printjson( t.findOne() );

Documents or subdocuments with empty string keys can cause $set updates to that document to fail.

Specifically, $set ing a new field at the same level as an empty-string key, will silently fail. Same for updates to children of siblings to empty-string key ("nephew/niece" fields).

This behaviour sometimes changes depending on the number of updates inside $set.

The test script output should make it clear enough:

      • Single - $set updates *** //php test.php 1 multi_tests
        [...]

        	*** TEST 3 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.m.new":1}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"new":1,"rock":1},"o":{"funny":2}}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        	Test FAILED
        

        	*** TEST 4 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.p.p":10}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"p":{"p":10}}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        	Test FAILED
        

        	*** TEST 5 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.p":1}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"p":1}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        	Test FAILED
        

        	*** TEST 6 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.other":10}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"other":10}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        	Test FAILED
        

        *** Multi - $set updates *** //php test.php 1 multi_tests
        [...]

        	*** TEST 3 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.m.new":1,"tags.m.newer":1}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"new":1,"newer":1,"rock":1},"o":{"funny":2}}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"new":1,"newer":1,"rock":1},"o":{"funny":2}}}
        	Test passed //no problem creating two new tags (one had failed)
        

        	*** TEST 4 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.p.p":10,"tags.p.q":11}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"p":{"p":10,"q":11}}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"p":{"p":10,"q":11}}}
        	Test passed
        

        	*** TEST 5 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.p":1,"tags.q":1}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"p":1,"q":1}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"q":1}}
        	Test FAILED //note one update made it through
        

        	*** TEST 6 ***
        original:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2}}}
        update: 	{"$set":{"tags.other":10,"tags.music":11}}
        expected:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"music":11,"o":{"funny":2},"other":10}}
        received:	{"_id":"1","tags":{"":{"oops":1},"m":{"rock":1},"o":{"funny":2},"other":10}}
        	Test FAILED //note one update made it through
        

If the empty-string fields are removed from the original object, all tests succeed (run with 0 instead of 1)

Attached tests are PHP (PHPMongo 1.2.9) but behaviour is the same on the shell.



 Comments   
Comment by J Rassi [ 20/May/16 ]

I've confirmed Asya's finding that this issue has been fixed. Closing as a dup of SERVER-6399.

Note also that the documents listed in the description of this ticket (with empty field names) will be disallowed for insert when SERVER-2051 and SERVER-6852 are implemented.

Comment by Asya Kamsky [ 18/May/16 ]

Every single test mentioned in the description and in the comments now passes. I'm guessing it was update rewrite in 2.6 that fixed it, but I'm testing with latest 3.3.6.

Suggest we close this leaving the related ones to take care of disallowing empty key names.

Comment by Jesse Pasichnyk [ 21/May/14 ]

Just an update, that the clean() function did work for me as expected, just took forever.

Comment by Jesse Pasichnyk [ 19/May/14 ]

Scott, this bug caused us countless hours of wasted time in the last couple weeks, trying to get to the root cause of missing data in our documents. The current "silently fail" behavior is very bad, as it causes issues to go unnoticed for some time, allowing for data loss. Had it simply thrown an error out, we would have identified the issue when it first manifested...

I just wanted to throw my 2 cents in here, and suggest that this bug get some attention in the near future, before it causes some real damage.

Thanks,
Jesse

Comment by Jesse Pasichnyk [ 19/May/14 ]

Thanks Tasos. My collections are quite large, but I'll give it a shot on a copy of one of them and see how it performs.

Comment by Tasos Bitsios [ 17/May/14 ]

I remember trying to do this in a single query and not being able to. I ended up using a clean() function and a cursor on the mongo shell.

This version replaces blank keys with the string BLANK. Uncomment the appropriate line to delete the blank keys altogether.

function clean( x ) {
    for( var key in x ) {
        if (x.hasOwnProperty(key)) {
            var val = x[key];
            if ( typeof val == "object" && val ) {
                x[key] = clean(val);
            }
            if ( key === "" ) {
                x.BLANK = x[key]; //comment me out to remove entirely
                delete x[key];
            } 
        }
    }
    return x;

To use it over db.test:

mongo clean.js --shell
> var d = db.test, c = d.find(), item; while( c.hasNext() ) { d.save( clean( c.next() ) ); }

Comment by Jesse Pasichnyk [ 16/May/14 ]

I just ran into this issue on 2.4.10, while using $inc commands, so it looks like it's not isolated to only $set.

Anyone have a suggestion on how to clean up these field names? I'm trying to $unset them, but get the error "Invalid mod field name, may not end in a period". Suggestions would be appreciated.

Comment by Tasos Bitsios [ 30/Aug/12 ]

Please let me know if I can do anything further to help you with this.

Comment by Tasos Bitsios [ 28/May/12 ]

I am seeing the same results with the 2.0 and 2.1 nightlies from www.mongodb.org/downloads

db version v2.0.6-rc1-pre-, pdfile version 4.5
git version: eba41d31367a4ff057607750ae66447cb724af92
build info: Darwin erh2.10gen.cc 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40

and

db version v2.1.2-pre-, pdfile version 4.5
git version: 9e4abe3c2ad87c8ac13024b8525a14e6e9989b07
build info: Darwin erh2.10gen.cc 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49

Comment by Tasos Bitsios [ 28/May/12 ]

The nightly I was getting was 2.1 after all - where can I get a 2.2 one? I could only find the release notes on the site.

Comment by Tasos Bitsios [ 28/May/12 ]

Apologies if it is a duplicate, I tried a few combinations or "empty-string" and "zero-length key" but I didn't see anything relevant.

I am getting the nightly and will try it now.

Comment by Eliot Horowitz (Inactive) [ 28/May/12 ]

I believe this if fixed in head.
Can you try a 2.2 nightly?
Will also look for the case, just can't seem to find it quickly.

Generated at Thu Feb 08 03:10:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.