[SERVER-8694] mongod generated _id values can grow document beyond 16mb limit, which can cause consequent problems Created: 24/Feb/13  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Write Ops
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: query-44-grooming
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File no_id_gen.patch     File test.cpp    
Assigned Teams:
Query Execution
Operating System: ALL
Participants:

 Description   

Mongod enforces a 16mb document size limit. However, the size is checked before mongod inserts an _id into documents lacking an _id. If the document size after insertion of a mongod generated _id exceeds 16mb, the resulting improperly sized document will still be inserted.

The existence of a document exceeding the maximum document size can cause subsequent problems. For example, a size changing modifier update of the document may fail because the size of the resulting document exceeds the maximum document size. There may be other errors that can result because of the improperly sized document, for example during replication.

Test

#include <iostream>
#include <cstdlib>
 
#include "mongo/client/dbclient.h"
 
using namespace std;
using namespace mongo;
 
int main( int argc, const char **argv ) {
 
    const char* port = "27017";
 
    mongo::DBClientConnection conn;
    string errmsg;
    if ( ! conn.connect( string( "127.0.0.1:" ) + port , errmsg ) ) {
        cout << "couldn't connect : " << errmsg << endl;
        return EXIT_FAILURE;
    }
 
    conn.dropCollection( "test.test" );
    
    string large( 16 * 1024 * 1024 - 20, 'z' );
 
    // Attempt to insert a document with a client generated OID (fails).
    conn.insert( "test.test", BSON( "_id" << OID::gen() << "a" << 1 << "z" << large ) );
    log() << "insert, client oid err: " << conn.getLastError() << endl;
    
    // Insert a document with a server generated OID (succeeds).
    conn.insert( "test.test", BSON( "a" << 1 << "z" << large ) );
    log() << "insert, server oid err: " << conn.getLastError() << endl;
    log() << "insert, server oid doc size: " <<
        conn.findOne( "test.test", BSONObj() ).objsize() << endl;
 
    // Attempt to update the recently inserted document.
    conn.ensureIndex( "test.test", BSON( "a" << 1 ) );
    conn.update( "test.test", BSONObj(), BSON( "$set" << BSON( "a" << 2 ) ) );
    log() << "update err: " << conn.getLastError() << endl;
}

Output

Sun Feb 24 14:03:18.891 insert, client oid err: object to insert too large
Sun Feb 24 14:03:18.973 insert, server oid err: 
Sun Feb 24 14:03:19.017 insert, server oid doc size: 16777233
Sun Feb 24 14:03:19.099 update err: $ operator made object too large



 Comments   
Comment by Asya Kamsky [ 06/Feb/18 ]

I just tested this in the shell and I can reproduce it just using insert command directly.

Updates are accepted as long as the type of the updated field is not being changed (whether it's getting bigger or smaller). But anything that changes the type of the field/size of document causes an error.

db.testbig.drop()
true
test@127.0.0.1:27017(3.6.0) > var doc={a:1, x:new Array(16*1024*1024-28).join('x')}
test@127.0.0.1:27017(3.6.0) > Object.bsonsize(doc)
16777211
test@127.0.0.1:27017(3.6.0) > db.runCommand({insert:"testbig", documents:[ doc ]})
{ "n" : 1, "ok" : 1 }
test@127.0.0.1:27017(3.6.0) > var docin=db.testbig.findOne();
test@127.0.0.1:27017(3.6.0) > Object.bsonsize(docin)
16777228
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$inc:{a:1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$inc:{a:1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$inc:{a:1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$inc:{a:1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$set:{a:true}})
WriteResult({
	"nMatched" : 0,
	"nUpserted" : 0,
	"nModified" : 0,
	"writeError" : {
		"code" : 17419,
		"errmsg" : "Resulting document after update is larger than 16777216"
	}
})
test@127.0.0.1:27017(3.6.0) > db.testbig.update({},{$unset:{a:1}})
WriteResult({
	"nMatched" : 0,
	"nUpserted" : 0,
	"nModified" : 0,
	"writeError" : {
		"code" : 17419,
		"errmsg" : "Resulting document after update is larger than 16777216"
	}
})

Generated at Thu Feb 08 03:18:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.