Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-92904

Reply size exceeds BSONObjMaxInternalSize whilst batch is within BSONObjMaxUserSize

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 7.0.13
    • Component/s: Query Execution
    • None
    • Query Execution
    • ALL
    • Hide

      So far I could reproduce it using the data from the linked HELP-62024 ticket with the following script:

      (function(){
"use strict"

let csPipeline = [{$project:{updateDescription:0}},{$match:{$and:[{"ns.db":{$not:{$regex:"^(admin|local|config|mongosync_reserved_for_internal_use|mongosync_internal_.+)$"}}},{"ns.coll":{$not:{$regex:"^system\\."}}}]}}];
let csOptions = {fullDocument:"updateLookup",showExpandedEvents:true,startAtOperationTime: Timestamp(1721256543, 307), batchSize: 0};
let csCursor = db.getMongo().watch(csPipeline, csOptions);
print(tojson(csCursor.next()));

})();
      Show
      So far I could reproduce it using the data from the linked HELP-62024 ticket with the following script: (function(){ "use strict" let csPipeline = [{$project:{updateDescription:0}},{$match:{$and:[{ "ns.db" :{$not:{$regex: "^(admin|local|config|mongosync_reserved_for_internal_use|mongosync_internal_.+)$" }}},{ "ns.coll" :{$not:{$regex: "^system\\." }}}]}}]; let csOptions = {fullDocument: "updateLookup" ,showExpandedEvents: true ,startAtOperationTime: Timestamp(1721256543, 307), batchSize: 0}; let csCursor = db.getMongo().watch(csPipeline, csOptions); print(tojson(csCursor.next())); })();
    • QE 2024-09-30

      The generateBatch() method in getmore_cmd.cpp code link allows creating a result batch BSON object of size 16 757 334 containing 28099 documents which is still within BSONObjMaxUserSize (16MB = 16 777 216), however, the reply object containing that batch is 16 798 526 which exceeds BSONObjMaxInternalSize (16MB + 16KB = 16 793 600) and so the query fails with BSONObjectTooLarge error.

      The problem behind this issue is a large postBatchResumeToken (41 147 bytes) which is unaccounted when estimating the reply size. Change event resume tokens can be large (larger than 16KB "buffer" we accommodate in BSONObjMaxInternalSize), because they encode the document ID, which can also be large. This error is a bug since we are controlling how big batches can grow, and the proper solution in this case (without any error) is to go for a smaller batch, e.g., stop on 28098 documents.

      The problem has manifested in mongosync internal test running the current v7.0 with enableTestCommands=1. The failure call stack:

      #0  BSONObj::_assertInvalid() at bsonobj.cpp:94
#1  BSONObj::init<mongo::BSONObj::DefaultSizeTrait> () at bsonobj.h:720
#2  BSONObj::BSONObj<mongo::BSONObj::DefaultSizeTrait> () at bsonobj.h:144
#3  BSONObjBuilderBase<mongo::BSONObjBuilder, mongo::BufBuilder>::done<mongo::BSONObj::DefaultSizeTrait> () at bsonobjbuilder.h:579
#4  BSONObjBuilder::obj<mongo::BSONObj::DefaultSizeTrait> () at bsonobjbuilder.h:775
#5  BSONObj::removeField () at bsonobj.cpp:724
#6  (anonymous namespace)::GetMoreCmd::Invocation::validateResult () at getmore_cmd.cpp:800
#7  (anonymous namespace)::GetMoreCmd::Invocation::run () at getmore_cmd.cpp:792
...

      If test commands are not enabled it will fail somewhere else.

            Assignee:
            romans.kasperovics@mongodb.com Romans Kasperovics
            Reporter:
            romans.kasperovics@mongodb.com Romans Kasperovics
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: