[SERVER-26488] Bulk build of _id index uses excessive memory Created: 05/Oct/16  Updated: 19/Nov/16  Resolved: 06/Oct/16

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.3.15
Fix Version/s: 3.4.0-rc1

Type: Bug Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Tess Avitabile (Inactive)
Resolution: Done Votes: 0
Labels: DF
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File index.png    
Issue Links:
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Query 2016-10-10
Participants:

 Description   

To reproduce create a collection with a large number of documents (and no secondary indexes) and then reIndex. This also affects initial sync where a bulk build of the _id index is done. The index build uses >1 GB of memory, whereas it should only use 100 MB.

  • during the reindex operation from A-B we see >1 GB of memory allocated outside the WT cache.
  • the bulk of that is accounted for by stack239:

    heapProfile stack239: { 0: "tc_malloc", 1: "mongo::mongoMalloc", 2: "mongo::BtreeKeyGeneratorV1::getKeysImpl", 3: "mongo::BtreeKeyGenerator::getKeys", 4: "mongo::IndexAccessMethod::BulkBuilder::insert", 5: "mongo::MultiIndexBlock::insert", 6: "mongo::MultiIndexBlock::insertAllDocumentsInCollection", 7: "mongo::CmdReIndex::run", 8: "mongo::Command::run", 9: "mongo::Command::execCommand", 10: "mongo::runCommands", 11: "mongo::assembleResponse", 12: "mongo::ServiceEntryPointMongod::_sessionLoop", 13: "0x5654d9a7dea0", 14: "0x5654da69d73a", 15: "0x7fc92e81c6aa", 16: "clone" }
    

It appears that memory is being allocated here in BtreeKeyGeneratorV1::getKeysImpl by using a BSONObjBuilder with a default initial size and never reducing the size, resulting in more memory being used than is accounted for by the sorter, which takes into account only the object size, not the allocated buffer size, similar to SERVER-25075.

TBD whether this affects other indexes besides _id - so far in my testing it does not seem to, unclear why.



 Comments   
Comment by Githook User [ 06/Oct/16 ]

Author:

{u'username': u'tessavitabile', u'name': u'Tess Avitabile', u'email': u'tess.avitabile@mongodb.com'}

Message: SERVER-26488 BSONObj buffer size used by _id index keys should match object size
Branch: master
https://github.com/mongodb/mongo/commit/e00e6edde50c8eb1ac31ec70e7e8d91c1c6742e6

Comment by Eric Milkie [ 05/Oct/16 ]

This is actually great news as it's an easy bug to fix, and it is in code specific to _id index builds and was newly added when we added collation support. Bruce, this is a great find.

Generated at Thu Feb 08 04:12:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.