Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26488

Bulk build of _id index uses excessive memory

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.3.15
    • Fix Version/s: 3.4.0-rc1
    • Component/s: Index Maintenance
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Query 2016-10-10

      Description

      To reproduce create a collection with a large number of documents (and no secondary indexes) and then reIndex. This also affects initial sync where a bulk build of the _id index is done. The index build uses >1 GB of memory, whereas it should only use 100 MB.

      • during the reindex operation from A-B we see >1 GB of memory allocated outside the WT cache.
      • the bulk of that is accounted for by stack239:

        heapProfile stack239: { 0: "tc_malloc", 1: "mongo::mongoMalloc", 2: "mongo::BtreeKeyGeneratorV1::getKeysImpl", 3: "mongo::BtreeKeyGenerator::getKeys", 4: "mongo::IndexAccessMethod::BulkBuilder::insert", 5: "mongo::MultiIndexBlock::insert", 6: "mongo::MultiIndexBlock::insertAllDocumentsInCollection", 7: "mongo::CmdReIndex::run", 8: "mongo::Command::run", 9: "mongo::Command::execCommand", 10: "mongo::runCommands", 11: "mongo::assembleResponse", 12: "mongo::ServiceEntryPointMongod::_sessionLoop", 13: "0x5654d9a7dea0", 14: "0x5654da69d73a", 15: "0x7fc92e81c6aa", 16: "clone" }
        

      It appears that memory is being allocated here in BtreeKeyGeneratorV1::getKeysImpl by using a BSONObjBuilder with a default initial size and never reducing the size, resulting in more memory being used than is accounted for by the sorter, which takes into account only the object size, not the allocated buffer size, similar to SERVER-25075.

      TBD whether this affects other indexes besides _id - so far in my testing it does not seem to, unclear why.

        Attachments

          Activity

            People

            Assignee:
            tess.avitabile Tess Avitabile
            Reporter:
            bruce.lucas Bruce Lucas
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: