Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-83145

Shared buffer fragment incorrectly tracks memory usage in freeUnused()

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 7.0.6, 6.0.14, 8.0.0-rc0, 7.3.0-rc2
    • Affects Version/s: 6.0.4, 6.2.0-rc0
    • Component/s: None
    • None
    • Storage Execution
    • Fully Compatible
    • ALL
    • v7.3, v7.0, v6.0
    • Hide

      const maxMemUsageMB = 50;
      const replSet = new ReplSetTest({
          nodes: 1,
          nodeOptions:

      {         setParameter: \{maxIndexBuildMemoryUsageMegabytes: maxMemUsageMB, heapProfilingEnabled: true}

          },
      });
      replSet.startSet();
      replSet.initiate();

      const primary = replSet.getPrimary();
      const testDB = primary.getDB('test');
      const coll = testDB.index_build_wildcard;

       
      const docs = 1000;
      const bigArr = new Array(100000).fill('x'.repeat(10));
      let d = 0;
      while (d < docs) {
      constbulk=coll.initializeUnorderedBulkOp();
      for (leti=0; i<10; i++)

      { bulk.insert(\{a: (d+i) +'x'.repeat(1000), arr: bigArr}

      );
      }
      d+=10;
      print(d);
      assert.commandWorked(bulk.execute());
      }

      coll.createIndex({"$**": 1});

      Show
      const maxMemUsageMB = 50; const replSet = new ReplSetTest({     nodes: 1,     nodeOptions: {         setParameter: \{maxIndexBuildMemoryUsageMegabytes: maxMemUsageMB, heapProfilingEnabled: true}     }, }); replSet.startSet(); replSet.initiate(); const primary = replSet.getPrimary(); const testDB = primary.getDB('test'); const coll = testDB.index_build_wildcard;   const docs = 1000; const bigArr = new Array(100000).fill('x'.repeat(10)); let d = 0; while (d < docs) { constbulk=coll.initializeUnorderedBulkOp(); for (leti=0; i<10; i++) { bulk.insert(\{a: (d+i) +'x'.repeat(1000), arr: bigArr} ); } d+=10; print(d); assert.commandWorked(bulk.execute()); } coll.createIndex({"$**": 1});
    • Execution Team 2023-12-11, Execution Team 2024-01-08, Execution Team 2024-01-22, Execution Team 2024-02-05

      We decrement from memUsage when we free unused buffers from _activeBuffers here. The calculation is incorrect and undercounts actual freed memory causing the memUsage stat to balloon. This can be a problem when the memusage stat grows past the in-memory limit for the sorter resulting in spilling to disk on every key added to the sorter, eventually leading to an OOM issue.

            Assignee:
            louis.williams@mongodb.com Louis Williams
            Reporter:
            shinyee.tan@mongodb.com Shin Yee Tan
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: