Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16546

Mongod memory grows until killed with wiredTiger repl set with small oplog

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 2.8.0-rc2
    • Fix Version/s: 2.8.0-rc4
    • Component/s: Replication, Storage
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Steps To Reproduce:
      Hide

      ycsb load of 400M documents. 32 threads. Mongod started with ops below. It's a single node replication set. Ignore the string shard in the replset name – that was left over from an earlier experiment.

       "cmdLineOpts": {
              "parsed": {
                  "replication": {
                      "oplogSizeMB": 200,
                      "replSetName": "BenchmarkShard_0"
                  },
                  "processManagement": {"fork": true},
                  "systemLog": {
                      "path": "/mongolog/BenchmarkCluster_BenchmarkShard_0_81/mongodb.log",
                      "destination": "file"
                  },
                  "config": "/data/BenchmarkCluster_BenchmarkShard_0_81/automation-mongod.conf",
                  "net": {
                      "port": 27000,
                      "http": {"enabled": false}
                  },
                  "storage": {
                      "engine": "wiredTiger",
                      "dbPath": "/data/BenchmarkCluster_BenchmarkShard_0_81"
                  }
              },

      Show
      ycsb load of 400M documents. 32 threads. Mongod started with ops below. It's a single node replication set. Ignore the string shard in the replset name – that was left over from an earlier experiment. "cmdLineOpts": { "parsed": { "replication": { "oplogSizeMB": 200, "replSetName": "BenchmarkShard_0" }, "processManagement": {"fork": true}, "systemLog": { "path": "/mongolog/BenchmarkCluster_BenchmarkShard_0_81/mongodb.log", "destination": "file" }, "config": "/data/BenchmarkCluster_BenchmarkShard_0_81/automation-mongod.conf", "net": { "port": 27000, "http": {"enabled": false} }, "storage": { "engine": "wiredTiger", "dbPath": "/data/BenchmarkCluster_BenchmarkShard_0_81" } },

      Description

      Running the load phase for ycsb with a single node repl set with small oplog, the memory grows consistently until mongod is killed.

      Running MCI build for githash e067fff4e3f3079d070ec168f32c24db9a51a944
      and RC2.

      Mongod on Amazon EC2 Linux AMI. c3.4xlarge server instance. m3.medium instance for ycsb client. Data on SSD. 32 GB of RAM. Uses up to 30 GB before being killed or descheduled.

        Attachments

        1. 16546.js
          1 kB
        2. 16546.stats.tgz
          128 kB
        3. gdbmon.rc3.log
          73.66 MB
        4. iostat.rc3.log
          1.15 MB
        5. mongodb.rc3.log
          4.26 MB
        6. rc3-single-repl.pdf
          153 kB
        7. rc3-single-repl-dbstats.pdf
          81 kB
        8. rc3-single-repl-hardware.pdf
          141 kB
        9. Screen Shot 2014-12-18 at 10.37.02 am.png
          Screen Shot 2014-12-18 at 10.37.02 am.png
          49 kB
        10. ss.rc3.log
          23.95 MB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: