-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 3.0.0-rc8
-
Component/s: Storage, WiredTiger
-
Fully Compatible
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Unlike prior RCs, WiredTiger-enabled mongod (rc7, rc8, 2/12 nightly 79492d9cc1885d74b31b5fe24194dbc227096d6e, rc9-pre ea5f871b550c1c3a8a5f0cd749fb47570557a067) in a standalone topology seems to grow the heap without bound until the Linux kernel kills the process. I assume this is the heap growing because dirty .data pages (like those in the WT cache) would simply be paged out (written to block I/O) by the kernel if an acute memory deficit occurs.
We found this in a sysbench-based longevity (stress) test after about 15 hours. To get started, Sysbench loads data (320 million docs) with 8 threads then goes into a 64 thread execute phase with a mix of read and write operations. The OOM occured during the 64-thread execute phase.
We did not see any OOM with a seven day YCSB test. YCSB runs with 8 threads.
We have seen this OOM when running against SSD block storage and with rotating magnetic hard disk.
We have seen the OOM a few times now in rc7 and rc8, only when running the sysbench 64 thread execute workload.
Reproduction steps:
A. procure a multi-socket machine with 12 cores, like a C3 8XL in EC2
B. start with a clean database and a standalone single node of rc8 mongod configured for wiredTiger
rm -rf /data/db/* ; numactl --interleave=all ./mongod --dbpath /data/db --logpath mongodb-sysbench.log --storageEngine wiredTiger --fork
C. checkout the sysbench benchmark and modify config.bash:
git clone https://github.com/tmcallaghan/sysbench-mongodb.git git checkout 7c8e12916fa1c7a58ff6b36c6ba4bfc28453104c
diff --git a/config.bash b/config.bash
index aaa346d..abf5fcb 100644
--- a/config.bash
+++ b/config.bash
@@ -39,7 +39,7 @@ export NUM_COLLECTIONS=16
# number of documents to maintain per collection
# valid values : integer > 0
-export NUM_DOCUMENTS_PER_COLLECTION=10000000
+export NUM_DOCUMENTS_PER_COLLECTION=20000000
# total number of documents to insert per "batch"
# valid values : integer > 0
@@ -55,7 +55,8 @@ export NUM_WRITER_THREADS=64
# run the benchmark for this many minutes
# valid values : intever > 0
-export RUN_TIME_MINUTES=10
+#export RUN_TIME_MINUTES=10
+export RUN_TIME_MINUTES=10080
export RUN_TIME_SECONDS=$[RUN_TIME_MINUTES*60]
# write concern for the benchmark client
@@ -106,12 +107,12 @@ export SYSBENCH_DISTINCT_RANGES=1
# number of indexed updates per sysbench "transaction"
# valid values : integer >= 0
-export SYSBENCH_INDEX_UPDATES=1
+export SYSBENCH_INDEX_UPDATES=3
# number of non-indexed updates per sysbench "transaction"
# valid values : integer >= 0
-export SYSBENCH_NON_INDEX_UPDATES=1
+export SYSBENCH_NON_INDEX_UPDATES=3
# number of delete/insert operations per sysbench "transaction"
# valid values : integer >= 0
-export SYSBENCH_INSERTS=1
+export SYSBENCH_INSERTS=2
diff --git a/src/jmongosysbenchexecute.java b/src/jmongosysbenchexecute.java
index bf35445..fa82032 100644
--- a/src/jmongosysbenchexecute.java
+++ b/src/jmongosysbenchexecute.java
@@ -164,8 +164,7 @@ public class jmongosysbenchexecute {
MongoClientOptions clientOptions = new MongoClientOptions.Builder().connectionsPerHost(2048).socketTimeout(60000).writeConcern(myWC).build();
ServerAddress srvrAdd = new ServerAddress(serverName,serverPort);
- MongoCredential credential = MongoCredential.createMongoCRCredential(userName, dbName, passWord.toCharArray());
- MongoClient m = new MongoClient(srvrAdd, Arrays.asList(credential));
+ MongoClient m = new MongoClient(srvrAdd);
logMe("mongoOptions | " + m.getMongoOptions().toString());
logMe("mongoWriteConcern | " + m.getWriteConcern().toString());
diff --git a/src/jmongosysbenchload.java b/src/jmongosysbenchload.java
index 420039e..cc8a4f1 100644
--- a/src/jmongosysbenchload.java
+++ b/src/jmongosysbenchload.java
@@ -116,8 +116,7 @@ public class jmongosysbenchload {
MongoClientOptions clientOptions = new MongoClientOptions.Builder().connectionsPerHost(2048).socketTimeout(60000).writeConcern(myWC).build();
ServerAddress srvrAdd = new ServerAddress(serverName,serverPort);
- MongoCredential credential = MongoCredential.createMongoCRCredential(userName, dbName, passWord.toCharArray());
- MongoClient m = new MongoClient(srvrAdd, Arrays.asList(credential));
+ MongoClient m = new MongoClient(srvrAdd);
logMe("mongoOptions | " + m.getMongoOptions().toString());
logMe("mongoWriteConcern | " + m.getWriteConcern().toString());
D. download the 2.12.4 Java driver for mongoDB
curl -O http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.12.4/mongo-java-driver-2.12.4.jar
E. run the workload
CLASSPATH=`pwd`/mongo-java-driver-2.12.4.jar numactl --interleave=all ./run.simple.bash
