-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 3.0.13
-
Component/s: WiredTiger
-
Labels:None
-
Fully Compatible
-
ALL
-
(copied to CRM)
ISSUE DESCRIPTION AND IMPACT
MongoDB 3.0.13 introduced a race condition in the WiredTiger engine that, under specific circumstances, may cause an invalid memory access (segmentation fault). This generally exhibits as an invalid memory access during a checkpoint operation (with __wt_checkpoint) in the stack trace.
This bug is most likely to fire in workloads that are doing concurrent, out-of-order insert operations. In particular, inserting MongoDB documents into a collection from multiple clients concurrently with an index on a field that is out-of-order with inserts may trigger this race in the index.
DIAGNOSIS AND AFFECTED VERSIONS
This bug is present in 3.0.13 only, and it may only affect users running MongoDB with the WiredTiger engine. No other versions are affected, and MMAPv1 users can't be affected by this bug.
REMEDIATION AND WORKAROUNDS
There are no known workarounds for this issue. The only way to be sure to avoid this bug is to upgrade to 3.0.14.
Original description
server version 3.0.13 running on Amazon AMI EC2 node. Starting seeing
Got signal: 11 (Segmentation fault). errors (Three so far in last 24 hours).
Have not seen this error in previous versions. The server run without a crash about a year on previous versions and handles long running (many hours, sometimes days) batch processes.
# mongod.conf systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log # Where and how to store data. storage: dbPath: /data/foundry-mongo engine: "wiredTiger" # how the process runs processManagement: fork: true # fork and run in background pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
Stack Trace:
2016-11-02T21:07:02.954+0000 F - Invalid access at address: 0xd7
2016-11-02T21:07:02.960+0000 F - Got signal: 11 (Segmentation fault).
0xfc6b02 0xfc63b3 0xfc6714 0x7f7d78a80100 0x7f7d77885d99 0x14033f0 0x1404542 0x13949f9 0x14268b2 0x14256d8 0x1427365 0x141a04c 0x13ae4dc 0x7f7d78a78dc5 0x7f7d7782ec9d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BC6B02","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"BC63B3"},{"b":"400000","o":"BC6714"},{"b":"7F7D78A71000","o":"F100"},{"b":"7F7D77738000","o":"14DD99"},{"b":"400000","o":"10033F0"},{"b":"400000","o":"1004542","s":"__wt_reconcile"},{"b":"400000","o":"F949F9","s":"__wt_cache_op"},{"b":"400000","o":"10268B2","s":"__wt_checkpoint"},{"b":"400000","o":"10256D8"},{"b":"400000","o":"1027365","s":"__wt_txn_checkpoint"},{"b":"400000","o":"101A04C"},{"b":"400000","o":"FAE4DC"},{"b":"7F7D78A71000","o":"7DC5"},{"b":"7F7D77738000","o":"F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.0.13", "gitVersion" : "44ae815066cdee7127ddeff34d3a04d75378fd61", "uname" : { "sysname" : "Linux", "release" : "4.4.19-29.55.amzn1.x86_64", "version" : "#1 SMP Mon Aug 29 23:29:40 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "2DFFFF43E9109891916A1167D058F32FF816CD02" }, { "b" : "7FFE2B376000", "elfType" : 3, "buildId" : "7EB0FBEF551697268DACDFC3E53A63ADDF93D154" }, { "b" : "7F7D78A71000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0836319AA81CDFE97DA2666963F62DE6A2A61346" }, { "b" : "7F7D78804000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AF827B6FD200DFDFE70B2BC8D66BBC9881E8817" }, { "b" : "7F7D7841E000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "83F15DBCD0653F417E98354BC1EED6F96A758367" }, { "b" : "7F7D78216000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2B3151901240D9E854E18E6D0B181C4D580ABA9C" }, { "b" : "7F7D78012000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7F7D77D10000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6E343508D15886FE83C438DF4560CE40BEB64B56" }, { "b" : "7F7D77AFA000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7F7D77738000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "5D38A77E8D79E98D717281031C39B9A341323BD1" }, { "b" : "7F7D78C8D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D931BA041229929E5F099514B20E36A70BD651" }, { "b" : "7F7D774EC000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7F7D77207000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7F7D77004000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F7D76DD2000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7F7D76BBC000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F7D769AD000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "b" : "7F7D767AA000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F7D76590000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1285F9516FFCF13FC00BD135C5634AF2EB16C80B" }, { "b" : "7F7D7636F000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xfc6b02]
mongod(+0xBC63B3) [0xfc63b3]
mongod(+0xBC6714) [0xfc6714]
libpthread.so.0(+0xF100) [0x7f7d78a80100]
libc.so.6(+0x14DD99) [0x7f7d77885d99]
mongod(+0x10033F0) [0x14033f0]
mongod(__wt_reconcile+0x6D2) [0x1404542]
mongod(__wt_cache_op+0x4E9) [0x13949f9]
mongod(__wt_checkpoint+0x992) [0x14268b2]
mongod(+0x10256D8) [0x14256d8]
mongod(__wt_txn_checkpoint+0x8B5) [0x1427365]
mongod(+0x101A04C) [0x141a04c]
mongod(+0xFAE4DC) [0x13ae4dc]
libpthread.so.0(+0x7DC5) [0x7f7d78a78dc5]
libc.so.6(clone+0x6D) [0x7f7d7782ec9d]
- is duplicated by
-
-
- Closed
-
- links to