|
Ramon ,can not upload logs and files cause firewall stop this action
but i paste the full context about this panic things and describe on what condition it happend
condition , just large amount of insert ops , no query, update , delete, wiredtiger panic happened on we have insert about 500GB data(5 million records) into collection
first ,normal starting mongod configuration log:
2015-05-27T08:20:17.624Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=62G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=zlib),checkpoint=(wait
|
=60,log_size=2GB),statistics_log=(wait=0),
|
2015-05-27T08:20:17.885Z I CONTROL [initandlisten] MongoDB starting : pid=13186 port=27017 dbpath=/data 64-bit host=ipattern-mongodb13.iad1
|
2015-05-27T08:20:17.885Z I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
|
2015-05-27T08:20:17.885Z I CONTROL [initandlisten]
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten]
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten]
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] db version v3.0.3
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] git version: b40106b36eecd1b4407eb1ad1af6bc60593c6105
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] build info: Linux ip-10-182-86-231 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 BOOST_LIB_VERSION=1_49
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] allocator: tcmalloc
|
2015-05-27T08:20:17.886Z I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { JSONPEnabled: true, RESTInterfaceEnabled: true, enabled: true }, maxIncomingConnections: 10240, port: 27017, unixDomainSocket: { e
|
nabled: true } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { oplogSizeMB: 20480, replSetName: "setA" }, security: { authorization: "enabled", keyFile: "/data/keyfile" }, storage: { dbPa
|
th: "/data", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "zlib" }, engineConfig: { journalCompressor: "zlib" }, indexConfig: { prefixCompression: true } } },
|
systemLog: { destination: "file", logAppend: true, logRotate: "rename", path: "/var/log/mongodb/mongod.log", timeStampFormat: "iso8601-utc" } }
|
second , wiredtiger engine panic context log:
2015-05-31T12:46:17.692Z I WRITE [conn79] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 290ms
|
2015-05-31T12:46:17.692Z I WRITE [conn45] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 292ms
|
2015-05-31T12:46:17.694Z I WRITE [conn48] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 248ms
|
2015-05-31T12:46:17.696Z I WRITE [conn17] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 217ms
|
2015-05-31T12:46:17.697Z I WRITE [conn23] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 217ms
|
2015-05-31T12:46:17.699Z I WRITE [conn98] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 203ms
|
2015-05-31T12:46:17.703Z E STORAGE [WiredTigerRecordStoreThread for local.oplog.rs] WiredTiger (22) [1433076377:695131][9099:0x7f724b942700], file:local/collection/6-7285080005515704958.wt, cursor.next: session 0x4770000: clear hazard pointer: 0x15f79cf960: not found: Invalid argument
|
2015-05-31T12:46:17.703Z E STORAGE [WiredTigerRecordStoreThread for local.oplog.rs] WiredTiger (-31804) [1433076377:703658][9099:0x7f724b942700], file:local/collection/6-7285080005515704958.wt, cursor.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2015-05-31T12:46:17.704Z I - [conn103] Fatal Assertion 28559
|
2015-05-31T12:46:17.704Z I - [WiredTigerRecordStoreThread for local.oplog.rs] Fatal Assertion 28558
|
2015-05-31T12:46:17.707Z I WRITE [conn44] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 214ms
|
2015-05-31T12:46:17.707Z I - [conn44] Fatal Assertion 28559
|
2015-05-31T12:46:17.707Z I - [conn110] Fatal Assertion 28559
|
2015-05-31T12:46:17.708Z I - [conn104] Fatal Assertion 28559
|
2015-05-31T12:46:17.710Z I - [conn13] Fatal Assertion 28559
|
2015-05-31T12:46:17.710Z I - [conn48] Fatal Assertion 28559
|
2015-05-31T12:46:17.710Z I WRITE [conn100] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 214ms
|
2015-05-31T12:46:17.710Z I - [conn100] Fatal Assertion 28559
|
2015-05-31T12:46:17.712Z I WRITE [conn54] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 213ms
|
2015-05-31T12:46:17.712Z I - [conn54] Fatal Assertion 28559
|
2015-05-31T12:46:17.715Z I WRITE [conn70] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 322ms
|
2015-05-31T12:46:17.715Z I - [conn70] Fatal Assertion 28559
|
2015-05-31T12:46:17.718Z I - [conn64] Fatal Assertion 28559
|
2015-05-31T12:46:17.719Z I WRITE [conn19] insert ipattern.sample_attribute ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 228ms
|
2015-05-31T12:46:17.719Z I - [conn19] Fatal Assertion 28559
|
2015-05-31T12:46:17.719Z I - [conn34] Fatal Assertion 28559
|
2015-05-31T12:46:17.721Z I WRITE [conn77] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 644ms
|
2015-05-31T12:46:17.721Z I - [conn77] Fatal Assertion 28559
|
2015-05-31T12:46:17.721Z I WRITE [conn49] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 223ms
|
2015-05-31T12:46:17.721Z I - [conn49] Fatal Assertion 28559
|
2015-05-31T12:46:17.722Z I - [conn72] Fatal Assertion 28559
|
2015-05-31T12:46:17.723Z I WRITE [conn95] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 323ms
|
2015-05-31T12:46:17.723Z I - [conn95] Fatal Assertion 28559
|
2015-05-31T12:46:17.725Z I WRITE [conn113] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 606ms
|
2015-05-31T12:46:17.725Z I - [conn113] Fatal Assertion 28559
|
2015-05-31T12:46:17.727Z I - [conn91] Fatal Assertion 28559
|
2015-05-31T12:46:17.729Z I - [conn114] Fatal Assertion 28559
|
2015-05-31T12:46:17.730Z I - [conn40] Fatal Assertion 28559
|
2015-05-31T12:46:17.730Z I WRITE [conn41] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 277ms
|
2015-05-31T12:46:17.730Z I - [conn41] Fatal Assertion 28559
|
2015-05-31T12:46:17.731Z I - [conn102] Fatal Assertion 28559
|
2015-05-31T12:46:17.731Z I WRITE [conn43] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 275ms
|
2015-05-31T12:46:17.731Z I - [conn43] Fatal Assertion 28559
|
2015-05-31T12:46:17.732Z I - [conn9] Fatal Assertion 28559
|
2015-05-31T12:46:17.734Z I - [conn86] Fatal Assertion 28559
|
2015-05-31T12:46:17.735Z I - [conn109] Fatal Assertion 28559
|
2015-05-31T12:46:17.736Z I - [conn28] Fatal Assertion 28559
|
2015-05-31T12:46:17.738Z I WRITE [conn75] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 559ms
|
2015-05-31T12:46:17.738Z I - [conn75] Fatal Assertion 28559
|
2015-05-31T12:46:17.738Z I - [conn21] Fatal Assertion 28559
|
2015-05-31T12:46:17.738Z I WRITE [conn39] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 411ms
|
2015-05-31T12:46:17.738Z I - [conn39] Fatal Assertion 28559
|
2015-05-31T12:46:17.740Z I - [conn60] Fatal Assertion 28559
|
2015-05-31T12:46:17.741Z I WRITE [conn10] insert ipattern.dex_string ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 466ms
|
2015-05-31T12:46:17.741Z I - [conn10] Fatal Assertion 28559
|
2015-05-31T12:46:17.741Z I - [conn25] Fatal Assertion 28559
|
2015-05-31T12:46:17.746Z I - [conn24] Fatal Assertion 28559
|
2015-05-31T12:46:17.746Z I CONTROL [conn44]
|
0xf6a889 0xf08321 0xeec161 0xd88fd0 0xd83586 0xd7d969 0xd7d9f9 0xaff5f7 0x7fad23 0xf1bd5b 0x7f7258cc7846 0x7f72578093cd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6A889"},{"b":"400000","o":"B08321"},{"b":"400000","o":"AEC161"},{"b":"400000","o":"988FD0"},{"b":"400000","o":"983586"},{"b":"400000","o":"97D969"},{"b":"400000","o":"97D9F9"},{"b":"400000","o":"6FF5F7"},{"b":"400000","o":"3FAD23"},{"b":"400000","o":"B1BD5B"},{"b":"7F7258CC0000","o":"7846"},{"b":"7F7257722000","o":"E73CD"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.x86_64", "version" : "#1 SMP Fri Nov 22 03:15:09 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "108A63CA14A4BD5E599BAC10885DBD3A85DA5439" }, { "b" : "7FFF38241000", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "b" : "7F7258CC0000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4E32EC9F935CEA5A74F7AFF8CDB6A92B823634EC" }, { "b" : "7F7258A54000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "DAF114120DA5C9DBEB1E5A704CE83ACB9B8B7B54" }, { "b" : "7F7258671000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F523EAC46D068A8E0869CF93BCD84B414937993A" }, { "b" : "7F7258469000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "FA33B7ED6CC07555C335C237AD2048AE7E756B3F" }, { "b" : "7F7258265000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "55DB1A52AF46A030846823B5BC73D4CD7E4AFB62" }, { "b" : "7F7257F5F000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "F07F2E7CF4BFB393CC9BBE8CDC6463652E14DB07" }, { "b" : "7F7257CDA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "CB0450E602989E3A4436577E2146F3A849E619C9" }, { "b" : "7F7257AC4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "246C3BAB0AB093AFD59D34C8CBF29E786DE4BE97" }, { "b" : "7F7257722000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CC5049764632DF2247ED7AB10F80D04673772834" }, { "b" : "7F7258EDC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D5B026B9F73EEFDCBE957C6F6610CD37AB91F180" }, { "b" : "7F72574DE000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B7F7FF323B3A4A12310A6285412F01ACE8C74E47" }, { "b" : "7F72571F8000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "7920917F74AFAD0B8CB197CABBE472AF39D94C34" }, { "b" : "7F7256FF4000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "b" : "7F7256DC8000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "b" : "7F7256BB2000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7F72569A7000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "C8D01C2839F6950988CE32B4266A8F89C521ACB0" }, { "b" : "7F72567A4000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7F725658B000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2F19213416DEA01BFBE2DDB807B5E4F52230B309" }, { "b" : "7F725636C000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6a889]
|
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf08321]
|
mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeec161]
|
mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x2D0) [0xd88fd0]
|
mongod(_ZN5mongo22WiredTigerSessionCache14releaseSessionEPNS_17WiredTigerSessionE+0x686) [0xd83586]
|
mongod(_ZN5mongo22WiredTigerRecoveryUnitD1Ev+0x39) [0xd7d969]
|
mongod(_ZN5mongo22WiredTigerRecoveryUnitD0Ev+0x9) [0xd7d9f9]
|
mongod(_ZN5mongo20OperationContextImplD1Ev+0x37) [0xaff5f7]
|
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x603) [0x7fad23]
|
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf1bd5b]
|
libpthread.so.0(+0x7846) [0x7f7258cc7846]
|
libc.so.6(clone+0x6D) [0x7f72578093cd]
|
----- END BACKTRACE -----
|
2015-05-31T12:46:17.746Z I - [conn44]
|
|
***aborting after fassert() failure
|
third , i can restart mongod again after i manually remove mongod.lock file
maybe after a few days insertion , it will panic again
BTW ,it seems alike with this issue
https://jira.mongodb.org/browse/WT-1937
|