2016-09-22T14:41:20.932+0800 I CONTROL [main] ***** SERVER RESTARTED ***** 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] MongoDB starting : pid=4180 port=27017 dbpath=/usr/local/mongodb/data 64-bit host=iZ625ztxpiuZ 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] db version v3.2.7 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] allocator: tcmalloc 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] modules: none 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] build environment: 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] distmod: rhel62 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] distarch: x86_64 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] target_arch: x86_64 2016-09-22T14:41:20.962+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/mongodb.conf", net: { maxIncomingConnections: 20000, port: 27017 }, processManagement: { fork: true, pidFilePath: "/usr/local/mongodb/mongod.pid" }, storage: { dbPath: "/usr/local/mongodb/data", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/mongodb/logs/mongodb.log" } } 2016-09-22T14:41:20.962+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-09-22T14:41:20.976+0800 E STORAGE [initandlisten] WiredTiger (0) [1474526480:976372][4180:0x7f934706ad40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1717798939 doesn't match expected checksum of 3759426581 2016-09-22T14:41:20.976+0800 E STORAGE [initandlisten] WiredTiger (0) [1474526480:976406][4180:0x7f934706ad40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2016-09-22T14:41:20.976+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1474526480:976417][4180:0x7f934706ad40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-09-22T14:41:20.976+0800 I - [initandlisten] Fatal Assertion 28558 2016-09-22T14:41:20.976+0800 I - [initandlisten] ***aborting after fassert() failure 2016-09-22T14:41:20.995+0800 F - [initandlisten] Got signal: 6 (Aborted). 0x1337cb2 0x1336be9 0x13373f2 0x393e60f7e0 0x393e2325e5 0x393e233dc5 0x12bea52 0x10af8d3 0x1a9f62c 0x1a9faed 0x1a9fed4 0x19d18fc 0x19d1e73 0x19cee15 0x19d2eb9 0x19ef92c 0x1a257c0 0x1a9e357 0x1a9e899 0x1a9e9bb 0x1a33db8 0x1a9b515 0x1a660ef 0x1a661ee 0x1a223a1 0x1097517 0x1093893 0xfbad38 0x9970dd 0x99acbd 0x393e21ed1d 0x9937e9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F37CB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F36BE9"},{"b":"400000","o":"F373F2"},{"b":"393E600000","o":"F7E0"},{"b":"393E200000","o":"325E5","s":"gsignal"},{"b":"393E200000","o":"33DC5","s":"abort"},{"b":"400000","o":"EBEA52","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF8D3"},{"b":"400000","o":"169F62C","s":"__wt_eventv"},{"b":"400000","o":"169FAED","s":"__wt_err"},{"b":"400000","o":"169FED4","s":"__wt_panic"},{"b":"400000","o":"15D18FC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15D1E73","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15CEE15","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15D2EB9"},{"b":"400000","o":"15EF92C","s":"__wt_btree_open"},{"b":"400000","o":"16257C0","s":"__wt_conn_btree_open"},{"b":"400000","o":"169E357","s":"__wt_session_get_btree"},{"b":"400000","o":"169E899","s":"__wt_session_get_btree"},{"b":"400000","o":"169E9BB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1633DB8","s":"__wt_curfile_open"},{"b":"400000","o":"169B515"},{"b":"400000","o":"16660EF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"16661EE","s":"__wt_metadata_cursor"},{"b":"400000","o":"16223A1","s":"wiredtiger_open"},{"b":"400000","o":"C97517","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C93893"},{"b":"400000","o":"BBAD38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5970DD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"59ACBD","s":"main"},{"b":"393E200000","o":"1ED1D","s":"__libc_start_main"},{"b":"400000","o":"5937E9"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-642.1.1.el6.x86_64", "version" : "#1 SMP Tue May 31 21:57:07 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D6BB40ACB6CFEAA43C3A651DDD8E547FCFE19166" }, { "b" : "7FFD5C99E000", "elfType" : 3, "buildId" : "7F719AF91EE951B4FCB6647E7868F95F766A616B" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E83FD82B1AE56360B270189F636423FB3016594" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE4607C86FEEC14F47B46D0DCC" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "F14F3F5B55FC325FDFA7FB6F22F73CA1C6977379" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DB04BDF55FE66BADD1081D833C17B4881E0EEDB9" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EEE9AC218BFC460" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1337cb2] mongod(+0xF36BE9) [0x1336be9] mongod(+0xF373F2) [0x13373f2] libpthread.so.0(+0xF7E0) [0x393e60f7e0] libc.so.6(gsignal+0x35) [0x393e2325e5] libc.so.6(abort+0x175) [0x393e233dc5] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12bea52] mongod(+0xCAF8D3) [0x10af8d3] mongod(__wt_eventv+0x42C) [0x1a9f62c] mongod(__wt_err+0x8D) [0x1a9faed] mongod(__wt_panic+0x24) [0x1a9fed4] mongod(__wt_block_extlist_read+0x6C) [0x19d18fc] mongod(__wt_block_extlist_read_avail+0x33) [0x19d1e73] mongod(__wt_block_checkpoint_load+0x3C5) [0x19cee15] mongod(+0x15D2EB9) [0x19d2eb9] mongod(__wt_btree_open+0xC7C) [0x19ef92c] mongod(__wt_conn_btree_open+0x140) [0x1a257c0] mongod(__wt_session_get_btree+0xE7) [0x1a9e357] mongod(__wt_session_get_btree+0x629) [0x1a9e899] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9e9bb] mongod(__wt_curfile_open+0x218) [0x1a33db8] mongod(+0x169B515) [0x1a9b515] mongod(__wt_metadata_cursor_open+0x5F) [0x1a660ef] mongod(__wt_metadata_cursor+0x7E) [0x1a661ee] mongod(wiredtiger_open+0x1541) [0x1a223a1] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x567) [0x1097517] mongod(+0xC93893) [0x1093893] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbad38] mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9970dd] mongod(main+0x15D) [0x99acbd] libc.so.6(__libc_start_main+0xFD) [0x393e21ed1d] mongod(+0x5937E9) [0x9937e9] ----- END BACKTRACE ----- 2016-09-22T14:43:30.875+0800 I CONTROL [main] ***** SERVER RESTARTED ***** 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] MongoDB starting : pid=1331 port=27017 dbpath=/usr/local/mongodb/data 64-bit host=iZ625ztxpiuZ 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] db version v3.2.7 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] allocator: tcmalloc 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] modules: none 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] build environment: 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] distmod: rhel62 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] distarch: x86_64 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] target_arch: x86_64 2016-09-22T14:43:30.914+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/mongodb.conf", net: { maxIncomingConnections: 20000, port: 27017 }, processManagement: { fork: true, pidFilePath: "/usr/local/mongodb/mongod.pid" }, storage: { dbPath: "/usr/local/mongodb/data", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/mongodb/logs/mongodb.log" } } 2016-09-22T14:43:30.914+0800 I NETWORK [initandlisten] --maxConns too high, can only handle 819 2016-09-22T14:43:30.916+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-09-22T14:43:30.951+0800 E STORAGE [initandlisten] WiredTiger (0) [1474526610:951169][1331:0x7f50003bbd40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1717798939 doesn't match expected checksum of 3759426581 2016-09-22T14:43:30.951+0800 E STORAGE [initandlisten] WiredTiger (0) [1474526610:951245][1331:0x7f50003bbd40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2016-09-22T14:43:30.951+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1474526610:951258][1331:0x7f50003bbd40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-09-22T14:43:30.951+0800 I - [initandlisten] Fatal Assertion 28558 2016-09-22T14:43:30.951+0800 I - [initandlisten] ***aborting after fassert() failure 2016-09-22T14:43:30.984+0800 F - [initandlisten] Got signal: 6 (Aborted). 0x1337cb2 0x1336be9 0x13373f2 0x393e60f7e0 0x393e2325e5 0x393e233dc5 0x12bea52 0x10af8d3 0x1a9f62c 0x1a9faed 0x1a9fed4 0x19d18fc 0x19d1e73 0x19cee15 0x19d2eb9 0x19ef92c 0x1a257c0 0x1a9e357 0x1a9e899 0x1a9e9bb 0x1a33db8 0x1a9b515 0x1a660ef 0x1a661ee 0x1a223a1 0x1097517 0x1093893 0xfbad38 0x9970dd 0x99acbd 0x393e21ed1d 0x9937e9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F37CB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F36BE9"},{"b":"400000","o":"F373F2"},{"b":"393E600000","o":"F7E0"},{"b":"393E200000","o":"325E5","s":"gsignal"},{"b":"393E200000","o":"33DC5","s":"abort"},{"b":"400000","o":"EBEA52","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF8D3"},{"b":"400000","o":"169F62C","s":"__wt_eventv"},{"b":"400000","o":"169FAED","s":"__wt_err"},{"b":"400000","o":"169FED4","s":"__wt_panic"},{"b":"400000","o":"15D18FC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15D1E73","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15CEE15","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15D2EB9"},{"b":"400000","o":"15EF92C","s":"__wt_btree_open"},{"b":"400000","o":"16257C0","s":"__wt_conn_btree_open"},{"b":"400000","o":"169E357","s":"__wt_session_get_btree"},{"b":"400000","o":"169E899","s":"__wt_session_get_btree"},{"b":"400000","o":"169E9BB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1633DB8","s":"__wt_curfile_open"},{"b":"400000","o":"169B515"},{"b":"400000","o":"16660EF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"16661EE","s":"__wt_metadata_cursor"},{"b":"400000","o":"16223A1","s":"wiredtiger_open"},{"b":"400000","o":"C97517","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C93893"},{"b":"400000","o":"BBAD38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5970DD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"59ACBD","s":"main"},{"b":"393E200000","o":"1ED1D","s":"__libc_start_main"},{"b":"400000","o":"5937E9"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-642.1.1.el6.x86_64", "version" : "#1 SMP Tue May 31 21:57:07 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D6BB40ACB6CFEAA43C3A651DDD8E547FCFE19166" }, { "b" : "7FFCFB3B1000", "elfType" : 3, "buildId" : "7F719AF91EE951B4FCB6647E7868F95F766A616B" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E83FD82B1AE56360B270189F636423FB3016594" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE4607C86FEEC14F47B46D0DCC" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "F14F3F5B55FC325FDFA7FB6F22F73CA1C6977379" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DB04BDF55FE66BADD1081D833C17B4881E0EEDB9" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EEE9AC218BFC460" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1337cb2] mongod(+0xF36BE9) [0x1336be9] mongod(+0xF373F2) [0x13373f2] libpthread.so.0(+0xF7E0) [0x393e60f7e0] libc.so.6(gsignal+0x35) [0x393e2325e5] libc.so.6(abort+0x175) [0x393e233dc5] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12bea52] mongod(+0xCAF8D3) [0x10af8d3] mongod(__wt_eventv+0x42C) [0x1a9f62c] mongod(__wt_err+0x8D) [0x1a9faed] mongod(__wt_panic+0x24) [0x1a9fed4] mongod(__wt_block_extlist_read+0x6C) [0x19d18fc] mongod(__wt_block_extlist_read_avail+0x33) [0x19d1e73] mongod(__wt_block_checkpoint_load+0x3C5) [0x19cee15] mongod(+0x15D2EB9) [0x19d2eb9] mongod(__wt_btree_open+0xC7C) [0x19ef92c] mongod(__wt_conn_btree_open+0x140) [0x1a257c0] mongod(__wt_session_get_btree+0xE7) [0x1a9e357] mongod(__wt_session_get_btree+0x629) [0x1a9e899] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9e9bb] mongod(__wt_curfile_open+0x218) [0x1a33db8] mongod(+0x169B515) [0x1a9b515] mongod(__wt_metadata_cursor_open+0x5F) [0x1a660ef] mongod(__wt_metadata_cursor+0x7E) [0x1a661ee] mongod(wiredtiger_open+0x1541) [0x1a223a1] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x567) [0x1097517] mongod(+0xC93893) [0x1093893] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbad38] mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9970dd] mongod(main+0x15D) [0x99acbd] libc.so.6(__libc_start_main+0xFD) [0x393e21ed1d] mongod(+0x5937E9) [0x9937e9] ----- END BACKTRACE -----