Major - P3 A customer's server has stopped working after upgrading from 3.4.5 to 3.4.9.
Attempting to start the mongod process with the configuration file results in the following log:
2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] MongoDB starting : pid=3040 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] db version v3.4.9 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] modules: none 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] build environment: 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] distmod: ubuntu1604 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] distarch: x86_64 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-10-04T17:44:53.801+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2017-10-04T17:44:53.801+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2017-10-04T17:44:53.824+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-10-04T17:44:53.824+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2017-10-04T17:44:53.824+0000 I STORAGE [initandlisten] 2017-10-04T17:44:53.824+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine 2017-10-04T17:44:53.824+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem 2017-10-04T17:44:53.824+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-10-04T17:44:53.838+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507139093:838762][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933 2017-10-04T17:44:53.838+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507139093:838813][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value 2017-10-04T17:44:53.838+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1507139093:838822][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-10-04T17:44:53.838+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-10-04T17:44:53.838+0000 I - [initandlisten] ***aborting after fassert() failure 2017-10-04T17:44:53.857+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x5638a678e551 0x5638a678d769 0x5638a678dc4d 0x7f9544922390 0x7f954457c428 0x7f954457e02a 0x5638a5a3af93 0x5638a64a1e56 0x5638a5a4522c 0x5638a5a45451 0x5638a5a456a9 0x5638a7093e1f 0x5638a709436b 0x5638a709095d 0x5638a7095537 0x5638a70af12b 0x5638a70e8493 0x5638a7172d6b 0x5638a71732ad 0x5638a717356c 0x5638a70f6b91 0x5638a71690c8 0x5638a71861f0 0x5638a718673f 0x5638a711c5cc 0x5638a71874f2 0x5638a70ec4b7 0x5638a70e4726 0x5638a6486c2f 0x5638a647f4ac 0x5638a63727a0 0x5638a5a271be 0x5638a5a46a56 0x7f9544567830 0x5638a5aa5a59 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"5638A5220000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5638A5220000","o":"156D769"},{"b":"5638A5220000","o":"156DC4D"},{"b":"7F9544911000","o":"11390"},{"b":"7F9544547000","o":"35428","s":"gsignal"},{"b":"7F9544547000","o":"3702A","s":"abort"},{"b":"5638A5220000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5638A5220000","o":"1281E56"},{"b":"5638A5220000","o":"82522C","s":"__wt_eventv"},{"b":"5638A5220000","o":"825451","s":"__wt_err"},{"b":"5638A5220000","o":"8256A9","s":"__wt_panic"},{"b":"5638A5220000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"5638A5220000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"5638A5220000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"5638A5220000","o":"1E75537"},{"b":"5638A5220000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"5638A5220000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"5638A5220000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"5638A5220000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"5638A5220000","o":"1F490C8"},{"b":"5638A5220000","o":"1F661F0"},{"b":"5638A5220000","o":"1F6673F"},{"b":"5638A5220000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"5638A5220000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"5638A5220000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"5638A5220000","o":"1EC4726","s":"wiredtiger_open"},{"b":"5638A5220000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5638A5220000","o":"125F4AC"},{"b":"5638A5220000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5638A5220000","o":"8071BE"},{"b":"5638A5220000","o":"826A56","s":"main"},{"b":"7F9544547000","o":"20830","s":"__libc_start_main"},{"b":"5638A5220000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5638A5220000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFEAEF62000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F954589D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9545459000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9545251000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F954504D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9544D44000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9544B2E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9544911000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9544547000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9545B06000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5638a678e551] mongod(+0x156D769) [0x5638a678d769] mongod(+0x156DC4D) [0x5638a678dc4d] libpthread.so.0(+0x11390) [0x7f9544922390] libc.so.6(gsignal+0x38) [0x7f954457c428] libc.so.6(abort+0x16A) [0x7f954457e02a] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5638a5a3af93] mongod(+0x1281E56) [0x5638a64a1e56] mongod(__wt_eventv+0x3D7) [0x5638a5a4522c] mongod(__wt_err+0x9D) [0x5638a5a45451] mongod(__wt_panic+0x24) [0x5638a5a456a9] mongod(__wt_block_extlist_read+0x8F) [0x5638a7093e1f] mongod(__wt_block_extlist_read_avail+0x2B) [0x5638a709436b] mongod(__wt_block_checkpoint_load+0x26D) [0x5638a709095d] mongod(+0x1E75537) [0x5638a7095537] mongod(__wt_btree_open+0xB3B) [0x5638a70af12b] mongod(__wt_conn_btree_open+0x163) [0x5638a70e8493] mongod(__wt_session_get_btree+0xFB) [0x5638a7172d6b] mongod(__wt_session_get_btree+0x63D) [0x5638a71732ad] mongod(__wt_session_get_btree_ckpt+0x14C) [0x5638a717356c] mongod(__wt_curfile_open+0x161) [0x5638a70f6b91] mongod(+0x1F490C8) [0x5638a71690c8] mongod(+0x1F661F0) [0x5638a71861f0] mongod(+0x1F6673F) [0x5638a718673f] mongod(__wt_log_scan+0xC7C) [0x5638a711c5cc] mongod(__wt_txn_recover+0x572) [0x5638a71874f2] mongod(__wt_connection_workers+0x37) [0x5638a70ec4b7] mongod(wiredtiger_open+0x1666) [0x5638a70e4726] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5638a6486c2f] mongod(+0x125F4AC) [0x5638a647f4ac] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5638a63727a0] mongod(+0x8071BE) [0x5638a5a271be] mongod(main+0x966) [0x5638a5a46a56] libc.so.6(__libc_start_main+0xF0) [0x7f9544567830] mongod(_start+0x29) [0x5638a5aa5a59] ----- END BACKTRACE -----
Files in the directory:
-rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-0-1422455344354141630.wt -rw-r--r-- 1 mongodb mongodb 36864 Aug 30 14:47 collection-0-2199009336346498560.wt -rw-r--r-- 1 mongodb mongodb 61440 Oct 4 17:44 collection-0--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-10--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 18 20:29 collection-12--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 18 21:41 collection-14--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 14:46 collection-16--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-21--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-2-2199009336346498560.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-23--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 757760 Sep 18 21:41 collection-2--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 12 18:11 collection-25--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 12288 Sep 14 18:30 collection-27--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 collection-29--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-31--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 14:46 collection-33--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-35--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-37--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-39--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-41--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-43--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 53248 Sep 18 10:28 collection-4--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-45--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-47--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-49--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 collection-51--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 45056 Aug 30 14:46 collection-6--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 33275269120 Sep 18 21:41 collection-8--3433020917902207860.wt drwxr-xr-x 2 mongodb mongodb 4096 Sep 18 21:41 diagnostic.data/ -rw-r--r-- 1 mongodb mongodb 51812122624 Sep 18 21:41 index-0-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 243556352 Sep 18 21:41 index-1-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 14:45 index-11--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 32768 Jul 3 20:41 index-1-1422455344354141630.wt -rw-r--r-- 1 mongodb mongodb 36864 Aug 30 14:47 index-1-2199009336346498560.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 18 20:29 index-13--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Aug 30 14:50 index-1--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 14:47 index-15--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Sep 1 19:18 index-17--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 208896 Sep 18 21:41 index-18--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 294912 Sep 18 21:41 index-19--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 126976 Sep 18 21:41 index-20--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 1536196608 Sep 18 21:41 index-2-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 14:45 index-22--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 15:43 index-24--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 1 19:20 index-26--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 12288 Sep 14 18:30 index-28--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Sep 1 07:27 index-30--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 714182656 Sep 18 21:41 index-3-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 32768 Aug 30 14:46 index-3-2199009336346498560.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-32--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 118784 Sep 18 21:41 index-3--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 16384 Aug 30 15:14 index-34--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Sep 1 19:18 index-36--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:45 index-38--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-40--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 714121216 Sep 18 21:41 index-4-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 32768 Jun 27 20:48 index-4-2199009336346498560.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-42--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-44--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-46--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-48--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-50--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 7794946048 Sep 18 21:41 index-5-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 4096 Jul 4 14:25 index-52--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 36864 Sep 18 10:28 index-5--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 243535872 Sep 18 21:41 index-6-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 1599356928 Sep 18 21:41 index-7-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 36864 Aug 30 14:50 index-7--3433020917902207860.wt -rw-r--r-- 1 mongodb mongodb 1590890496 Sep 18 21:41 index-8-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 707862528 Sep 18 21:41 index-9-1143364533558430943.wt -rw-r--r-- 1 mongodb mongodb 643907584 Sep 18 21:41 index-9--3433020917902207860.wt drwxr-xr-x 2 mongodb mongodb 4096 Oct 4 17:44 journal/ -rw-r--r-- 1 mongodb mongodb 36864 Aug 30 14:46 _mdb_catalog.wt -rw-r--r-- 1 mongodb mongodb 5 Aug 30 14:46 mongod.lock -rw-r--r-- 1 mongodb mongodb 45056 Sep 18 21:41 sizeStorer.wt -rw-r--r-- 1 mongodb mongodb 95 Jun 27 20:06 storage.bson -rw-r--r-- 1 mongodb mongodb 49 Jun 27 20:06 WiredTiger -rw-r--r-- 1 mongodb mongodb 4096 Aug 30 14:46 WiredTigerLAS.wt -rw-r--r-- 1 mongodb mongodb 21 Jun 27 20:06 WiredTiger.lock -rw-r--r-- 1 mongodb mongodb 1017 Sep 18 21:41 WiredTiger.turtle -rw-r--r-- 1 mongodb mongodb 241664 Oct 4 17:44 WiredTiger.wt
Any help would be appreciated to get the database back and running. Thanks!