[SERVER-31383] WiredTiger error:read checksum error Created: 04/Oct/17  Updated: 27/Jul/18  Resolved: 11/Oct/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Idan Aharoni [X] Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envm, rge, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-31383-repair.tar.gz     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt    
Operating System: Linux
Participants:

 Description   

A customer's server has stopped working after upgrading from 3.4.5 to 3.4.9.
Attempting to start the mongod process with the configuration file results in the following log:

2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] MongoDB starting : pid=3040 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] modules: none
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] build environment:
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-04T17:44:53.801+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-10-04T17:44:53.801+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-10-04T17:44:53.824+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-04T17:44:53.824+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten]
2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-04T17:44:53.824+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507139093:838762][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507139093:838813][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
2017-10-04T17:44:53.838+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1507139093:838822][3040:0x7f9545d1bd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-04T17:44:53.838+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-10-04T17:44:53.838+0000 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-10-04T17:44:53.857+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x5638a678e551 0x5638a678d769 0x5638a678dc4d 0x7f9544922390 0x7f954457c428 0x7f954457e02a 0x5638a5a3af93 0x5638a64a1e56 0x5638a5a4522c 0x5638a5a45451 0x5638a5a456a9 0x5638a7093e1f 0x5638a709436b 0x5638a709095d 0x5638a7095537 0x5638a70af12b 0x5638a70e8493 0x5638a7172d6b 0x5638a71732ad 0x5638a717356c 0x5638a70f6b91 0x5638a71690c8 0x5638a71861f0 0x5638a718673f 0x5638a711c5cc 0x5638a71874f2 0x5638a70ec4b7 0x5638a70e4726 0x5638a6486c2f 0x5638a647f4ac 0x5638a63727a0 0x5638a5a271be 0x5638a5a46a56 0x7f9544567830 0x5638a5aa5a59
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5638A5220000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5638A5220000","o":"156D769"},{"b":"5638A5220000","o":"156DC4D"},{"b":"7F9544911000","o":"11390"},{"b":"7F9544547000","o":"35428","s":"gsignal"},{"b":"7F9544547000","o":"3702A","s":"abort"},{"b":"5638A5220000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5638A5220000","o":"1281E56"},{"b":"5638A5220000","o":"82522C","s":"__wt_eventv"},{"b":"5638A5220000","o":"825451","s":"__wt_err"},{"b":"5638A5220000","o":"8256A9","s":"__wt_panic"},{"b":"5638A5220000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"5638A5220000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"5638A5220000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"5638A5220000","o":"1E75537"},{"b":"5638A5220000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"5638A5220000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"5638A5220000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"5638A5220000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"5638A5220000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"5638A5220000","o":"1F490C8"},{"b":"5638A5220000","o":"1F661F0"},{"b":"5638A5220000","o":"1F6673F"},{"b":"5638A5220000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"5638A5220000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"5638A5220000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"5638A5220000","o":"1EC4726","s":"wiredtiger_open"},{"b":"5638A5220000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5638A5220000","o":"125F4AC"},{"b":"5638A5220000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5638A5220000","o":"8071BE"},{"b":"5638A5220000","o":"826A56","s":"main"},{"b":"7F9544547000","o":"20830","s":"__libc_start_main"},{"b":"5638A5220000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5638A5220000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFEAEF62000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F954589D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9545459000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9545251000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F954504D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9544D44000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9544B2E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9544911000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9544547000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9545B06000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5638a678e551]
mongod(+0x156D769) [0x5638a678d769]
 mongod(+0x156DC4D) [0x5638a678dc4d]
 libpthread.so.0(+0x11390) [0x7f9544922390]
 libc.so.6(gsignal+0x38) [0x7f954457c428]
 libc.so.6(abort+0x16A) [0x7f954457e02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5638a5a3af93]
 mongod(+0x1281E56) [0x5638a64a1e56]
 mongod(__wt_eventv+0x3D7) [0x5638a5a4522c]
 mongod(__wt_err+0x9D) [0x5638a5a45451]
 mongod(__wt_panic+0x24) [0x5638a5a456a9]
 mongod(__wt_block_extlist_read+0x8F) [0x5638a7093e1f]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x5638a709436b]
 mongod(__wt_block_checkpoint_load+0x26D) [0x5638a709095d]
 mongod(+0x1E75537) [0x5638a7095537]
 mongod(__wt_btree_open+0xB3B) [0x5638a70af12b]
 mongod(__wt_conn_btree_open+0x163) [0x5638a70e8493]
 mongod(__wt_session_get_btree+0xFB) [0x5638a7172d6b]
 mongod(__wt_session_get_btree+0x63D) [0x5638a71732ad]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x5638a717356c]
 mongod(__wt_curfile_open+0x161) [0x5638a70f6b91]
 mongod(+0x1F490C8) [0x5638a71690c8]
 mongod(+0x1F661F0) [0x5638a71861f0]
 mongod(+0x1F6673F) [0x5638a718673f]
 mongod(__wt_log_scan+0xC7C) [0x5638a711c5cc]
 mongod(__wt_txn_recover+0x572) [0x5638a71874f2]
 mongod(__wt_connection_workers+0x37) [0x5638a70ec4b7]
 mongod(wiredtiger_open+0x1666) [0x5638a70e4726]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5638a6486c2f]
 mongod(+0x125F4AC) [0x5638a647f4ac]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5638a63727a0]
 mongod(+0x8071BE) [0x5638a5a271be]
 mongod(main+0x966) [0x5638a5a46a56]
 libc.so.6(__libc_start_main+0xF0) [0x7f9544567830]
 mongod(_start+0x29) [0x5638a5aa5a59]
-----  END BACKTRACE  -----

Files in the directory:

-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-0-1422455344354141630.wt
-rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:47 collection-0-2199009336346498560.wt
-rw-r--r--  1 mongodb mongodb       61440 Oct  4 17:44 collection-0--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-10--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep 18 20:29 collection-12--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep 18 21:41 collection-14--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:46 collection-16--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-21--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-2-2199009336346498560.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-23--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb      757760 Sep 18 21:41 collection-2--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep 12 18:11 collection-25--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       12288 Sep 14 18:30 collection-27--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 collection-29--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-31--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:46 collection-33--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-35--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-37--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-39--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-41--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-43--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       53248 Sep 18 10:28 collection-4--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-45--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-47--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-49--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 collection-51--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       45056 Aug 30 14:46 collection-6--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb 33275269120 Sep 18 21:41 collection-8--3433020917902207860.wt
drwxr-xr-x  2 mongodb mongodb        4096 Sep 18 21:41 diagnostic.data/
-rw-r--r--  1 mongodb mongodb 51812122624 Sep 18 21:41 index-0-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb   243556352 Sep 18 21:41 index-1-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:45 index-11--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       32768 Jul  3 20:41 index-1-1422455344354141630.wt
-rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:47 index-1-2199009336346498560.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep 18 20:29 index-13--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:50 index-1--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:47 index-15--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Sep  1 19:18 index-17--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb      208896 Sep 18 21:41 index-18--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb      294912 Sep 18 21:41 index-19--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb      126976 Sep 18 21:41 index-20--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb  1536196608 Sep 18 21:41 index-2-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 14:45 index-22--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 15:43 index-24--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep  1 19:20 index-26--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       12288 Sep 14 18:30 index-28--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Sep  1 07:27 index-30--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb   714182656 Sep 18 21:41 index-3-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb       32768 Aug 30 14:46 index-3-2199009336346498560.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-32--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb      118784 Sep 18 21:41 index-3--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       16384 Aug 30 15:14 index-34--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Sep  1 19:18 index-36--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:45 index-38--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-40--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb   714121216 Sep 18 21:41 index-4-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb       32768 Jun 27 20:48 index-4-2199009336346498560.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-42--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-44--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-46--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-48--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-50--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb  7794946048 Sep 18 21:41 index-5-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb        4096 Jul  4 14:25 index-52--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb       36864 Sep 18 10:28 index-5--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb   243535872 Sep 18 21:41 index-6-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb  1599356928 Sep 18 21:41 index-7-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:50 index-7--3433020917902207860.wt
-rw-r--r--  1 mongodb mongodb  1590890496 Sep 18 21:41 index-8-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb   707862528 Sep 18 21:41 index-9-1143364533558430943.wt
-rw-r--r--  1 mongodb mongodb   643907584 Sep 18 21:41 index-9--3433020917902207860.wt
drwxr-xr-x  2 mongodb mongodb        4096 Oct  4 17:44 journal/
-rw-r--r--  1 mongodb mongodb       36864 Aug 30 14:46 _mdb_catalog.wt
-rw-r--r--  1 mongodb mongodb           5 Aug 30 14:46 mongod.lock
-rw-r--r--  1 mongodb mongodb       45056 Sep 18 21:41 sizeStorer.wt
-rw-r--r--  1 mongodb mongodb          95 Jun 27 20:06 storage.bson
-rw-r--r--  1 mongodb mongodb          49 Jun 27 20:06 WiredTiger
-rw-r--r--  1 mongodb mongodb        4096 Aug 30 14:46 WiredTigerLAS.wt
-rw-r--r--  1 mongodb mongodb          21 Jun 27 20:06 WiredTiger.lock
-rw-r--r--  1 mongodb mongodb        1017 Sep 18 21:41 WiredTiger.turtle
-rw-r--r--  1 mongodb mongodb      241664 Oct  4 17:44 WiredTiger.wt

Any help would be appreciated to get the database back and running. Thanks!



 Comments   
Comment by Mark Agarunov [ 11/Oct/17 ]

Hello Aharoni,

Thanks for your response. I'm glad to hear that this issue has been resolved and everything is working again. To prevent this type of problem in the future, we recommend implementing regular backups and/or replication to mitigate any issues related to unreliable storage layers or server failures.

Thanks,
Mark

Comment by Idan Aharoni [X] [ 11/Oct/17 ]

Hi, issue has been resolved...

Comment by Idan Aharoni [X] [ 05/Oct/17 ]

I was able to recover the collection on a different instance of mongod. The question now is how do I remove the problematic collection from the database in /var/lib/mongodb/ without running mongod?
If I could do that, just remove the collection from the database and run mongod without it, I could use mongorestore to import it from the other instance and the problem would be fixed.
Trying to rename the original .wt file just results in an error that the file cannot be opened.
Any help would be appreciated.

Comment by Idan Aharoni [X] [ 04/Oct/17 ]

Customer has reverted to an old backup (without my request) - the issue persists though.
Here is the latest error (which is the same). Apologies for the change. Please note that the db version that now appears is v3.4.7.

This previous comments / ticket should no longer be taken into account, as the below is the only active instance of the server
I do believe they are the same, though.

2017-10-04T21:31:04.340+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1198 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] db version v3.4.7
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] modules: none
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] build environment:
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-04T21:31:04.350+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-10-04T21:31:04.350+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-10-04T21:31:04.374+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-04T21:31:04.375+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-10-04T21:31:04.375+0000 I STORAGE  [initandlisten]
2017-10-04T21:31:04.375+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-04T21:31:04.375+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-04T21:31:04.375+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31702M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-04T21:31:04.415+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507152664:415053][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
2017-10-04T21:31:04.415+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507152664:415101][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
2017-10-04T21:31:04.415+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1507152664:415117][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-04T21:31:04.415+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-10-04T21:31:04.415+0000 I -        [initandlisten]
***aborting after fassert() failure
 
 
2017-10-04T21:31:04.440+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x5605ad90dea1 0x5605ad90d0b9 0x5605ad90d59d 0x7f02b4322390 0x7f02b3f7c428 0x7f02b3f7e02a 0x5605acbbc733 0x5605ad621c26 0x5605acbc69cc 0x5605acbc6bf1 0x5605acbc6e49 0x5605ae21376f 0x5605ae213cbb 0x5605ae2102ad 0x5605ae214e87 0x5605ae22ea7b 0x5605ae267df3 0x5605ae2f265b 0x5605ae2f2b9d 0x5605ae2f2e5c 0x5605ae2764f1 0x5605ae2e8a08 0x5605ae305a40 0x5605ae305f8f 0x5605ae29bfac 0x5605ae306d42 0x5605ae26be17 0x5605ae264076 0x5605ad606a6f 0x5605ad5ff2ec 0x5605ad4f2a00 0x5605acba895e 0x5605acbc81f6 0x7f02b3f67830 0x5605acc26e09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5605AC3A2000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5605AC3A2000","o":"156B0B9"},{"b":"5605AC3A2000","o":"156B59D"},{"b":"7F02B4311000","o":"11390"},{"b":"7F02B3F47000","o":"35428","s":"gsignal"},{"b":"7F02B3F47000","o":"3702A","s":"abort"},{"b":"5605AC3A2000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5605AC3A2000","o":"127FC26"},{"b":"5605AC3A2000","o":"8249CC","s":"__wt_eventv"},{"b":"5605AC3A2000","o":"824BF1","s":"__wt_err"},{"b":"5605AC3A2000","o":"824E49","s":"__wt_panic"},{"b":"5605AC3A2000","o":"1E7176F","s":"__wt_block_extlist_read"},{"b":"5605AC3A2000","o":"1E71CBB","s":"__wt_block_extlist_read_avail"},{"b":"5605AC3A2000","o":"1E6E2AD","s":"__wt_block_checkpoint_load"},{"b":"5605AC3A2000","o":"1E72E87"},{"b":"5605AC3A2000","o":"1E8CA7B","s":"__wt_btree_open"},{"b":"5605AC3A2000","o":"1EC5DF3","s":"__wt_conn_btree_open"},{"b":"5605AC3A2000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"5605AC3A2000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"5605AC3A2000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"5605AC3A2000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"5605AC3A2000","o":"1F46A08"},{"b":"5605AC3A2000","o":"1F63A40"},{"b":"5605AC3A2000","o":"1F63F8F"},{"b":"5605AC3A2000","o":"1EF9FAC","s":"__wt_log_scan"},{"b":"5605AC3A2000","o":"1F64D42","s":"__wt_txn_recover"},{"b":"5605AC3A2000","o":"1EC9E17","s":"__wt_connection_workers"},{"b":"5605AC3A2000","o":"1EC2076","s":"wiredtiger_open"},{"b":"5605AC3A2000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5605AC3A2000","o":"125D2EC"},{"b":"5605AC3A2000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5605AC3A2000","o":"80695E"},{"b":"5605AC3A2000","o":"8261F6","s":"main"},{"b":"7F02B3F47000","o":"20830","s":"__libc_start_main"},{"b":"5605AC3A2000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5605AC3A2000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFF020F8000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F02B529D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F02B4E59000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F02B4C51000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F02B4A4D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F02B4744000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F02B452E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F02B4311000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F02B3F47000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F02B5506000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5605ad90dea1]
 mongod(+0x156B0B9) [0x5605ad90d0b9]
 mongod(+0x156B59D) [0x5605ad90d59d]
libpthread.so.0(+0x11390) [0x7f02b4322390]
 libc.so.6(gsignal+0x38) [0x7f02b3f7c428]
 libc.so.6(abort+0x16A) [0x7f02b3f7e02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5605acbbc733]
 mongod(+0x127FC26) [0x5605ad621c26]
 mongod(__wt_eventv+0x3D7) [0x5605acbc69cc]
 mongod(__wt_err+0x9D) [0x5605acbc6bf1]
 mongod(__wt_panic+0x24) [0x5605acbc6e49]
 mongod(__wt_block_extlist_read+0x8F) [0x5605ae21376f]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x5605ae213cbb]
 mongod(__wt_block_checkpoint_load+0x26D) [0x5605ae2102ad]
 mongod(+0x1E72E87) [0x5605ae214e87]
 mongod(__wt_btree_open+0xB3B) [0x5605ae22ea7b]
 mongod(__wt_conn_btree_open+0x163) [0x5605ae267df3]
 mongod(__wt_session_get_btree+0xFB) [0x5605ae2f265b]
 mongod(__wt_session_get_btree+0x63D) [0x5605ae2f2b9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x5605ae2f2e5c]
 mongod(__wt_curfile_open+0x161) [0x5605ae2764f1]
 mongod(+0x1F46A08) [0x5605ae2e8a08]
 mongod(+0x1F63A40) [0x5605ae305a40]
 mongod(+0x1F63F8F) [0x5605ae305f8f]
 mongod(__wt_log_scan+0xC7C) [0x5605ae29bfac]
 mongod(__wt_txn_recover+0x572) [0x5605ae306d42]
 mongod(__wt_connection_workers+0x37) [0x5605ae26be17]
 mongod(wiredtiger_open+0x1666) [0x5605ae264076]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5605ad606a6f]
 mongod(+0x125D2EC) [0x5605ad5ff2ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5605ad4f2a00]
 mongod(+0x80695E) [0x5605acba895e]
 mongod(main+0x966) [0x5605acbc81f6]
 libc.so.6(__libc_start_main+0xF0) [0x7f02b3f67830]
 mongod(_start+0x29) [0x5605acc26e09]
-----  END BACKTRACE  -----

Comment by Idan Aharoni [X] [ 04/Oct/17 ]

Answers to questions:

What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?

  • It's a SSD SAN
    Would you please check the integrity of your disks?
  • The disks are fine
    Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
    -I'm assuming that it was through the Ubuntu package manager's update process
    Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
    No
    Have you ever restored this instance from backups?
    No
    What method do you use to create backups?
    I have a clone of the VM
    When was the underlying filesystem last checked and is it currently marked clean?
    The filesystem is ext4 and it is clean
Comment by Idan Aharoni [X] [ 04/Oct/17 ]

I have copied the file to /var/lib/mongodb and extracted it. It created a subdirectoy called /data/.
I have tried running the mongoDB again like that, as well as move the contents of /var/lib/mongodb/data to /var/lib/mongodb.

The two attempts are as follows:

2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] MongoDB starting : pid=3637 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] modules: none
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] build environment:
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-04T18:28:38.150+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-10-04T18:28:38.150+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-10-04T18:28:38.173+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-04T18:28:38.173+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-10-04T18:28:38.173+0000 I STORAGE  [initandlisten]
2017-10-04T18:28:38.173+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-04T18:28:38.173+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-04T18:28:38.173+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-04T18:28:38.194+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507141718:194838][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
2017-10-04T18:28:38.194+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507141718:194892][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
2017-10-04T18:28:38.194+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1507141718:194901][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-04T18:28:38.194+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-10-04T18:28:38.194+0000 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-10-04T18:28:38.213+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x55e6a873d551 0x55e6a873c769 0x55e6a873cc4d 0x7f9ae6563390 0x7f9ae61bd428 0x7f9ae61bf02a 0x55e6a79e9f93 0x55e6a8450e56 0x55e6a79f422c 0x55e6a79f4451 0x55e6a79f46a9 0x55e6a9042e1f 0x55e6a904336b 0x55e6a903f95d 0x55e6a9044537 0x55e6a905e12b 0x55e6a9097493 0x55e6a9121d6b 0x55e6a91222ad 0x55e6a912256c 0x55e6a90a5b91 0x55e6a91180c8 0x55e6a91351f0 0x55e6a913573f 0x55e6a90cb5cc 0x55e6a91364f2 0x55e6a909b4b7 0x55e6a9093726 0x55e6a8435c2f 0x55e6a842e4ac 0x55e6a83217a0 0x55e6a79d61be 0x55e6a79f5a56 0x7f9ae61a8830 0x55e6a7a54a59
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55E6A71CF000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55E6A71CF000","o":"156D769"},{"b":"55E6A71CF000","o":"156DC4D"},{"b":"7F9AE6552000","o":"11390"},{"b":"7F9AE6188000","o":"35428","s":"gsignal"},{"b":"7F9AE6188000","o":"3702A","s":"abort"},{"b":"55E6A71CF000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55E6A71CF000","o":"1281E56"},{"b":"55E6A71CF000","o":"82522C","s":"__wt_eventv"},{"b":"55E6A71CF000","o":"825451","s":"__wt_err"},{"b":"55E6A71CF000","o":"8256A9","s":"__wt_panic"},{"b":"55E6A71CF000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"55E6A71CF000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"55E6A71CF000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"55E6A71CF000","o":"1E75537"},{"b":"55E6A71CF000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"55E6A71CF000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"55E6A71CF000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"55E6A71CF000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"55E6A71CF000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"55E6A71CF000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"55E6A71CF000","o":"1F490C8"},{"b":"55E6A71CF000","o":"1F661F0"},{"b":"55E6A71CF000","o":"1F6673F"},{"b":"55E6A71CF000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"55E6A71CF000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"55E6A71CF000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"55E6A71CF000","o":"1EC4726","s":"wiredtiger_open"},{"b":"55E6A71CF000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55E6A71CF000","o":"125F4AC"},{"b":"55E6A71CF000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55E6A71CF000","o":"8071BE"},{"b":"55E6A71CF000","o":"826A56","s":"main"},{"b":"7F9AE6188000","o":"20830","s":"__libc_start_main"},{"b":"55E6A71CF000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55E6A71CF000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFFC7E34000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F9AE74DE000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9AE709A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9AE6E92000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9AE6C8E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9AE6985000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9AE676F000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9AE6552000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9AE6188000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9AE7747000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55e6a873d551]
 mongod(+0x156D769) [0x55e6a873c769]
 mongod(+0x156DC4D) [0x55e6a873cc4d]
 libpthread.so.0(+0x11390) [0x7f9ae6563390]
 libc.so.6(gsignal+0x38) [0x7f9ae61bd428]
 libc.so.6(abort+0x16A) [0x7f9ae61bf02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55e6a79e9f93]
 mongod(+0x1281E56) [0x55e6a8450e56]
 mongod(__wt_eventv+0x3D7) [0x55e6a79f422c]
 mongod(__wt_err+0x9D) [0x55e6a79f4451]
 mongod(__wt_panic+0x24) [0x55e6a79f46a9]
 mongod(__wt_block_extlist_read+0x8F) [0x55e6a9042e1f]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x55e6a904336b]
 mongod(__wt_block_checkpoint_load+0x26D) [0x55e6a903f95d]
 mongod(+0x1E75537) [0x55e6a9044537]
 mongod(__wt_btree_open+0xB3B) [0x55e6a905e12b]
 mongod(__wt_conn_btree_open+0x163) [0x55e6a9097493]
 mongod(__wt_session_get_btree+0xFB) [0x55e6a9121d6b]
 mongod(__wt_session_get_btree+0x63D) [0x55e6a91222ad]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55e6a912256c]
 mongod(__wt_curfile_open+0x161) [0x55e6a90a5b91]
 mongod(+0x1F490C8) [0x55e6a91180c8]
 mongod(+0x1F661F0) [0x55e6a91351f0]
 mongod(+0x1F6673F) [0x55e6a913573f]
 mongod(__wt_log_scan+0xC7C) [0x55e6a90cb5cc]
 mongod(__wt_txn_recover+0x572) [0x55e6a91364f2]
 mongod(__wt_connection_workers+0x37) [0x55e6a909b4b7]
 mongod(wiredtiger_open+0x1666) [0x55e6a9093726]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55e6a8435c2f]
 mongod(+0x125F4AC) [0x55e6a842e4ac]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55e6a83217a0]
 mongod(+0x8071BE) [0x55e6a79d61be]
 mongod(main+0x966) [0x55e6a79f5a56]
 libc.so.6(__libc_start_main+0xF0) [0x7f9ae61a8830]
 mongod(_start+0x29) [0x55e6a7a54a59]
-----  END BACKTRACE  -----
2017-10-04T18:29:38.954+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] MongoDB starting : pid=3685 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] modules: none
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] build environment:
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-04T18:29:38.958+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-10-04T18:29:38.958+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-10-04T18:29:38.980+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-04T18:29:38.980+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-10-04T18:29:38.980+0000 I STORAGE  [initandlisten]
2017-10-04T18:29:38.980+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-04T18:29:38.980+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-04T18:29:38.980+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-04T18:29:38.993+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507141778:993235][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
2017-10-04T18:29:38.993+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1507141778:993295][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
2017-10-04T18:29:38.993+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1507141778:993304][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-04T18:29:38.993+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-10-04T18:29:38.993+0000 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-10-04T18:29:39.011+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x559be0db4551 0x559be0db3769 0x559be0db3c4d 0x7f9de892a390 0x7f9de8584428 0x7f9de858602a 0x559be0060f93 0x559be0ac7e56 0x559be006b22c 0x559be006b451 0x559be006b6a9 0x559be16b9e1f 0x559be16ba36b 0x559be16b695d 0x559be16bb537 0x559be16d512b 0x559be170e493 0x559be1798d6b 0x559be17992ad 0x559be179956c 0x559be171cb91 0x559be178f0c8 0x559be17ac1f0 0x559be17ac73f 0x559be17425cc 0x559be17ad4f2 0x559be17124b7 0x559be170a726 0x559be0aacc2f 0x559be0aa54ac 0x559be09987a0 0x559be004d1be 0x559be006ca56 0x7f9de856f830 0x559be00cba59
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"559BDF846000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"559BDF846000","o":"156D769"},{"b":"559BDF846000","o":"156DC4D"},{"b":"7F9DE8919000","o":"11390"},{"b":"7F9DE854F000","o":"35428","s":"gsignal"},{"b":"7F9DE854F000","o":"3702A","s":"abort"},{"b":"559BDF846000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"559BDF846000","o":"1281E56"},{"b":"559BDF846000","o":"82522C","s":"__wt_eventv"},{"b":"559BDF846000","o":"825451","s":"__wt_err"},{"b":"559BDF846000","o":"8256A9","s":"__wt_panic"},{"b":"559BDF846000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"559BDF846000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"559BDF846000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"559BDF846000","o":"1E75537"},{"b":"559BDF846000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"559BDF846000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"559BDF846000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"559BDF846000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"559BDF846000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"559BDF846000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"559BDF846000","o":"1F490C8"},{"b":"559BDF846000","o":"1F661F0"},{"b":"559BDF846000","o":"1F6673F"},{"b":"559BDF846000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"559BDF846000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"559BDF846000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"559BDF846000","o":"1EC4726","s":"wiredtiger_open"},{"b":"559BDF846000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"559BDF846000","o":"125F4AC"},{"b":"559BDF846000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"559BDF846000","o":"8071BE"},{"b":"559BDF846000","o":"826A56","s":"main"},{"b":"7F9DE854F000","o":"20830","s":"__libc_start_main"},{"b":"559BDF846000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "559BDF846000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFE30BFB000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F9DE98A5000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9DE9461000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9DE9259000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9DE9055000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9DE8D4C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9DE8B36000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9DE8919000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9DE854F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9DE9B0E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x559be0db4551]
 mongod(+0x156D769) [0x559be0db3769]
 mongod(+0x156DC4D) [0x559be0db3c4d]
 libpthread.so.0(+0x11390) [0x7f9de892a390]
 libc.so.6(gsignal+0x38) [0x7f9de8584428]
 libc.so.6(abort+0x16A) [0x7f9de858602a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x559be0060f93]
 mongod(+0x1281E56) [0x559be0ac7e56]
 mongod(__wt_eventv+0x3D7) [0x559be006b22c]
 mongod(__wt_err+0x9D) [0x559be006b451]
 mongod(__wt_panic+0x24) [0x559be006b6a9]
 mongod(__wt_block_extlist_read+0x8F) [0x559be16b9e1f]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x559be16ba36b]
 mongod(__wt_block_checkpoint_load+0x26D) [0x559be16b695d]
 mongod(+0x1E75537) [0x559be16bb537]
 mongod(__wt_btree_open+0xB3B) [0x559be16d512b]
 mongod(__wt_conn_btree_open+0x163) [0x559be170e493]
 mongod(__wt_session_get_btree+0xFB) [0x559be1798d6b]
 mongod(__wt_session_get_btree+0x63D) [0x559be17992ad]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x559be179956c]
 mongod(__wt_curfile_open+0x161) [0x559be171cb91]
 mongod(+0x1F490C8) [0x559be178f0c8]
 mongod(+0x1F661F0) [0x559be17ac1f0]
 mongod(+0x1F6673F) [0x559be17ac73f]
 mongod(__wt_log_scan+0xC7C) [0x559be17425cc]
 mongod(__wt_txn_recover+0x572) [0x559be17ad4f2]
 mongod(__wt_connection_workers+0x37) [0x559be17124b7]
 mongod(wiredtiger_open+0x1666) [0x559be170a726]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x559be0aacc2f]
 mongod(+0x125F4AC) [0x559be0aa54ac]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x559be09987a0]
 mongod(+0x8071BE) [0x559be004d1be]
 mongod(main+0x966) [0x559be006ca56]
libc.so.6(__libc_start_main+0xF0) [0x7f9de856f830]
 mongod(_start+0x29) [0x559be00cba59]
-----  END BACKTRACE  -----

I will contact the customer to receive additional information for your questions.
I can already answer that the storage mechanism they are using is attached locally.
The database ran version 3.4.5 fine for months. The issue started due to an upgrade to 3.4.9. I have another customer who is beginning to report errors even though our product has not changed, around the same time.

Thanks

Comment by Mark Agarunov [ 04/Oct/17 ]

Hello Aharoni,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:26:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.