|
Customer has reverted to an old backup (without my request) - the issue persists though.
Here is the latest error (which is the same). Apologies for the change. Please note that the db version that now appears is v3.4.7.
This previous comments / ticket should no longer be taken into account, as the below is the only active instance of the server
I do believe they are the same, though.
2017-10-04T21:31:04.340+0000 I CONTROL [main] ***** SERVER RESTARTED *****
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] MongoDB starting : pid=1198 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] db version v3.4.7
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] allocator: tcmalloc
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] modules: none
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] build environment:
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] distmod: ubuntu1604
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] distarch: x86_64
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] target_arch: x86_64
|
2017-10-04T21:31:04.350+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
|
2017-10-04T21:31:04.350+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
|
2017-10-04T21:31:04.374+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2017-10-04T21:31:04.375+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
|
2017-10-04T21:31:04.375+0000 I STORAGE [initandlisten]
|
2017-10-04T21:31:04.375+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2017-10-04T21:31:04.375+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
|
2017-10-04T21:31:04.375+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31702M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2017-10-04T21:31:04.415+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507152664:415053][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
|
2017-10-04T21:31:04.415+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507152664:415101][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
|
2017-10-04T21:31:04.415+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1507152664:415117][1198:0x7f02b571bd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2017-10-04T21:31:04.415+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
|
2017-10-04T21:31:04.415+0000 I - [initandlisten]
|
***aborting after fassert() failure
|
|
|
2017-10-04T21:31:04.440+0000 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x5605ad90dea1 0x5605ad90d0b9 0x5605ad90d59d 0x7f02b4322390 0x7f02b3f7c428 0x7f02b3f7e02a 0x5605acbbc733 0x5605ad621c26 0x5605acbc69cc 0x5605acbc6bf1 0x5605acbc6e49 0x5605ae21376f 0x5605ae213cbb 0x5605ae2102ad 0x5605ae214e87 0x5605ae22ea7b 0x5605ae267df3 0x5605ae2f265b 0x5605ae2f2b9d 0x5605ae2f2e5c 0x5605ae2764f1 0x5605ae2e8a08 0x5605ae305a40 0x5605ae305f8f 0x5605ae29bfac 0x5605ae306d42 0x5605ae26be17 0x5605ae264076 0x5605ad606a6f 0x5605ad5ff2ec 0x5605ad4f2a00 0x5605acba895e 0x5605acbc81f6 0x7f02b3f67830 0x5605acc26e09
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"5605AC3A2000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5605AC3A2000","o":"156B0B9"},{"b":"5605AC3A2000","o":"156B59D"},{"b":"7F02B4311000","o":"11390"},{"b":"7F02B3F47000","o":"35428","s":"gsignal"},{"b":"7F02B3F47000","o":"3702A","s":"abort"},{"b":"5605AC3A2000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5605AC3A2000","o":"127FC26"},{"b":"5605AC3A2000","o":"8249CC","s":"__wt_eventv"},{"b":"5605AC3A2000","o":"824BF1","s":"__wt_err"},{"b":"5605AC3A2000","o":"824E49","s":"__wt_panic"},{"b":"5605AC3A2000","o":"1E7176F","s":"__wt_block_extlist_read"},{"b":"5605AC3A2000","o":"1E71CBB","s":"__wt_block_extlist_read_avail"},{"b":"5605AC3A2000","o":"1E6E2AD","s":"__wt_block_checkpoint_load"},{"b":"5605AC3A2000","o":"1E72E87"},{"b":"5605AC3A2000","o":"1E8CA7B","s":"__wt_btree_open"},{"b":"5605AC3A2000","o":"1EC5DF3","s":"__wt_conn_btree_open"},{"b":"5605AC3A2000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"5605AC3A2000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"5605AC3A2000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"5605AC3A2000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"5605AC3A2000","o":"1F46A08"},{"b":"5605AC3A2000","o":"1F63A40"},{"b":"5605AC3A2000","o":"1F63F8F"},{"b":"5605AC3A2000","o":"1EF9FAC","s":"__wt_log_scan"},{"b":"5605AC3A2000","o":"1F64D42","s":"__wt_txn_recover"},{"b":"5605AC3A2000","o":"1EC9E17","s":"__wt_connection_workers"},{"b":"5605AC3A2000","o":"1EC2076","s":"wiredtiger_open"},{"b":"5605AC3A2000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5605AC3A2000","o":"125D2EC"},{"b":"5605AC3A2000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5605AC3A2000","o":"80695E"},{"b":"5605AC3A2000","o":"8261F6","s":"main"},{"b":"7F02B3F47000","o":"20830","s":"__libc_start_main"},{"b":"5605AC3A2000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5605AC3A2000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFF020F8000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F02B529D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F02B4E59000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F02B4C51000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F02B4A4D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F02B4744000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F02B452E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F02B4311000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F02B3F47000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F02B5506000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5605ad90dea1]
|
mongod(+0x156B0B9) [0x5605ad90d0b9]
|
mongod(+0x156B59D) [0x5605ad90d59d]
|
libpthread.so.0(+0x11390) [0x7f02b4322390]
|
libc.so.6(gsignal+0x38) [0x7f02b3f7c428]
|
libc.so.6(abort+0x16A) [0x7f02b3f7e02a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5605acbbc733]
|
mongod(+0x127FC26) [0x5605ad621c26]
|
mongod(__wt_eventv+0x3D7) [0x5605acbc69cc]
|
mongod(__wt_err+0x9D) [0x5605acbc6bf1]
|
mongod(__wt_panic+0x24) [0x5605acbc6e49]
|
mongod(__wt_block_extlist_read+0x8F) [0x5605ae21376f]
|
mongod(__wt_block_extlist_read_avail+0x2B) [0x5605ae213cbb]
|
mongod(__wt_block_checkpoint_load+0x26D) [0x5605ae2102ad]
|
mongod(+0x1E72E87) [0x5605ae214e87]
|
mongod(__wt_btree_open+0xB3B) [0x5605ae22ea7b]
|
mongod(__wt_conn_btree_open+0x163) [0x5605ae267df3]
|
mongod(__wt_session_get_btree+0xFB) [0x5605ae2f265b]
|
mongod(__wt_session_get_btree+0x63D) [0x5605ae2f2b9d]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x5605ae2f2e5c]
|
mongod(__wt_curfile_open+0x161) [0x5605ae2764f1]
|
mongod(+0x1F46A08) [0x5605ae2e8a08]
|
mongod(+0x1F63A40) [0x5605ae305a40]
|
mongod(+0x1F63F8F) [0x5605ae305f8f]
|
mongod(__wt_log_scan+0xC7C) [0x5605ae29bfac]
|
mongod(__wt_txn_recover+0x572) [0x5605ae306d42]
|
mongod(__wt_connection_workers+0x37) [0x5605ae26be17]
|
mongod(wiredtiger_open+0x1666) [0x5605ae264076]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5605ad606a6f]
|
mongod(+0x125D2EC) [0x5605ad5ff2ec]
|
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5605ad4f2a00]
|
mongod(+0x80695E) [0x5605acba895e]
|
mongod(main+0x966) [0x5605acbc81f6]
|
libc.so.6(__libc_start_main+0xF0) [0x7f02b3f67830]
|
mongod(_start+0x29) [0x5605acc26e09]
|
----- END BACKTRACE -----
|
|
|
|
I have copied the file to /var/lib/mongodb and extracted it. It created a subdirectoy called /data/.
I have tried running the mongoDB again like that, as well as move the contents of /var/lib/mongodb/data to /var/lib/mongodb.
The two attempts are as follows:
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] MongoDB starting : pid=3637 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] db version v3.4.9
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] allocator: tcmalloc
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] modules: none
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] build environment:
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] distmod: ubuntu1604
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] distarch: x86_64
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] target_arch: x86_64
|
2017-10-04T18:28:38.150+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
|
2017-10-04T18:28:38.150+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
|
2017-10-04T18:28:38.173+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2017-10-04T18:28:38.173+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
|
2017-10-04T18:28:38.173+0000 I STORAGE [initandlisten]
|
2017-10-04T18:28:38.173+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2017-10-04T18:28:38.173+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
|
2017-10-04T18:28:38.173+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2017-10-04T18:28:38.194+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507141718:194838][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
|
2017-10-04T18:28:38.194+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507141718:194892][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
|
2017-10-04T18:28:38.194+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1507141718:194901][3637:0x7f9ae795cd00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2017-10-04T18:28:38.194+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
|
2017-10-04T18:28:38.194+0000 I - [initandlisten]
|
|
***aborting after fassert() failure
|
|
|
2017-10-04T18:28:38.213+0000 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x55e6a873d551 0x55e6a873c769 0x55e6a873cc4d 0x7f9ae6563390 0x7f9ae61bd428 0x7f9ae61bf02a 0x55e6a79e9f93 0x55e6a8450e56 0x55e6a79f422c 0x55e6a79f4451 0x55e6a79f46a9 0x55e6a9042e1f 0x55e6a904336b 0x55e6a903f95d 0x55e6a9044537 0x55e6a905e12b 0x55e6a9097493 0x55e6a9121d6b 0x55e6a91222ad 0x55e6a912256c 0x55e6a90a5b91 0x55e6a91180c8 0x55e6a91351f0 0x55e6a913573f 0x55e6a90cb5cc 0x55e6a91364f2 0x55e6a909b4b7 0x55e6a9093726 0x55e6a8435c2f 0x55e6a842e4ac 0x55e6a83217a0 0x55e6a79d61be 0x55e6a79f5a56 0x7f9ae61a8830 0x55e6a7a54a59
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"55E6A71CF000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55E6A71CF000","o":"156D769"},{"b":"55E6A71CF000","o":"156DC4D"},{"b":"7F9AE6552000","o":"11390"},{"b":"7F9AE6188000","o":"35428","s":"gsignal"},{"b":"7F9AE6188000","o":"3702A","s":"abort"},{"b":"55E6A71CF000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55E6A71CF000","o":"1281E56"},{"b":"55E6A71CF000","o":"82522C","s":"__wt_eventv"},{"b":"55E6A71CF000","o":"825451","s":"__wt_err"},{"b":"55E6A71CF000","o":"8256A9","s":"__wt_panic"},{"b":"55E6A71CF000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"55E6A71CF000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"55E6A71CF000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"55E6A71CF000","o":"1E75537"},{"b":"55E6A71CF000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"55E6A71CF000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"55E6A71CF000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"55E6A71CF000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"55E6A71CF000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"55E6A71CF000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"55E6A71CF000","o":"1F490C8"},{"b":"55E6A71CF000","o":"1F661F0"},{"b":"55E6A71CF000","o":"1F6673F"},{"b":"55E6A71CF000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"55E6A71CF000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"55E6A71CF000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"55E6A71CF000","o":"1EC4726","s":"wiredtiger_open"},{"b":"55E6A71CF000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55E6A71CF000","o":"125F4AC"},{"b":"55E6A71CF000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55E6A71CF000","o":"8071BE"},{"b":"55E6A71CF000","o":"826A56","s":"main"},{"b":"7F9AE6188000","o":"20830","s":"__libc_start_main"},{"b":"55E6A71CF000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55E6A71CF000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFFC7E34000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F9AE74DE000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9AE709A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9AE6E92000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9AE6C8E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9AE6985000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9AE676F000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9AE6552000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9AE6188000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9AE7747000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55e6a873d551]
|
mongod(+0x156D769) [0x55e6a873c769]
|
mongod(+0x156DC4D) [0x55e6a873cc4d]
|
libpthread.so.0(+0x11390) [0x7f9ae6563390]
|
libc.so.6(gsignal+0x38) [0x7f9ae61bd428]
|
libc.so.6(abort+0x16A) [0x7f9ae61bf02a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55e6a79e9f93]
|
mongod(+0x1281E56) [0x55e6a8450e56]
|
mongod(__wt_eventv+0x3D7) [0x55e6a79f422c]
|
mongod(__wt_err+0x9D) [0x55e6a79f4451]
|
mongod(__wt_panic+0x24) [0x55e6a79f46a9]
|
mongod(__wt_block_extlist_read+0x8F) [0x55e6a9042e1f]
|
mongod(__wt_block_extlist_read_avail+0x2B) [0x55e6a904336b]
|
mongod(__wt_block_checkpoint_load+0x26D) [0x55e6a903f95d]
|
mongod(+0x1E75537) [0x55e6a9044537]
|
mongod(__wt_btree_open+0xB3B) [0x55e6a905e12b]
|
mongod(__wt_conn_btree_open+0x163) [0x55e6a9097493]
|
mongod(__wt_session_get_btree+0xFB) [0x55e6a9121d6b]
|
mongod(__wt_session_get_btree+0x63D) [0x55e6a91222ad]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55e6a912256c]
|
mongod(__wt_curfile_open+0x161) [0x55e6a90a5b91]
|
mongod(+0x1F490C8) [0x55e6a91180c8]
|
mongod(+0x1F661F0) [0x55e6a91351f0]
|
mongod(+0x1F6673F) [0x55e6a913573f]
|
mongod(__wt_log_scan+0xC7C) [0x55e6a90cb5cc]
|
mongod(__wt_txn_recover+0x572) [0x55e6a91364f2]
|
mongod(__wt_connection_workers+0x37) [0x55e6a909b4b7]
|
mongod(wiredtiger_open+0x1666) [0x55e6a9093726]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55e6a8435c2f]
|
mongod(+0x125F4AC) [0x55e6a842e4ac]
|
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55e6a83217a0]
|
mongod(+0x8071BE) [0x55e6a79d61be]
|
mongod(main+0x966) [0x55e6a79f5a56]
|
libc.so.6(__libc_start_main+0xF0) [0x7f9ae61a8830]
|
mongod(_start+0x29) [0x55e6a7a54a59]
|
----- END BACKTRACE -----
|
2017-10-04T18:29:38.954+0000 I CONTROL [main] ***** SERVER RESTARTED *****
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] MongoDB starting : pid=3685 port=27017 dbpath=/var/lib/mongodb 64-bit host=r-forums
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] db version v3.4.9
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] allocator: tcmalloc
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] modules: none
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] build environment:
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] distmod: ubuntu1604
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] distarch: x86_64
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] target_arch: x86_64
|
2017-10-04T18:29:38.958+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
|
2017-10-04T18:29:38.958+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
|
2017-10-04T18:29:38.980+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2017-10-04T18:29:38.980+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
|
2017-10-04T18:29:38.980+0000 I STORAGE [initandlisten]
|
2017-10-04T18:29:38.980+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2017-10-04T18:29:38.980+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
|
2017-10-04T18:29:38.980+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31703M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2017-10-04T18:29:38.993+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507141778:993235][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: read checksum error for 32768B block at offset 33275236352: block header checksum of 859320671 doesn't match expected checksum of 2785962933
|
2017-10-04T18:29:38.993+0000 E STORAGE [initandlisten] WiredTiger error (0) [1507141778:993295][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: collection-8--3433020917902207860.wt: encountered an illegal file format or internal value
|
2017-10-04T18:29:38.993+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1507141778:993304][3685:0x7f9de9d23d00], file:collection-8--3433020917902207860.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2017-10-04T18:29:38.993+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
|
2017-10-04T18:29:38.993+0000 I - [initandlisten]
|
|
***aborting after fassert() failure
|
|
|
2017-10-04T18:29:39.011+0000 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x559be0db4551 0x559be0db3769 0x559be0db3c4d 0x7f9de892a390 0x7f9de8584428 0x7f9de858602a 0x559be0060f93 0x559be0ac7e56 0x559be006b22c 0x559be006b451 0x559be006b6a9 0x559be16b9e1f 0x559be16ba36b 0x559be16b695d 0x559be16bb537 0x559be16d512b 0x559be170e493 0x559be1798d6b 0x559be17992ad 0x559be179956c 0x559be171cb91 0x559be178f0c8 0x559be17ac1f0 0x559be17ac73f 0x559be17425cc 0x559be17ad4f2 0x559be17124b7 0x559be170a726 0x559be0aacc2f 0x559be0aa54ac 0x559be09987a0 0x559be004d1be 0x559be006ca56 0x7f9de856f830 0x559be00cba59
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"559BDF846000","o":"156E551","s":"_ZN5mongo15printStackTraceERSo"},{"b":"559BDF846000","o":"156D769"},{"b":"559BDF846000","o":"156DC4D"},{"b":"7F9DE8919000","o":"11390"},{"b":"7F9DE854F000","o":"35428","s":"gsignal"},{"b":"7F9DE854F000","o":"3702A","s":"abort"},{"b":"559BDF846000","o":"81AF93","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"559BDF846000","o":"1281E56"},{"b":"559BDF846000","o":"82522C","s":"__wt_eventv"},{"b":"559BDF846000","o":"825451","s":"__wt_err"},{"b":"559BDF846000","o":"8256A9","s":"__wt_panic"},{"b":"559BDF846000","o":"1E73E1F","s":"__wt_block_extlist_read"},{"b":"559BDF846000","o":"1E7436B","s":"__wt_block_extlist_read_avail"},{"b":"559BDF846000","o":"1E7095D","s":"__wt_block_checkpoint_load"},{"b":"559BDF846000","o":"1E75537"},{"b":"559BDF846000","o":"1E8F12B","s":"__wt_btree_open"},{"b":"559BDF846000","o":"1EC8493","s":"__wt_conn_btree_open"},{"b":"559BDF846000","o":"1F52D6B","s":"__wt_session_get_btree"},{"b":"559BDF846000","o":"1F532AD","s":"__wt_session_get_btree"},{"b":"559BDF846000","o":"1F5356C","s":"__wt_session_get_btree_ckpt"},{"b":"559BDF846000","o":"1ED6B91","s":"__wt_curfile_open"},{"b":"559BDF846000","o":"1F490C8"},{"b":"559BDF846000","o":"1F661F0"},{"b":"559BDF846000","o":"1F6673F"},{"b":"559BDF846000","o":"1EFC5CC","s":"__wt_log_scan"},{"b":"559BDF846000","o":"1F674F2","s":"__wt_txn_recover"},{"b":"559BDF846000","o":"1ECC4B7","s":"__wt_connection_workers"},{"b":"559BDF846000","o":"1EC4726","s":"wiredtiger_open"},{"b":"559BDF846000","o":"1266C2F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"559BDF846000","o":"125F4AC"},{"b":"559BDF846000","o":"11527A0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"559BDF846000","o":"8071BE"},{"b":"559BDF846000","o":"826A56","s":"main"},{"b":"7F9DE854F000","o":"20830","s":"__libc_start_main"},{"b":"559BDF846000","o":"885A59","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-96-generic", "version" : "#119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "559BDF846000", "elfType" : 3, "buildId" : "A97737C3A7656E2EFF91EE8950B4DE177415887F" }, { "b" : "7FFE30BFB000", "elfType" : 3, "buildId" : "0A63E8ED537104D9ABE07E9C9B77F8B49052BF37" }, { "b" : "7F9DE98A5000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F9DE9461000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F9DE9259000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9DE9055000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9DE8D4C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F9DE8B36000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F9DE8919000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F9DE854F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9DE9B0E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x559be0db4551]
|
mongod(+0x156D769) [0x559be0db3769]
|
mongod(+0x156DC4D) [0x559be0db3c4d]
|
libpthread.so.0(+0x11390) [0x7f9de892a390]
|
libc.so.6(gsignal+0x38) [0x7f9de8584428]
|
libc.so.6(abort+0x16A) [0x7f9de858602a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x559be0060f93]
|
mongod(+0x1281E56) [0x559be0ac7e56]
|
mongod(__wt_eventv+0x3D7) [0x559be006b22c]
|
mongod(__wt_err+0x9D) [0x559be006b451]
|
mongod(__wt_panic+0x24) [0x559be006b6a9]
|
mongod(__wt_block_extlist_read+0x8F) [0x559be16b9e1f]
|
mongod(__wt_block_extlist_read_avail+0x2B) [0x559be16ba36b]
|
mongod(__wt_block_checkpoint_load+0x26D) [0x559be16b695d]
|
mongod(+0x1E75537) [0x559be16bb537]
|
mongod(__wt_btree_open+0xB3B) [0x559be16d512b]
|
mongod(__wt_conn_btree_open+0x163) [0x559be170e493]
|
mongod(__wt_session_get_btree+0xFB) [0x559be1798d6b]
|
mongod(__wt_session_get_btree+0x63D) [0x559be17992ad]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x559be179956c]
|
mongod(__wt_curfile_open+0x161) [0x559be171cb91]
|
mongod(+0x1F490C8) [0x559be178f0c8]
|
mongod(+0x1F661F0) [0x559be17ac1f0]
|
mongod(+0x1F6673F) [0x559be17ac73f]
|
mongod(__wt_log_scan+0xC7C) [0x559be17425cc]
|
mongod(__wt_txn_recover+0x572) [0x559be17ad4f2]
|
mongod(__wt_connection_workers+0x37) [0x559be17124b7]
|
mongod(wiredtiger_open+0x1666) [0x559be170a726]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x559be0aacc2f]
|
mongod(+0x125F4AC) [0x559be0aa54ac]
|
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x559be09987a0]
|
mongod(+0x8071BE) [0x559be004d1be]
|
mongod(main+0x966) [0x559be006ca56]
|
libc.so.6(__libc_start_main+0xF0) [0x7f9de856f830]
|
mongod(_start+0x29) [0x559be00cba59]
|
----- END BACKTRACE -----
|
I will contact the customer to receive additional information for your questions.
I can already answer that the storage mechanism they are using is attached locally.
The database ran version 3.4.5 fine for months. The issue started due to an upgrade to 3.4.9. I have another customer who is beginning to report errors even though our product has not changed, around the same time.
Thanks
|