Major - P3 Mongo server was running fine for months and after a casual command line `reboot` of the linux box I began getting this error and unable to repair or salvage the database. I noticed others have had the same with in their cases they say "unusual shutdowns" and they uploaded their WiredTiger.wt - I've attached files. Thx in advance for any help how to resolve and recover my db!
admin@jenkins:/data/db$ sudo mongod
2016-12-07T18:38:56.511-0800 I CONTROL [initandlisten] MongoDB starting : pid=22869 port=27017 dbpath=/data/db 64-bit host=jenkins
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] db version v3.2.8
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] allocator: tcmalloc
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] modules: none
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] build environment:
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] distmod: ubuntu1404
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] distarch: x86_64
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] target_arch: x86_64
2016-12-07T18:38:56.512-0800 I CONTROL [initandlisten] options: {}
2016-12-07T18:38:56.542-0800 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-12-07T18:38:56.542-0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-12-07T18:38:56.568-0800 E STORAGE [initandlisten] WiredTiger (0) [1481164736:568070][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 3539301520 doesn't match expected checksum of 1790700336
2016-12-07T18:38:56.568-0800 E STORAGE [initandlisten] WiredTiger (0) [1481164736:568241][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-12-07T18:38:56.568-0800 E STORAGE [initandlisten] WiredTiger (-31804) [1481164736:568317][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-12-07T18:38:56.568-0800 I - [initandlisten] Fatal Assertion 28558
2016-12-07T18:38:56.568-0800 I - [initandlisten]
***aborting after fassert() failure
2016-12-07T18:38:56.592-0800 F - [initandlisten] Got signal: 6 (Aborted).
0x131ce72 0x131bfc9 0x131c7d2 0x7f760a23d340 0x7f7609e9ecc9 0x7f7609ea20d8 0x12a6772 0x10a0923 0x1a7e31c 0x1a7e7dd 0x1a7ebc4 0x19affac 0x19b0523 0x19ad4b5 0x19b1569 0x19cdffc 0x1a04270 0x1a7d047 0x1a7d589 0x1a7d6ab 0x1a12858 0x1a7a1d5 0x1a44ccf 0x1a44dce 0x1a00e11 0x10889cf 0x1084c93 0xfadc48 0x9b48fd 0x96e04d 0x7f7609e89ec5 0x9b1037
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1BFC9"},{"b":"400000","o":"F1C7D2"},{"b":"7F760A22D000","o":"10340"},{"b":"7F7609E68000","o":"36CC9","s":"gsignal"},{"b":"7F7609E68000","o":"3A0D8","s":"abort"},{"b":"400000","o":"EA6772","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0923"},{"b":"400000","o":"167E31C","s":"__wt_eventv"},{"b":"400000","o":"167E7DD","s":"__wt_err"},{"b":"400000","o":"167EBC4","s":"__wt_panic"},{"b":"400000","o":"15AFFAC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15B0523","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15AD4B5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15B1569"},{"b":"400000","o":"15CDFFC","s":"__wt_btree_open"},{"b":"400000","o":"1604270","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D047","s":"__wt_session_get_btree"},{"b":"400000","o":"167D589","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6AB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1612858","s":"__wt_curfile_open"},{"b":"400000","o":"167A1D5"},{"b":"400000","o":"1644CCF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1644DCE","s":"__wt_metadata_cursor"},{"b":"400000","o":"1600E11","s":"wiredtiger_open"},{"b":"400000","o":"C889CF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84C93"},{"b":"400000","o":"BADC48","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B48FD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7F7609E68000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-71-generic", "version" : "#114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFF3E09C000", "elfType" : 3, "buildId" : "C2B4C020D20EE489936A4B11AF2C695FD0ECD484" }, { "b" : "7F760B14E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F760AD73000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F760AB6B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F760A967000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F760A661000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F760A44B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F760A22D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7609E68000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F760B3AD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72]
mongod(+0xF1BFC9) [0x131bfc9]
mongod(+0xF1C7D2) [0x131c7d2]
libpthread.so.0(+0x10340) [0x7f760a23d340]
libc.so.6(gsignal+0x39) [0x7f7609e9ecc9]
libc.so.6(abort+0x148) [0x7f7609ea20d8]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a6772]
mongod(+0xCA0923) [0x10a0923]
mongod(__wt_eventv+0x42C) [0x1a7e31c]
mongod(__wt_err+0x8D) [0x1a7e7dd]
mongod(__wt_panic+0x24) [0x1a7ebc4]
mongod(__wt_block_extlist_read+0x6C) [0x19affac]
mongod(__wt_block_extlist_read_avail+0x33) [0x19b0523]
mongod(__wt_block_checkpoint_load+0x3C5) [0x19ad4b5]
mongod(+0x15B1569) [0x19b1569]
mongod(__wt_btree_open+0xC7C) [0x19cdffc]
mongod(__wt_conn_btree_open+0x140) [0x1a04270]
mongod(__wt_session_get_btree+0xE7) [0x1a7d047]
mongod(__wt_session_get_btree+0x629) [0x1a7d589]
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6ab]
mongod(__wt_curfile_open+0x218) [0x1a12858]
mongod(+0x167A1D5) [0x1a7a1d5]
mongod(__wt_metadata_cursor_open+0x5F) [0x1a44ccf]
mongod(__wt_metadata_cursor+0x7E) [0x1a44dce]
mongod(wiredtiger_open+0x1541) [0x1a00e11]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x10889cf]
mongod(+0xC84C93) [0x1084c93]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfadc48]
mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9b48fd]
mongod(main+0x15D) [0x96e04d]
libc.so.6(__libc_start_main+0xF5) [0x7f7609e89ec5]
mongod(+0x5B1037) [0x9b1037]
----- END BACKTRACE -----