Major - P3 My server crashed and the admins didn't use dump commands to make backups, but copied the database files directly and I cannot repair the database using those files.
Is there a way to recover the data? At least partially/something?
mongod --repair --dbpath /data/db --storageEngine wiredTiger
2017-03-30T07:01:51.837+0000 I CONTROL [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db 64-bit host=a71c1ff14088
2017-03-30T07:01:51.837+0000 I CONTROL [initandlisten] db version v3.2.6
2017-03-30T07:01:51.837+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-03-30T07:01:51.837+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] modules: none
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] build environment:
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] distmod: debian71
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] distarch: x86_64
2017-03-30T07:01:51.838+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-03-30T07:01:51.839+0000 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/data/db", engine: "wiredTiger" } }
2017-03-30T07:01:51.845+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2017-03-30T07:01:51.845+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2017-03-30T07:01:51.845+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2017-03-30T07:01:51.845+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-30T07:01:51.860+0000 E STORAGE [initandlisten] WiredTiger (0) [1490857311:860361][7:0x7fbf9f7e0c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 36864: block header checksum of 1809637400 doesn't match expected checksum of 3653253408
2017-03-30T07:01:51.860+0000 E STORAGE [initandlisten] WiredTiger (0) [1490857311:860430][7:0x7fbf9f7e0c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-03-30T07:01:51.860+0000 E STORAGE [initandlisten] WiredTiger (-31804) [1490857311:860458][7:0x7fbf9f7e0c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-03-30T07:01:51.860+0000 I - [initandlisten] Fatal Assertion 28558
2017-03-30T07:01:51.860+0000 I - [initandlisten]
***aborting after fassert() failure
2017-03-30T07:01:51.878+0000 F - [initandlisten] Got signal: 6 (Aborted).
0x12ffdb2 0x12fecd9 0x12ff4e2 0x7fbf9e4c20a0 0x7fbf9e158125 0x7fbf9e15b3a0 0x12864c2 0x106f7d3 0x1a74cdc 0x1a7519d 0x1a75584 0x19a56ac 0x19a5c23 0x19a2bb5 0x19a6c69 0x19c3afc 0x19fae10 0x1a73a07 0x1a73f49 0x1a7406b 0x1a09378 0x1a70bc5 0x1a3b70f 0x1a3b80e 0x19f7401 0x10572af 0x1053723 0xf7a868 0x9485e6 0x94c1fd 0x7fbf9e144ead 0x944f29
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EFFDB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EFECD9"},{"b":"400000","o":"EFF4E2"},{"b":"7FBF9E4B3000","o":"F0A0"},{"b":"7FBF9E126000","o":"32125","s":"gsignal"},{"b":"7FBF9E126000","o":"353A0","s":"abort"},{"b":"400000","o":"E864C2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C6F7D3"},{"b":"400000","o":"1674CDC","s":"__wt_eventv"},{"b":"400000","o":"167519D","s":"__wt_err"},{"b":"400000","o":"1675584","s":"__wt_panic"},{"b":"400000","o":"15A56AC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15A5C23","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15A2BB5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15A6C69"},{"b":"400000","o":"15C3AFC","s":"__wt_btree_open"},{"b":"400000","o":"15FAE10","s":"__wt_conn_btree_open"},{"b":"400000","o":"1673A07","s":"__wt_session_get_btree"},{"b":"400000","o":"1673F49","s":"__wt_session_get_btree"},{"b":"400000","o":"167406B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1609378","s":"__wt_curfile_open"},{"b":"400000","o":"1670BC5"},{"b":"400000","o":"163B70F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"163B80E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15F7401","s":"wiredtiger_open"},{"b":"400000","o":"C572AF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C53723"},{"b":"400000","o":"B7A868","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5485E6","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54C1FD","s":"main"},{"b":"7FBF9E126000","o":"1EEAD","s":"__libc_start_main"},{"b":"400000","o":"544F29"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5F7AC844520FF83AD50A0F7191BF780A5CB1FE5F" }, { "b" : "7FFFA98B0000", "elfType" : 3, "buildId" : "1F644FA66B1BCBBE1359CEF9A63CFD4C8F0C6011" }, { "b" : "7FBF9F36C000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7FBF9EF73000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "079D784F9F4A88B6D87FC3E6B0B07BFBE209540E" }, { "b" : "7FBF9ED6B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "77CB0070BB75470C626B899800FFCF8C8184E32A" }, { "b" : "7FBF9EB67000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "32CCD52B7228B03E969C21587F14C2322D4A9544" }, { "b" : "7FBF9E8E5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "98FAFAB6E776EA40FBFFFBCB22859A8D54028EC0" }, { "b" : "7FBF9E6CF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7FBF9E4B3000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "F260CD217A3C0D89FD80BCC80D486C1C2AD32DCA" }, { "b" : "7FBF9E126000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "BD7458ADC60E906DBAABB72A7DA5015D5BC8269F" }, { "b" : "7FBF9F5CB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B66F918D37C1E4122FC91C74210F2112A6A5148E" }, { "b" : "7FBF9DF0F000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12ffdb2]
mongod(+0xEFECD9) [0x12fecd9]
mongod(+0xEFF4E2) [0x12ff4e2]
libpthread.so.0(+0xF0A0) [0x7fbf9e4c20a0]
libc.so.6(gsignal+0x35) [0x7fbf9e158125]
libc.so.6(abort+0x180) [0x7fbf9e15b3a0]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12864c2]
mongod(+0xC6F7D3) [0x106f7d3]
mongod(__wt_eventv+0x42C) [0x1a74cdc]
mongod(__wt_err+0x8D) [0x1a7519d]
mongod(__wt_panic+0x24) [0x1a75584]
mongod(__wt_block_extlist_read+0x6C) [0x19a56ac]
mongod(__wt_block_extlist_read_avail+0x33) [0x19a5c23]
mongod(__wt_block_checkpoint_load+0x3C5) [0x19a2bb5]
mongod(+0x15A6C69) [0x19a6c69]
mongod(__wt_btree_open+0xC7C) [0x19c3afc]
mongod(__wt_conn_btree_open+0x140) [0x19fae10]
mongod(__wt_session_get_btree+0xE7) [0x1a73a07]
mongod(__wt_session_get_btree+0x629) [0x1a73f49]
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7406b]
mongod(__wt_curfile_open+0x218) [0x1a09378]
mongod(+0x1670BC5) [0x1a70bc5]
mongod(__wt_metadata_cursor_open+0x5F) [0x1a3b70f]
mongod(__wt_metadata_cursor+0x7E) [0x1a3b80e]
mongod(wiredtiger_open+0x1541) [0x19f7401]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x10572af]
mongod(+0xC53723) [0x1053723]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf7a868]
mongod(_ZN5mongo13initAndListenEi+0x376) [0x9485e6]
mongod(main+0x15D) [0x94c1fd]
libc.so.6(__libc_start_main+0xFD) [0x7fbf9e144ead]
mongod(+0x544F29) [0x944f29]
----- END BACKTRACE -----
Aborted (core dumped)
Backup files:
-rw-r--r-- 1 mongodb mongodb 46 Aug 19 2016 WiredTiger -rw-r--r-- 1 mongodb mongodb 21 Aug 19 2016 WiredTiger.lock -rw-r--r-- 1 mongodb mongodb 944 Mar 25 02:10 WiredTiger.turtle -rw-r--r-- 1 mongodb mongodb 278528 Mar 25 02:00 WiredTiger.wt -rw-r--r-- 1 mongodb mongodb 4096 Mar 15 13:39 WiredTigerLAS.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:39 _mdb_catalog.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:40 collection-0--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 24 15:45 collection-0-7932716027032970764.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 15 13:40 collection-11--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 15 13:40 collection-13--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 1007616 Mar 24 18:53 collection-15--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 2322432 Mar 25 01:50 collection-17--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 184320 Mar 25 02:00 collection-19--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 15 13:39 collection-2--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 1028096 Mar 24 14:29 collection-21--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 9778634752 Mar 24 14:29 collection-23--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 118784 Mar 24 14:29 collection-25--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:39 collection-27--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 90112 Mar 24 14:40 collection-29--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 07:05 collection-31--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 11:49 collection-33--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 11:49 collection-35--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 65536 Mar 24 15:45 collection-37--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 159744 Mar 24 15:45 collection-39--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 15 13:39 collection-4--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 16 15:04 collection-41--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 532480 Mar 24 18:53 collection-43--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 16 15:05 collection-45--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 16 13:35 collection-47--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 17 10:25 collection-49--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 917504 Mar 24 18:53 collection-7--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:40 collection-9--6859887277780056529.wt drwxr-xr-x 2 mongodb mongodb 4096 Mar 25 02:00 diagnostic.data -rw-r--r-- 1 mongodb mongodb 167936 Mar 24 14:29 index-0-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:40 index-1--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 409600 Mar 24 14:29 index-1-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 24 15:45 index-1-7932716027032970764.wt -rw-r--r-- 1 mongodb mongodb 36864 Jan 3 15:55 index-10--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 22 10:49 index-10-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 23 02:15 index-11-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 16 08:48 index-12--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 06:43 index-12-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 22 02:16 index-13-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Jan 3 15:55 index-14--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 118784 Mar 22 02:16 index-14-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 22 02:16 index-15-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 65536 Mar 24 14:29 index-16--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-16-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 86016 Mar 24 14:29 index-17-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 57344 Mar 24 12:00 index-18--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 14:29 index-18-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 86016 Mar 24 14:29 index-19-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 86016 Mar 24 14:29 index-2-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 15 13:45 index-20--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 229376 Mar 24 14:29 index-20-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 106496 Mar 24 14:29 index-21-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 143360 Mar 24 14:29 index-22--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 106496 Mar 24 14:29 index-22-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 110592 Mar 24 14:29 index-23-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 167936 Mar 24 14:29 index-24--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 1839104 Mar 24 14:29 index-24-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 77824 Mar 24 14:29 index-25-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 65536 Mar 24 14:29 index-26--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 86016 Mar 24 14:29 index-26-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-27-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Jan 3 15:55 index-28--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 14:29 index-28-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-29-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 15 13:40 index-3--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 143360 Mar 24 14:29 index-3-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 14:40 index-30--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-30-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 81920 Mar 24 14:29 index-31-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 24 07:05 index-32--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 94208 Mar 24 14:29 index-32-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 364544 Mar 24 14:29 index-33-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 24 09:20 index-34--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-34-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-35-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 24 11:49 index-36--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-36-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 61440 Mar 24 14:44 index-37-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 15:45 index-38--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 14:29 index-38-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 774144 Mar 24 14:29 index-4-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 65536 Mar 24 15:45 index-40--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 16 14:33 index-42--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 270336 Mar 24 18:53 index-44--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 16 15:05 index-46--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 15 13:47 index-48--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 16384 Jan 3 15:55 index-5--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 512000 Mar 24 14:29 index-5-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 36864 Mar 17 10:25 index-50--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 16384 Mar 15 13:42 index-6--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 24 12:00 index-6-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 45056 Mar 24 12:00 index-7-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 22 02:16 index-8--6859887277780056529.wt -rw-r--r-- 1 mongodb mongodb 53248 Mar 22 02:16 index-8-6235169181121104973.wt -rw-r--r-- 1 mongodb mongodb 118784 Mar 22 02:16 index-9-6235169181121104973.wt drwxr-xr-x 2 mongodb mongodb 4096 Mar 23 12:53 journal -rw-r--r-- 1 mongodb mongodb 2 Mar 15 13:40 mongod.lock -rw-r--r-- 1 mongodb mongodb 36864 Mar 25 01:21 sizeStorer.wt -rw-r--r-- 1 mongodb mongodb 95 Aug 19 2016 storage.bson