[SERVER-32780] DB directory snapshot from hardware failure Created: 18/Jan/18  Updated: 21/Mar/18  Resolved: 16/Feb/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Langston Assignee: Kelsey Schubert
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File _mdb_catalog.wt     File sizeStorer.wt     File storage.bson    
Issue Links:
Related
Operating System: ALL
Participants:
Linked BF Score: 0

 Description   

Hi There,

We have got snapshot mongodb directory from broken server hardware, when we are trying to start mongo

2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] MongoDB starting : pid=19431 port=27017 dbpath=/var/lib/mongodb 64-bit host=lang
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] db version v3.6.1
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] modules: none
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] build environment:
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-18T23:32:02.895+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2018-01-18T23:32:02.895+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-01-18T23:32:02.895+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-01-18T23:32:02.895+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-01-18T23:32:02.896+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-01-18T23:32:02.977+0300 I STORAGE  [initandlisten] WiredTiger message [1516307522:977147][19431:0x7f0765f38a80], txn-recover: Main recovery loop: starting at 59/128
2018-01-18T23:32:02.977+0300 I STORAGE  [initandlisten] WiredTiger message [1516307522:977419][19431:0x7f0765f38a80], txn-recover: Recovering log 59 through 60
2018-01-18T23:32:02.984+0300 I STORAGE  [initandlisten] WiredTiger message [1516307522:984322][19431:0x7f0765f38a80], txn-recover: Recovering log 60 through 60
2018-01-18T23:32:03.062+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1516307523:62572][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 28672: block header checksum of 0 doesn't match expected checksum of 1074498338
2018-01-18T23:32:03.062+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1516307523:62643][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: _mdb_catalog.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227)
2018-01-18T23:32:03.062+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1516307523:62669][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-01-18T23:32:03.062+0300 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-01-18T23:32:03.062+0300 F -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-01-18T23:32:03.090+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x557b6eaea8da 0x557b6eae9b9e 0x557b6eaea072 0x7f076325bdb0 0x7f0762ec7860 0x7f0762ec8ec9 0x557b6d41c8f7 0x557b6d4ed876 0x7f0765a0bc0e 0x7f0765a0be23 0x7f0765a0c177 0x7f0765a139d3 0x7f0765a13f23 0x7f0765a1035e 0x7f0765a15212 0x7f0765a31825 0x7f0765a774a9 0x7f0765b052ad 0x7f0765b057ef 0x7f0765b05aa0 0x7f0765a86ceb 0x7f0765b01d79 0x7f0765a9f5fe 0x7f0765b01c69 0x7f0765b021fa 0x557b6d4e71b6 0x557b6d4e6a4c 0x557b6d4dc588 0x557b6d4dc636 0x557b6d4e1ee1 0x557b6d4cce98 0x557b6d502160 0x557b6d4b7397 0x557b6d592bf3 0x557b6d494011 0x557b6d4975fa 0x557b6d41e449 0x7f0762eb3f4a 0x557b6d48141a
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"557B6CBC6000","o":"1F248DA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"557B6CBC6000","o":"1F23B9E"},{"b":"557B6CBC6000","o":"1F24072"},{"b":"7F076324A000","o":"11DB0"},{"b":"7F0762E93000","o":"34860","s":"gsignal"},{"b":"7F0762E93000","o":"35EC9","s":"abort"},{"b":"557B6CBC6000","o":"8568F7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"557B6CBC6000","o":"927876"},{"b":"7F07659EC000","o":"1FC0E","s":"__wt_eventv"},{"b":"7F07659EC000","o":"1FE23","s":"__wt_err"},{"b":"7F07659EC000","o":"20177","s":"__wt_panic"},{"b":"7F07659EC000","o":"279D3","s":"__wt_block_extlist_read"},{"b":"7F07659EC000","o":"27F23","s":"__wt_block_extlist_read_avail"},{"b":"7F07659EC000","o":"2435E","s":"__wt_block_checkpoint_load"},{"b":"7F07659EC000","o":"29212"},{"b":"7F07659EC000","o":"45825","s":"__wt_btree_open"},{"b":"7F07659EC000","o":"8B4A9","s":"__wt_conn_dhandle_open"},{"b":"7F07659EC000","o":"1192AD","s":"__wt_session_get_dhandle"},{"b":"7F07659EC000","o":"1197EF","s":"__wt_session_get_dhandle"},{"b":"7F07659EC000","o":"119AA0","s":"__wt_session_get_btree_ckpt"},{"b":"7F07659EC000","o":"9ACEB","s":"__wt_curfile_open"},{"b":"7F07659EC000","o":"115D79"},{"b":"7F07659EC000","o":"B35FE","s":"__wt_curtable_open"},{"b":"7F07659EC000","o":"115C69"},{"b":"7F07659EC000","o":"1161FA"},{"b":"557B6CBC6000","o":"9211B6","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"557B6CBC6000","o":"920A4C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"557B6CBC6000","o":"916588","s":"_ZN5mongo31WiredTigerRecordStoreCursorBaseC2EPNS_16OperationContextERKNS_21WiredTigerRecordStoreEb"},{"b":"557B6CBC6000","o":"916636","s":"_ZNK5mongo29StandardWiredTigerRecordStore9getCursorEPNS_16OperationContextEb"},{"b":"557B6CBC6000","o":"91BEE1","s":"_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE"},{"b":"557B6CBC6000","o":"906E98","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE"},{"b":"557B6CBC6000","o":"93C160","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE"},{"b":"557B6CBC6000","o":"8F1397"},{"b":"557B6CBC6000","o":"9CCBF3","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"557B6CBC6000","o":"8CE011"},{"b":"557B6CBC6000","o":"8D15FA","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"557B6CBC6000","o":"858449","s":"main"},{"b":"7F0762E93000","o":"20F4A","s":"__libc_start_main"},{"b":"557B6CBC6000","o":"8BB41A","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.6.1", "gitVersion" : "025d4f4fe61efd1fb6f0005be20cb45a004093d1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.13-1-ARCH", "version" : "#1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "557B6CBC6000", "elfType" : 3, "buildId" : "029D000C7CD46EBE64E61D75949D0CCA9AFE8548" }, { "b" : "7FFCB79F2000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "3832AB6026917A8D56323E7017DC052A5FC0F5C6" }, { "b" : "7F07659EC000", "path" : "/usr/lib/libwiredtiger-3.0.0.so", "elfType" : 3, "buildId" : "AF35154F05948BA92AEFEC42B9B37264437FAA8E" }, { "b" : "7F07657CF000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "FD2BC2C7EB41B83A66FF6D099037532027A47916" }, { "b" : "7F0765574000", "path" : "/usr/lib/libstemmer.so.0", "elfType" : 3, "buildId" : "AD50FFCB248B0279A46FA04967455B1EFD942AF0" }, { "b" : "7F076535D000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "9D65B31D958D72054843331A9D549B1D12ADA045" }, { "b" : "7F0765155000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "4945D214906C5C184B044DF00ED23A3A4DC08166" }, { "b" : "7F0764ED8000", "path" : "/usr/lib/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "4CA43F102CA009469006E0FA5A595D110E72A88A" }, { "b" : "7F0764C65000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "00A48F31C879241B1621FA0EE8A77E24333D006B" }, { "b" : "7F0764A5C000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "A97FFE1C498EB6B2E383ABADF091EAF02D7FCA18" }, { "b" : "7F0764845000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "C37EEAC29624BF0F52FA85E0C5035B0ABAEDC100" }, { "b" : "7F07645DB000", "path" : "/usr/lib/libssl.so.1.1", "elfType" : 3, "buildId" : "8B3582F677DE36D14D30BECE4A92F1CD19026299" }, { "b" : "7F076415E000", "path" : "/usr/lib/libcrypto.so.1.1", "elfType" : 3, "buildId" : "05A9A6C52AACA0043A1D4A046193015D972AC092" }, { "b" : "7F0763F56000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "D520E36F95020BC09653CBFA3E3533E2F823713C" }, { "b" : "7F0763D52000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "F4C88B82DD709ACA91EB652FFDBD91900B8A908D" }, { "b" : "7F07639CB000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "14EC457A4EF13B537BA15A26301F1512CE3277F5" }, { "b" : "7F076367F000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "FA35622098EC404C4787159F0AB09FB2F6C5466F" }, { "b" : "7F0763468000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC76DEC7E4673A3AF72A6B26BBB7916359B9802F" }, { "b" : "7F076324A000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "F2D89412406AED9373964B49A96BA88BAA53A0BC" }, { "b" : "7F0762E93000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "48CE2E5F2FF41F8C7911E882A033C8C222B7F209" }, { "b" : "7F0762A9E000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "475D699EF3D447B31EBBC2ED5B7EEE7702C85447" }, { "b" : "7F0762889000", "path" : "/usr/lib/liblz4.so.1", "elfType" : 3, "buildId" : "A274C0E4DBC85A4C859DDFFE638FBB9AE9A156AC" }, { "b" : "7F0765D61000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E372512D034C6ABD6C233F9CCE96597653F372D3" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x557b6eaea8da]
 mongod(+0x1F23B9E) [0x557b6eae9b9e]
 mongod(+0x1F24072) [0x557b6eaea072]
 libpthread.so.0(+0x11DB0) [0x7f076325bdb0]
 libc.so.6(gsignal+0x110) [0x7f0762ec7860]
 libc.so.6(abort+0x1C9) [0x7f0762ec8ec9]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x557b6d41c8f7]
 mongod(+0x927876) [0x557b6d4ed876]
 libwiredtiger-3.0.0.so(__wt_eventv+0x36D) [0x7f0765a0bc0e]
 libwiredtiger-3.0.0.so(__wt_err+0xA3) [0x7f0765a0be23]
 libwiredtiger-3.0.0.so(__wt_panic+0x35) [0x7f0765a0c177]
 libwiredtiger-3.0.0.so(__wt_block_extlist_read+0x93) [0x7f0765a139d3]
 libwiredtiger-3.0.0.so(__wt_block_extlist_read_avail+0x23) [0x7f0765a13f23]
 libwiredtiger-3.0.0.so(__wt_block_checkpoint_load+0x2CE) [0x7f0765a1035e]
 libwiredtiger-3.0.0.so(+0x29212) [0x7f0765a15212]
 libwiredtiger-3.0.0.so(__wt_btree_open+0xB15) [0x7f0765a31825]
 libwiredtiger-3.0.0.so(__wt_conn_dhandle_open+0x229) [0x7f0765a774a9]
 libwiredtiger-3.0.0.so(__wt_session_get_dhandle+0xDD) [0x7f0765b052ad]
 libwiredtiger-3.0.0.so(__wt_session_get_dhandle+0x61F) [0x7f0765b057ef]
 libwiredtiger-3.0.0.so(__wt_session_get_btree_ckpt+0x150) [0x7f0765b05aa0]
 libwiredtiger-3.0.0.so(__wt_curfile_open+0x4B) [0x7f0765a86ceb]
 libwiredtiger-3.0.0.so(+0x115D79) [0x7f0765b01d79]
 libwiredtiger-3.0.0.so(__wt_curtable_open+0xCE) [0x7f0765a9f5fe]
 libwiredtiger-3.0.0.so(+0x115C69) [0x7f0765b01c69]
 libwiredtiger-3.0.0.so(+0x1161FA) [0x7f0765b021fa]
 mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x76) [0x557b6d4e71b6]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x557b6d4e6a4c]
 mongod(_ZN5mongo31WiredTigerRecordStoreCursorBaseC2EPNS_16OperationContextERKNS_21WiredTigerRecordStoreEb+0x58) [0x557b6d4dc588]
 mongod(_ZNK5mongo29StandardWiredTigerRecordStore9getCursorEPNS_16OperationContextEb+0x86) [0x557b6d4dc636]
 mongod(_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE+0x41) [0x557b6d4e1ee1]
 mongod(_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE+0x2D8) [0x557b6d4cce98]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE+0x11D0) [0x557b6d502160]
 mongod(+0x8F1397) [0x557b6d4b7397]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x613) [0x557b6d592bf3]
 mongod(+0x8CE011) [0x557b6d494011]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x61A) [0x557b6d4975fa]
 mongod(main+0x9) [0x557b6d41e449]
 libc.so.6(__libc_start_main+0xEA) [0x7f0762eb3f4a]
 mongod(_start+0x2A) [0x557b6d48141a]
-----  END BACKTRACE  -----

Then repair attempt deleting all our data

mongod --repair --dbpath /var/lib/mongodb/
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] MongoDB starting : pid=19599 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=lang
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] db version v3.6.1
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] modules: none
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] build environment:
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-18T23:33:42.861+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
2018-01-18T23:33:42.861+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-01-18T23:33:42.862+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-01-18T23:33:42.862+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-01-18T23:33:42.862+0300 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-01-18T23:33:42.862+0300 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-01-18T23:33:42.960+0300 I STORAGE  [initandlisten] WiredTiger message [1516307622:960077][19599:0x7f54813f0a80], txn-recover: Main recovery loop: starting at 60/256
2018-01-18T23:33:42.960+0300 I STORAGE  [initandlisten] WiredTiger message [1516307622:960231][19599:0x7f54813f0a80], txn-recover: Recovering log 60 through 61
2018-01-18T23:33:42.999+0300 I STORAGE  [initandlisten] WiredTiger message [1516307622:999142][19599:0x7f54813f0a80], txn-recover: Recovering log 61 through 61
2018-01-18T23:33:43.153+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false),
2018-01-18T23:33:43.205+0300 I STORAGE  [initandlisten] Repairing size cache
2018-01-18T23:33:43.205+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2018-01-18T23:33:43.206+0300 I STORAGE  [initandlisten] Repairing catalog metadata
2018-01-18T23:33:43.206+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1516307623:206574][19599:0x7f54813f0a80], file:_mdb_catalog.wt, WT_SESSION.verify: read checksum error for 4096B block at offset 28672: block header checksum of 0 doesn't match expected checksum of 1074498338
2018-01-18T23:33:43.206+0300 I STORAGE  [initandlisten] Verify failed on uri table:_mdb_catalog. Running a salvage operation.
2018-01-18T23:33:43.214+0300 I CONTROL  [initandlisten] 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server. 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] 
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-01-18T23:33:43.215+0300 I CONTROL  [initandlisten] 
2018-01-18T23:33:43.215+0300 I STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: a3471fa1-b9a7-4066-9c51-f9114d90f70e
2018-01-18T23:33:43.278+0300 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.6
2018-01-18T23:33:43.278+0300 I STORAGE  [initandlisten] repairDatabase admin
2018-01-18T23:33:43.278+0300 I STORAGE  [initandlisten] Repairing collection admin.system.version
2018-01-18T23:33:43.306+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--803559634327283876. Not salvaging.
2018-01-18T23:33:43.352+0300 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }
2018-01-18T23:33:43.352+0300 I INDEX    [initandlisten] 	 building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-01-18T23:33:43.437+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-0--2170319704746865155
2018-01-18T23:33:43.455+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-0--2919941413266653091
2018-01-18T23:33:43.472+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-0--3119660283819234981
2018-01-18T23:33:43.489+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-0--5607842886892814276
2018-01-18T23:33:43.508+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-14--2919941413266653091
2018-01-18T23:33:43.539+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-16--2919941413266653091
2018-01-18T23:33:43.559+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-18--2919941413266653091
2018-01-18T23:33:43.585+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-2--2351367194744710572
2018-01-18T23:33:43.614+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-2--3119660283819234981
2018-01-18T23:33:43.639+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-2--5607842886892814276
2018-01-18T23:33:43.663+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-26--2919941413266653091
2018-01-18T23:33:43.682+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-30--2919941413266653091
2018-01-18T23:33:43.706+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-32--2919941413266653091
2018-01-18T23:33:43.726+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-34--2919941413266653091
2018-01-18T23:33:43.745+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-4--2351367194744710572
2018-01-18T23:33:43.764+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-4--5607842886892814276
2018-01-18T23:33:43.789+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-6--2351367194744710572
2018-01-18T23:33:43.808+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-6--5607842886892814276
2018-01-18T23:33:43.828+0300 I STORAGE  [initandlisten] Dropping unknown ident: collection-8--5607842886892814276
2018-01-18T23:33:43.847+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-0-7687852694704450203
2018-01-18T23:33:43.869+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-1-7687852694704450203
2018-01-18T23:33:43.887+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-10-7687852694704450203
2018-01-18T23:33:43.911+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-11-7687852694704450203
2018-01-18T23:33:43.928+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-12-7687852694704450203
2018-01-18T23:33:43.954+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-13-7687852694704450203
2018-01-18T23:33:43.971+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-14-7687852694704450203
2018-01-18T23:33:43.997+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-15-7687852694704450203
2018-01-18T23:33:44.015+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-16-7687852694704450203
2018-01-18T23:33:44.033+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-17-7687852694704450203
2018-01-18T23:33:44.051+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-18-7687852694704450203
2018-01-18T23:33:44.073+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-19-7687852694704450203
2018-01-18T23:33:44.092+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-2-7687852694704450203
2018-01-18T23:33:44.119+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-20-7687852694704450203
2018-01-18T23:33:44.142+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-21-7687852694704450203
2018-01-18T23:33:44.164+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-22-7687852694704450203
2018-01-18T23:33:44.188+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-23-7687852694704450203
2018-01-18T23:33:44.207+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-24-7687852694704450203
2018-01-18T23:33:44.226+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-25-7687852694704450203
2018-01-18T23:33:44.244+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-3-7687852694704450203
2018-01-18T23:33:44.267+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-4-7687852694704450203
2018-01-18T23:33:44.285+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-5-7687852694704450203
2018-01-18T23:33:44.309+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-6-7687852694704450203
2018-01-18T23:33:44.331+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-7-7687852694704450203
2018-01-18T23:33:44.353+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-8-7687852694704450203
2018-01-18T23:33:44.379+0300 I STORAGE  [initandlisten] Dropping unknown ident: index-9-7687852694704450203
2018-01-18T23:33:44.397+0300 I STORAGE  [initandlisten] finished checking dbs
2018-01-18T23:33:44.397+0300 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-01-18T23:33:44.398+0300 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-01-18T23:33:44.398+0300 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-01-18T23:33:44.458+0300 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-01-18T23:33:44.458+0300 I CONTROL  [initandlisten] now exiting
2018-01-18T23:33:44.458+0300 I CONTROL  [initandlisten] shutting down with code:0

there is no replica server or working backup unfortunately

I'm sharing files bellow any help would be really appreciated.

Regards



 Comments   
Comment by Kelsey Schubert [ 16/Feb/18 ]

Hi langston,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information about your data files and we will reopen the ticket.

Regards,
Kelsey

Comment by Kelsey Schubert [ 23/Jan/18 ]

Hi langston,

After reviewing the files, it appears that the _mdb_catalog.wt suffered from some form of os storage layer corruption. Unfortunately, during repair, this file is considered the authoritative list of collections and indexes stored by WiredTiger. Since these data files in the WT Catalog were not referenced by the corrupted _mdb_catalog.wt, the repair process dropped the files to free disk space taken up by data that could never be accessed.

We're looking into modifying the behavior of --repair to not drop unknown idents, but either move them aside or preferably recreate catalog entries for them (in a lost+found database or such) and let the user explicitly delete them again if they're unwanted.

In terms of data recovery, it is not possible at this time to determine whether only _mdb_catalog.wt was affected or if there is corruption on other files as well. If you have a backup prior to running the repair operation, we can attempt a manual repair using these files. If not, I would suggest taking a backup of your current files, and then run a filesystem recovery tool to recover the data files using a recovery software or service. If you're able to recover these files, please let us know so we can attempt to help repair this corruption.

For us to attempt to manually recover data, we would need a complete directory listing of the $dbpath as well as information about options that were set when creating these collections. Additionally, would you please clarify when this mongod was always running 3.6.1? If the collections were created in on an earlier version, would you please specify and clarify whether the Feature Compatibility was upgraded to 3.6?

If the size of your database is not unwieldly, I would be happy to provide a secure upload portal as it may be help us if you could provide your complete $dbpath, but I don't think it's necessary for us to proceed.

Thank you,
Kelsey

Comment by Ramon Fernandez Marina [ 19/Jan/18 ]

Sorry to hear you've run into issues langston, we're looking into this issue and the files you uploaded, and will post updates in this ticket as we have them.

Regards,
Ramón.

Generated at Thu Feb 08 04:31:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.