[SERVER-31728] Problem repair Database Created: 26/Oct/17  Updated: 07/Dec/17  Resolved: 10/Nov/17

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Antonio Assignee: Mark Agarunov
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We have this error recovering database

[PRO][eacxrdbp02].root:mongo > mongod --repair --dbpath /var/lib/mongo/ --storageEngine wiredTiger
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] MongoDB starting : pid=52936 port=27017 dbpath=/var/lib/mongo/ 64-bit host=eacxrdbp02
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] db version v3.2.5
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] modules: none
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] build environment:
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten]     distmod: rhel62
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-26T10:32:15.579+0200 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongo/", engine: "wiredTiger" } }
2017-10-26T10:32:15.606+0200 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2017-10-26T10:32:15.606+0200 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-10-26T10:32:15.606+0200 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-10-26T10:32:15.606+0200 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=22G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-26T10:32:15.613+0200 E STORAGE  [initandlisten] WiredTiger (0) [1509006735:613450][52936:0x7fc5d90c0d80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 2581697093 doesn't match expected checksum of 3463342742
2017-10-26T10:32:15.613+0200 E STORAGE  [initandlisten] WiredTiger (0) [1509006735:613521][52936:0x7fc5d90c0d80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-10-26T10:32:15.613+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1509006735:613535][52936:0x7fc5d90c0d80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-26T10:32:15.613+0200 I -        [initandlisten] Fatal Assertion 28558
2017-10-26T10:32:15.613+0200 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-10-26T10:32:15.634+0200 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1332c72 0x1331ba9 0x13323b2 0x7fc5d7ba97e0 0x7fc5d7838495 0x7fc5d7839c75 0x12ba5e2 0x10ac1c3 0x1a9761c 0x1a977bd 0x1a97ba4 0x19cc4dc 0x19cca53 0x19c9a37 0x19cdab9 0x19ea34c 0x1a1feb0 0x1a96237 0x1a96779 0x1a9689b 0x1a2dfa8 0x1a933e5 0x1a5f8cf 0x1a5f9ce 0x1a1cef9 0x1093d0f 0x1090183 0xfb8918 0x9956d6 0x9992bd 0x7fc5d7824d1d 0x991fc9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F32C72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F31BA9"},{"b":"400000","o":"F323B2"},{"b":"7FC5D7B9A000","o":"F7E0"},{"b":"7FC5D7806000","o":"32495","s":"gsignal"},{"b":"7FC5D7806000","o":"33C75","s":"abort"},{"b":"400000","o":"EBA5E2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAC1C3"},{"b":"400000","o":"169761C","s":"__wt_eventv"},{"b":"400000","o":"16977BD","s":"__wt_err"},{"b":"400000","o":"1697BA4","s":"__wt_panic"},{"b":"400000","o":"15CC4DC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15CCA53","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15C9A37","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15CDAB9"},{"b":"400000","o":"15EA34C","s":"__wt_btree_open"},{"b":"400000","o":"161FEB0","s":"__wt_conn_btree_open"},{"b":"400000","o":"1696237","s":"__wt_session_get_btree"},{"b":"400000","o":"1696779","s":"__wt_session_get_btree"},{"b":"400000","o":"169689B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"162DFA8","s":"__wt_curfile_open"},{"b":"400000","o":"16933E5"},{"b":"400000","o":"165F8CF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"165F9CE","s":"__wt_metadata_cursor"},{"b":"400000","o":"161CEF9","s":"wiredtiger_open"},{"b":"400000","o":"C93D0F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C90183"},{"b":"400000","o":"BB8918","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5956D6","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5992BD","s":"main"},{"b":"7FC5D7806000","o":"1ED1D","s":"__libc_start_main"},{"b":"400000","o":"591FC9"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-696.6.3.el6.x86_64", "version" : "#1 SMP Fri Jun 30 13:24:18 EDT 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "351982A2A6B22BDE14AD7D42FE7A293C7BA951AF" }, { "b" : "7FFE805C6000", "elfType" : 3, "buildId" : "0C1A2C37D0F2FBADB3C485F048F596F3A3DAA902" }, { "b" : "7FC5D8CAE000", "path" : "/lib64/liboit.so.1", "elfType" : 3, "buildId" : "25B8E82EDB88E39046DB67DCE9C30CACBF23308F" }, { "b" : "7FC5D8A42000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D256E285C5E11D9A99EB04CA7651003A8F67B64E" }, { "b" : "7FC5D865D000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "1EDB45C205A844A75EBBB4F0075E705803FFB85B" }, { "b" : "7FC5D8455000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "FDF3A36FFFE08375456D59DA959EAB2FC30B6186" }, { "b" : "7FC5D8251000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, { "b" : "7FC5D7FCD000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8A852AC42F0B64F0F30C760EBBCFA3FE4A228F12" }, { "b" : "7FC5D7DB7000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC7550A8A7C2D706FE4E489058BADC963465DBB7" }, { "b" : "7FC5D7B9A000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "85104ECFE42C606B31C2D0D0D2E5DACD3286A341" }, { "b" : "7FC5D7806000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8A7E7404A2335231BE759CB54F8041344CAC0C1B" }, { "b" : "7FC5D8EB9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1CC2165E019D43F71FDE0A47AF9F4C8EB5E51963" }, { "path" : "/lib64/libuuid.so.1", "elfType" : 3, "buildId" : "666D9F17296D8AB78F665EB1E3B5528BE2D5F9F2" }, { "b" : "7FC5D75C2000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "b" : "7FC5D72DB000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "b" : "7FC5D70AF000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "b" : "7FC5D6E95000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7FC5D6C8A000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "2D0F26E648D9661ABD83ED8B4BBE8F2CFA50393B" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1332c72]
 mongod(+0xF31BA9) [0x1331ba9]
 mongod(+0xF323B2) [0x13323b2]
 libpthread.so.0(+0xF7E0) [0x7fc5d7ba97e0]
 libc.so.6(gsignal+0x35) [0x7fc5d7838495]
 libc.so.6(abort+0x175) [0x7fc5d7839c75]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12ba5e2]
 mongod(+0xCAC1C3) [0x10ac1c3]
 mongod(__wt_eventv+0x40C) [0x1a9761c]
 mongod(__wt_err+0x8D) [0x1a977bd]
 mongod(__wt_panic+0x24) [0x1a97ba4]
 mongod(__wt_block_extlist_read+0x6C) [0x19cc4dc]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19cca53]
 mongod(__wt_block_checkpoint_load+0x3B7) [0x19c9a37]
 mongod(+0x15CDAB9) [0x19cdab9]
 mongod(__wt_btree_open+0xC7C) [0x19ea34c]
 mongod(__wt_conn_btree_open+0x140) [0x1a1feb0]
 mongod(__wt_session_get_btree+0xE7) [0x1a96237]
 mongod(__wt_session_get_btree+0x629) [0x1a96779]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9689b]
 mongod(__wt_curfile_open+0x218) [0x1a2dfa8]
 mongod(+0x16933E5) [0x1a933e5]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a5f8cf]
 mongod(__wt_metadata_cursor+0x7E) [0x1a5f9ce]
 mongod(wiredtiger_open+0x19F9) [0x1a1cef9]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x1093d0f]
 mongod(+0xC90183) [0x1090183]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfb8918]
 mongod(_ZN5mongo13initAndListenEi+0x376) [0x9956d6]
 mongod(main+0x15D) [0x9992bd]
 libc.so.6(__libc_start_main+0xFD) [0x7fc5d7824d1d]
 mongod(+0x591FC9) [0x991fc9]
-----  END BACKTRACE  -----
Aborted
 



 Comments   
Comment by Mark Agarunov [ 10/Nov/17 ]

Hello dokuflex,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 26/Oct/17 ]

Hello dokuflex,

Thank you for the report. Would you please upload the WiredTiger.wt and WiredTiger.turtle files so we can attempt a repair?

Thanks,
Mark

Generated at Thu Feb 08 04:28:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.