[SERVER-27947] Failed to repair mongo after system crash Created: 08/Feb/17  Updated: 13/Aug/18  Resolved: 21/Feb/17

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Thoang Tran Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

db version v3.2.5
git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel62
distarch: x86_64
target_arch: x86_64


Attachments: File WT-Repaired-27947.tar.gz     HTML File WiredTiger     HTML File WiredTiger     File WiredTiger.turtle     File WiredTiger.turtle     File WiredTiger.wt     File WiredTiger.wt     File WiredTigerLAS.wt     File _mdb_catalog.wt     File mongod.log     Text File mongod1.log     File repair-27947-2.tar.gz     File sizeStorer.wt     File sizeStorer.wt     File storage.bson    
Operating System: Linux
Participants:

 Description   

The system was crushed, but when i start mong with --repair key , i got read checksum error:

~[root@iocean ~]# sudo mongod --repair --dbpath /data/mongo/
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] MongoDB starting : pid=6801 port=27017 dbpath=/data/mongo/ 64-bit host=iocean
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] db version v3.2.5
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] allocator: tcmalloc
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] modules: none
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] build environment:
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     distmod: rhel62
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     distarch: x86_64
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     target_arch: x86_64
2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/data/mongo/" } }
2017-02-08T09:48:58.738+0700 I -        [initandlisten] Detected data files in /data/mongo/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-02-08T09:48:58.738+0700 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-02-08T09:48:58.738+0700 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (0) [1486522138:751195][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 61440: block header checksum of 842544182 doesn't match expected checksum of 4211214560
2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (0) [1486522138:751233][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (-31804) [1486522138:751249][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:48:58.751+0700 I -        [initandlisten] Fatal Assertion 28558
2017-02-08T09:48:58.751+0700 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-02-08T09:48:58.765+0700 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1332c72 0x1331ba9 0x13323b2 0x7fcfd5f407e0 0x7fcfd5bcf625 0x7fcfd5bd0e05 0x12ba5e2 0x10ac1c3 0x1a9761c 0x1a977bd 0x1a97ba4 0x19cc4dc 0x19cca53 0x19c9a37 0x19cdab9 0x19ea34c 0x1a1feb0 0x1a96237 0x1a96779 0x1a9689b 0x1a2dfa8 0x1a933e5 0x1a5f8cf 0x1a5f9ce 0x1a1cef9 0x1093d0f 0x1090183 0xfb8918 0x9956d6 0x9992bd 0x7fcfd5bbbd5d 0x991fc9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F32C72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F31BA9"},{"b":"400000","o":"F323B2"},{"b":"7FCFD5F31000","o":"F7E0"},{"b":"7FCFD5B9D000","o":"32625","s":"gsignal"},{"b":"7FCFD5B9D000","o":"33E05","s":"abort"},{"b":"400000","o":"EBA5E2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAC1C3"},{"b":"400000","o":"169761C","s":"__wt_eventv"},{"b":"400000","o":"16977BD","s":"__wt_err"},{"b":"400000","o":"1697BA4","s":"__wt_panic"},{"b":"400000","o":"15CC4DC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15CCA53","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15C9A37","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15CDAB9"},{"b":"400000","o":"15EA34C","s":"__wt_btree_open"},{"b":"400000","o":"161FEB0","s":"__wt_conn_btree_open"},{"b":"400000","o":"1696237","s":"__wt_session_get_btree"},{"b":"400000","o":"1696779","s":"__wt_session_get_btree"},{"b":"400000","o":"169689B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"162DFA8","s":"__wt_curfile_open"},{"b":"400000","o":"16933E5"},{"b":"400000","o":"165F8CF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"165F9CE","s":"__wt_metadata_cursor"},{"b":"400000","o":"161CEF9","s":"wiredtiger_open"},{"b":"400000","o":"C93D0F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C90183"},{"b":"400000","o":"BB8918","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5956D6","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5992BD","s":"main"},{"b":"7FCFD5B9D000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"591FC9"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "351982A2A6B22BDE14AD7D42FE7A293C7BA951AF" }, { "b" : "7FFE35F64000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "b" : "7FCFD6DD8000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "b" : "7FCFD69F4000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "b" : "7FCFD67EC000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "b" : "7FCFD65E8000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "b" : "7FCFD6364000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "b" : "7FCFD614E000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "b" : "7FCFD5F31000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "b" : "7FCFD5B9D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "b" : "7FCFD7044000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "b" : "7FCFD5959000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "b" : "7FCFD5672000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "b" : "7FCFD546E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "b" : "7FCFD5242000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "b" : "7FCFD502C000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FCFD4E21000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "b" : "7FCFD4C1E000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7FCFD4A04000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "b" : "7FCFD47E5000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1332c72]
 mongod(+0xF31BA9) [0x1331ba9]
 mongod(+0xF323B2) [0x13323b2]
 libpthread.so.0(+0xF7E0) [0x7fcfd5f407e0]
 libc.so.6(gsignal+0x35) [0x7fcfd5bcf625]
 libc.so.6(abort+0x175) [0x7fcfd5bd0e05]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12ba5e2]
 mongod(+0xCAC1C3) [0x10ac1c3]
 mongod(__wt_eventv+0x40C) [0x1a9761c]
 mongod(__wt_err+0x8D) [0x1a977bd]
 mongod(__wt_panic+0x24) [0x1a97ba4]
 mongod(__wt_block_extlist_read+0x6C) [0x19cc4dc]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19cca53]
 mongod(__wt_block_checkpoint_load+0x3B7) [0x19c9a37]
 mongod(+0x15CDAB9) [0x19cdab9]
 mongod(__wt_btree_open+0xC7C) [0x19ea34c]
 mongod(__wt_conn_btree_open+0x140) [0x1a1feb0]
 mongod(__wt_session_get_btree+0xE7) [0x1a96237]
 mongod(__wt_session_get_btree+0x629) [0x1a96779]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9689b]
 mongod(__wt_curfile_open+0x218) [0x1a2dfa8]
 mongod(+0x16933E5) [0x1a933e5]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a5f8cf]
 mongod(__wt_metadata_cursor+0x7E) [0x1a5f9ce]
 mongod(wiredtiger_open+0x19F9) [0x1a1cef9]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x1093d0f]
 mongod(+0xC90183) [0x1090183]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfb8918]
 mongod(_ZN5mongo13initAndListenEi+0x376) [0x9956d6]
 mongod(main+0x15D) [0x9992bd]
 libc.so.6(__libc_start_main+0xFD) [0x7fcfd5bbbd5d]
 mongod(+0x591FC9) [0x991fc9]
-----  END BACKTRACE  -----

Pls support me resolve this problem,
Thanks & Best regards,



 Comments   
Comment by Mark Agarunov [ 17/Feb/17 ]

Hello vagabond_vn,

I'm glad to hear that the repair worked. Unfortunately, in cases like this it is not clear what failure caused the corruption you observed. If you encounter this issue again, I would recommend checking the integrity of your storage layer.

However, please be aware that MongoDB 3.2.5 is affected by SERVER-22970. Therefore, I would strongly recommend upgrading to MongoDB 3.2.12 or MongoDB 3.4.2 for this fix as well as a number of other improvements.

Thanks,
Mark

Comment by Thoang Tran [ 17/Feb/17 ]

Hello @Mark Agarunov,
It's worked. Thank you so much.
I have a question: Could you please let us know what's version of mongodb, which this issue had been fixed. I will upgrade mongodb.
Thanks,

Comment by Mark Agarunov [ 16/Feb/17 ]

Hello vagabond_vn,

I'm glad to hear that the repair was a success. I've attached a repair attempt of the newer files you've provided. Would you please extract these files and replace them in your $dbpath as before and let us know if it resolves the issue?

Thanks,
Mark

Comment by Thoang Tran [ 16/Feb/17 ]

Thanks for support,
It's worked !!
But, I still a database with the same error because my system was been the power outage.
( I've attached files at Feb 16 2017 10:38:00 AM in attachments)
Pls help me.

Comment by Mark Agarunov [ 08/Feb/17 ]

Hello vagabond_vn,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Thanks,
Mark

Generated at Thu Feb 08 04:16:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.