[SERVER-37747] the process must exit and restart: WT_PANIC: WiredTiger library panic Created: 25/Oct/18 Updated: 06/Mar/19 Resolved: 30/Oct/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Shell, WiredTiger |
| Affects Version/s: | 3.0 Required |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | hanyudong | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | The above description is my most complete startup log.
For more details, please see the attachment. |
||||||||
| Participants: | |||||||||
| Description |
|
2018-10-25T10:17:15.030+0800 W - [initandlisten] Detected unclean shutdown - /data/mongo_data/mongodb/mongod.lock is not empty. ,{"b":"400000","o":"B12271"},{"b":"400000","o":"AF60F1"},{"b":"400000","o":"993F8A"},{"b":"400000","o":"FAB369"},{"b":"400000","o":"FAB525"},{"b":"400000","o":"FAB9C4"},{"b":"400000","o":"EFE3D2"},{"b":"400000","o":"F1811C"},{"b":"400000","o":"F15EF8"},{"b":"400000","o":"F16D52"},{"b":"400000","o":"F40B4B"},{"b":"400000","o":"FAA50B"},{"b":"400000","o":"F780AB"},{"b":"400000","o":"F3E207"},{"b":"400000","o":"97E61C"},{"b":"400000","o":"97C3A8"},{"b":"400000","o":"67DC5D"},{"b":"400000","o":"3F57A2"},{"b":"400000","o":"3FA739"},{"b":"7F1CC184D000","o":"22445"},{"b":"400000","o":"3F3549"}],"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "1ef45a23a4c5e3480ac919b28afcba3c615488f2", "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.el7.x86_64", "version" : "#1 SMP Tue Aug 22 21:09:27 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7AA372EDE22BA34234ADA10A8AF2E665681140" }, { "b" : "7FFFF8CC1000", "elfType" : 3, "buildId" : "7FB8E16CEA1B913E2703A6E4159FB468CD1E3507" }, { "b" : "7F1CC2F18000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F1CC2CA6000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "ED0AC7DEB91A242C194B3DEF27A215F41CE43116" }, { "b" : "7F1CC2845000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "BC0AE9CA0705BEC1F0C0375AAD839843BB219CB1" }, { "b" : "7F1CC263D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F1CC2439000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F1CC2132000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "9589AE0FDA6AEB1183EBA1C62A328F933E7817FD" }, { "b" : "7F1CC1E30000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F1CC1C1A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F1CC184D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F1CC3134000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F1CC1600000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DA322D74F55A0C4293085371A8D0E94B5962F5E7" }, { "b" : "7F1CC1318000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B69E63024D408E400401EEA6815317BDA38FB7C2" }, { "b" : "7F1CC1114000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7F1CC0EE1000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A48639BF901DB554479BFAD114CB354CF63D7D6E" }, { "b" : "7F1CC0CCB000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F1CC0ABD000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "6FDF5B013FD2739D304CFB9D723DCBC149EE03C9" }, { "b" : "7F1CC08B9000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1CC06A0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F1CC0479000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7F1CC0217000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }} ***aborting after fassert() failure |
| Comments |
| Comment by hexiaojie [ 06/Mar/19 ] |
|
I have the same problem,can you help me repair the file or share some methods to repair the file.WiredTiger.turtle |
| Comment by hanyudong [ 01/Nov/18 ] |
|
Hi Kelsey, Thank you for Kelsey's answer, and then I'll follow your guidelines to prevent the next unfortunate incident. Thank you again for Kelsey's answer. Kind regards, |
| Comment by Kelsey Schubert [ 01/Nov/18 ] |
|
Hi hanyd, Yes, it appears that other files (in addition to one originally provided) are corrupted likely as the result of disk failure. Unfortunately, in cases like this, we cannot attempt repair the the damaged files. Kind regards, |
| Comment by hanyudong [ 31/Oct/18 ] |
|
Hi Kelsey, Thank you, |
| Comment by Kelsey Schubert [ 30/Oct/18 ] |
|
Hi hanyd, Unfortunately, this error indicates that there was corruption on the disk, most often caused by a faulty storage layer beneath mongod. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible. To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.
Thank you, |
| Comment by hanyudong [ 30/Oct/18 ] |
|
Hi Kelsey, If you need any more log files, please specify the path name of the file and so on, I will find and upload them Thank you, hanyudong |
| Comment by hanyudong [ 30/Oct/18 ] |
|
Hi Kelsey, I still reported an error when I replaced the file you submitted and executed the startup command, and the relevant screenshots and logs were submitted to 20181030.tar.gz Thank you, hanyudong |
| Comment by Kelsey Schubert [ 26/Oct/18 ] |
|
Hi hanyd, Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz Thank you, |
| Comment by hanyudong [ 25/Oct/18 ] |
|
The above description is my most complete startup log.
For more details, please see the attachment. |