[SERVER-32079] WiredTiger checksum error Created: 23/Nov/17  Updated: 06/Dec/22  Resolved: 12/Feb/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Arthur Alfredo de Souza Assignee: Backlog - Triage Team
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tgz    
Assigned Teams:
Server Triage
Operating System: ALL
Participants:

 Description   

My mongo stopped working unexpectedly. When I tried to repair, it returned the error below.

2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] db version v3.4.4
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] git version: 888390515874a9debd1b6c5d36559ca86b44babd
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] allocator: tcmalloc
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] modules: none
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten] build environment:
2017-11-23T09:05:51.408-0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-11-23T09:05:51.409-0200 I CONTROL  [initandlisten]     distarch: x86_64
2017-11-23T09:05:51.409-0200 I CONTROL  [initandlisten]     target_arch: x86_64
2017-11-23T09:05:51.409-0200 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/home/arthur/Projetos/gideoes/db/_mongodb/" } }
2017-11-23T09:05:51.520-0200 I -        [initandlisten] Detected data files in /home/arthur/Projetos/gideoes/db/_mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-11-23T09:05:51.520-0200 I STORAGE  [initandlisten] 
2017-11-23T09:05:51.520-0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-11-23T09:05:51.520-0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-11-23T09:05:51.540-0200 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-11-23T09:05:51.540-0200 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1395M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-11-23T09:06:14.554-0200 E STORAGE  [initandlisten] WiredTiger error (0) [1511435174:554941][11008:0x7fbb9e3facc0], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 16384B block at offset 327680: block header checksum of 3712086212 doesn't match expected checksum of 3321884558
2017-11-23T09:06:14.555-0200 E STORAGE  [initandlisten] WiredTiger error (0) [1511435174:555087][11008:0x7fbb9e3facc0], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value
2017-11-23T09:06:14.555-0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1511435174:555108][11008:0x7fbb9e3facc0], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-11-23T09:06:14.555-0200 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-11-23T09:06:14.555-0200 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-11-23T09:06:14.803-0200 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x560150aabbe1 0x560150aaacd9 0x560150aab1bd 0x7fbb9d018390 0x7fbb9cc72428 0x7fbb9cc7402a 0x56014fd462c7 0x5601507ba466 0x56014fd50801 0x56014fd50a26 0x56014fd50c7e 0x5601513af765 0x5601513c90ab 0x5601513d014f 0x5601513e871e 0x5601513b4b19 0x56015140d873 0x56015149e951 0x56015149edd3 0x5601514045f7 0x5601513fc556 0x56015079dd53 0x56015079692c 0x560150685ea0 0x56014fd3153e 0x56014fd51fdf 0x7fbb9cc5d830 0x56014fdb0349
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"56014F535000","o":"1576BE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56014F535000","o":"1575CD9"},{"b":"56014F535000","o":"15761BD"},{"b":"7FBB9D007000","o":"11390"},{"b":"7FBB9CC3D000","o":"35428","s":"gsignal"},{"b":"7FBB9CC3D000","o":"3702A","s":"abort"},{"b":"56014F535000","o":"8112C7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"56014F535000","o":"1285466"},{"b":"56014F535000","o":"81B801"},{"b":"56014F535000","o":"81BA26","s":"__wt_err"},{"b":"56014F535000","o":"81BC7E","s":"__wt_panic"},{"b":"56014F535000","o":"1E7A765"},{"b":"56014F535000","o":"1E940AB"},{"b":"56014F535000","o":"1E9B14F"},{"b":"56014F535000","o":"1EB371E"},{"b":"56014F535000","o":"1E7FB19"},{"b":"56014F535000","o":"1ED8873"},{"b":"56014F535000","o":"1F69951"},{"b":"56014F535000","o":"1F69DD3"},{"b":"56014F535000","o":"1ECF5F7"},{"b":"56014F535000","o":"1EC7556","s":"wiredtiger_open"},{"b":"56014F535000","o":"1268D53","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"56014F535000","o":"126192C"},{"b":"56014F535000","o":"1150EA0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"56014F535000","o":"7FC53E"},{"b":"56014F535000","o":"81CFDF","s":"main"},{"b":"7FBB9CC3D000","o":"20830","s":"__libc_start_main"},{"b":"56014F535000","o":"87B349","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.4", "gitVersion" : "888390515874a9debd1b6c5d36559ca86b44babd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.10.0-40-generic", "version" : "#44~16.04.1-Ubuntu SMP Thu Nov 9 15:37:44 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "56014F535000", "elfType" : 3, "buildId" : "5754B63E6FC9683BABEC9CDF7BF6764E6939C8A2" }, { "b" : "7FFD055FE000", "elfType" : 3, "buildId" : "3A644B3A3B09EF95D694352A0C04ECFB65C2D065" }, { "b" : "7FBB9DF93000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "42D1D917636C64CF990465F087B181697E4CDD7F" }, { "b" : "7FBB9DB4F000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "C09D2E075184E1CA71F7E08957982A947A377939" }, { "b" : "7FBB9D947000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FBB9D743000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FBB9D43A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FBB9D224000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FBB9D007000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FBB9CC3D000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FBB9E1FC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x560150aabbe1]
 mongod(+0x1575CD9) [0x560150aaacd9]
 mongod(+0x15761BD) [0x560150aab1bd]
 libpthread.so.0(+0x11390) [0x7fbb9d018390]
 libc.so.6(gsignal+0x38) [0x7fbb9cc72428]
 libc.so.6(abort+0x16A) [0x7fbb9cc7402a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x56014fd462c7]
 mongod(+0x1285466) [0x5601507ba466]
 mongod(+0x81B801) [0x56014fd50801]
 mongod(__wt_err+0x9D) [0x56014fd50a26]
 mongod(__wt_panic+0x24) [0x56014fd50c7e]
 mongod(+0x1E7A765) [0x5601513af765]
 mongod(+0x1E940AB) [0x5601513c90ab]
 mongod(+0x1E9B14F) [0x5601513d014f]
 mongod(+0x1EB371E) [0x5601513e871e]
 mongod(+0x1E7FB19) [0x5601513b4b19]
 mongod(+0x1ED8873) [0x56015140d873]
 mongod(+0x1F69951) [0x56015149e951]
 mongod(+0x1F69DD3) [0x56015149edd3]
 mongod(+0x1ECF5F7) [0x5601514045f7]
 mongod(wiredtiger_open+0x1666) [0x5601513fc556]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xD33) [0x56015079dd53]
 mongod(+0x126192C) [0x56015079692c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x560150685ea0]
 mongod(+0x7FC53E) [0x56014fd3153e]
 mongod(main+0x95F) [0x56014fd51fdf]
 libc.so.6(__libc_start_main+0xF0) [0x7fbb9cc5d830]
 mongod(_start+0x29) [0x56014fdb0349]
-----  END BACKTRACE  -----



 Comments   
Comment by Ramon Fernandez Marina [ 12/Feb/18 ]

arthurras, we haven't heard back from you for some time so I'm closing this ticket.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 27/Nov/17 ]

Sorry to hear you've run into this issue. Please find attached the repair_attempt.tgz tarball, extract it in your dbpath and try to start mongod again. I'd also recommend you upgrade to MongoDB 3.4.10 in the process.

Thanks,
Ramón.

Generated at Thu Feb 08 04:29:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.