[SERVER-27999] WiredTiger.wt read checksum error Created: 14/Feb/17  Updated: 06/Aug/18  Resolved: 16/Feb/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: hy Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-27999-repair.tar.gz     File WiredTiger.turtle     File WiredTiger.wt     File sizeStorer.wt    
Operating System: Linux
Participants:

 Description   

Hi,
I've a problem with mongodb. I wasn't able to start it.
When I try to start I've the following errors:

2017-02-13T05:35:12.154+0000 I CONTROL  [main] ** WARNING: --rest is specified without --httpinterface,
2017-02-13T05:35:12.154+0000 I CONTROL  [main] **          enabling http interface
2017-02-13T05:35:12.163+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=ebd0973efda0
2017-02-13T05:35:12.163+0000 I CONTROL  [initandlisten] db version v3.2.6
2017-02-13T05:35:12.163+0000 I CONTROL  [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-02-13T05:35:12.163+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten] modules: none
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten] build environment:
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten]     distmod: debian71
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-02-13T05:35:12.164+0000 I CONTROL  [initandlisten] options: { net: { http: { RESTInterfaceEnabled: true, enabled: true } }, storage: { mmapv1: { smallFiles: true } } }
2017-02-13T05:35:12.177+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-13T05:35:12.611+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2017-02-13T05:35:12.615+0000 I NETWORK  [websvr] admin web console waiting for connections on port 28017
2017-02-13T05:35:12.615+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2017-02-13T05:35:12.615+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-02-13T05:35:12.634+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2017-02-13T05:35:44.446+0000 I NETWORK  [initandlisten] connection accepted from 172.17.0.1:47916 #1 (1 connection now open)
2017-02-13T05:35:51.921+0000 I NETWORK  [conn1] end connection 172.17.0.1:47916 (0 connections now open)
2017-02-13T05:37:12.635+0000 E STORAGE  [thread1] WiredTiger (-31802) [1486964232:634906][1:0x7fd54785c700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.wt: handle-read: pread: failed to read 4096 bytes at offset 208896: WT_ERROR: non-specific WiredTiger error
2017-02-13T05:37:12.635+0000 E STORAGE  [thread1] WiredTiger (-31802) [1486964232:635696][1:0x7fd54785c700], checkpoint-server: checkpoint server error: WT_ERROR: non-specific WiredTiger error
2017-02-13T05:37:12.635+0000 E STORAGE  [thread1] WiredTiger (-31804) [1486964232:635776][1:0x7fd54785c700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-13T05:37:12.635+0000 I -        [thread1] Fatal Assertion 28558
2017-02-13T05:37:12.635+0000 I -        [thread1] 
 
***aborting after fassert() failure

I tried to install WiredTiger tool on my mac and use the Salvage command but it still logs the same error.
Could you please help to recover my database, it's very important.



 Comments   
Comment by Mark Agarunov [ 16/Feb/17 ]

Hello hy,

I'm sorry the repair attempt was unsuccessful. Unfortunately, this error indicates that additional files were corrupted. In this situation, my recommendation would be to restore from a backup if possible.

Thanks,
Mark

Comment by hy [ 15/Feb/17 ]

@Mark Agarunov Thanks for you help. When I replace the two files above, the mongo server still isn't able to start. May be it relates to sizeStorer.wt, so I attached that file. Here is the logs, could you help me to check again please.

2017-02-15T02:52:03.714+0000 I CONTROL  [main] ** WARNING: --rest is specified without --httpinterface,
2017-02-15T02:52:03.715+0000 I CONTROL  [main] **          enabling http interface
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=87e859284346
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] db version v3.2.6
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] modules: none
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] build environment:
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten]     distmod: debian71
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-02-15T02:52:03.723+0000 I CONTROL  [initandlisten] options: { net: { http: { RESTInterfaceEnabled: true, enabled: true } }, storage: { mmapv1: { smallFiles: true } } }
2017-02-15T02:52:03.744+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-02-15T02:52:03.764+0000 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2017-02-15T02:52:03.768+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-02-15T02:52:03.772+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-15T02:52:12.593+0000 E STORAGE  [initandlisten] WiredTiger (-31802) [1487127132:593864][1:0x7f1544e82c80], file:sizeStorer.wt, WT_SESSION.open_cursor: unable to read root page from file:sizeStorer.wt: WT_ERROR: non-specific WiredTiger error
2017-02-15T02:52:12.597+0000 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_size_storer.cpp 67
2017-02-15T02:52:12.709+0000 I CONTROL  [initandlisten] 
 0x12ffdb2 0x129b2c8 0x12871ad 0x106b812 0x1057527 0x1053723 0xf7a868 0x9485e6 0x94c1fd 0x7f15437e6ead 0x944f29
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EFFDB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"E9B2C8","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"E871AD","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"C6B812","s":"_ZN5mongo20WiredTigerSizeStorerC1EP15__wt_connectionRKSs"},{"b":"400000","o":"C57527","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C53723"},{"b":"400000","o":"B7A868","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5485E6","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54C1FD","s":"main"},{"b":"7F15437C8000","o":"1EEAD","s":"__libc_start_main"},{"b":"400000","o":"544F29"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.27-moby", "version" : "#1 SMP Wed Oct 26 14:21:29 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5F7AC844520FF83AD50A0F7191BF780A5CB1FE5F" }, { "b" : "7FFD53BAD000", "elfType" : 3, "buildId" : "E6835FB8E537C5F5E61CC1B051F3E4FE80AD48EE" }, { "b" : "7F1544A0E000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7F1544615000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "079D784F9F4A88B6D87FC3E6B0B07BFBE209540E" }, { "b" : "7F154440D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "77CB0070BB75470C626B899800FFCF8C8184E32A" }, { "b" : "7F1544209000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "32CCD52B7228B03E969C21587F14C2322D4A9544" }, { "b" : "7F1543F87000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "98FAFAB6E776EA40FBFFFBCB22859A8D54028EC0" }, { "b" : "7F1543D71000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7F1543B55000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "F260CD217A3C0D89FD80BCC80D486C1C2AD32DCA" }, { "b" : "7F15437C8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "BD7458ADC60E906DBAABB72A7DA5015D5BC8269F" }, { "b" : "7F1544C6D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B66F918D37C1E4122FC91C74210F2112A6A5148E" }, { "b" : "7F15435B1000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12ffdb2]
 mongod(_ZN5mongo10logContextEPKc+0x138) [0x129b2c8]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xAD) [0x12871ad]
 mongod(_ZN5mongo20WiredTigerSizeStorerC1EP15__wt_connectionRKSs+0x222) [0x106b812]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x6E7) [0x1057527]
 mongod(+0xC53723) [0x1053723]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf7a868]
 mongod(_ZN5mongo13initAndListenEi+0x376) [0x9485e6]
 mongod(main+0x15D) [0x94c1fd]
 libc.so.6(__libc_start_main+0xFD) [0x7f15437e6ead]
 mongod(+0x544F29) [0x944f29]
-----  END BACKTRACE  -----
2017-02-15T02:52:12.709+0000 I -        [initandlisten] 
 
***aborting after invariant() failure
 
 
2017-02-15T03:23:53.343+0000 I CONTROL  [main] ** WARNING: --rest is specified without --httpinterface,
2017-02-15T03:23:53.344+0000 I CONTROL  [main] **          enabling http interface
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=87e859284346
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] db version v3.2.6
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] modules: none
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] build environment:
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten]     distmod: debian71
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-02-15T03:23:53.352+0000 I CONTROL  [initandlisten] options: { net: { http: { RESTInterfaceEnabled: true, enabled: true } }, storage: { mmapv1: { smallFiles: true } } }
2017-02-15T03:23:53.367+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-02-15T03:23:53.387+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-15T03:23:53.876+0000 E STORAGE  [initandlisten] WiredTiger (-31803) [1487129033:876114][1:0x7f3bac899c80], txn-recover: Recovery failed: WT_NOTFOUND: item not found
2017-02-15T03:23:53.883+0000 I -        [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
2017-02-15T03:23:53.885+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
2017-02-15T03:23:53.885+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

Comment by Mark Agarunov [ 14/Feb/17 ]

Hello hy,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Thanks,
Mark

Generated at Thu Feb 08 04:16:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.