[SERVER-30099] Error with wiredTiger.wt Created: 12/Jul/17  Updated: 14/Aug/18  Resolved: 19/Jul/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kesava Raju Mudunuri [X] Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: docker, envc, envm, rge, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-30099-repair.tar.gz     File WiredTiger.turtle     File WiredTiger.wt    
Operating System: Linux
Steps To Reproduce:

it terminal i run mongod

Participants:

 Description   

2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] MongoDB starting : pid=31832 port=27017 dbpath=/data/db 64-bit host=kesav-Latitude-3550
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] db version v3.4.5
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] git version: 520b8f3092c48d934f0cd78ab5f40fe594f96863
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] modules: none
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] build environment:
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-07-12T03:21:38.699+0530 I CONTROL  [initandlisten] options: {}
2017-07-12T03:21:38.725+0530 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-07-12T03:21:38.725+0530 I STORAGE  [initandlisten] 
2017-07-12T03:21:38.725+0530 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-07-12T03:21:38.725+0530 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-07-12T03:21:38.725+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3432M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-07-12T03:21:38.796+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1499809898:796540][31832:0x7fbb81c9ed00], file:WiredTiger.wt, WT_CURSOR.search_near: read checksum error for 32768B block at offset 16384: block header checksum of 415079445 doesn't match expected checksum of 985779774
2017-07-12T03:21:38.796+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1499809898:796578][31832:0x7fbb81c9ed00], file:WiredTiger.wt, WT_CURSOR.search_near: WiredTiger.wt: encountered an illegal file format or internal value
2017-07-12T03:21:38.796+0530 E STORAGE  [initandlisten] WiredTiger error (-31804) [1499809898:796589][31832:0x7fbb81c9ed00], file:WiredTiger.wt, WT_CURSOR.search_near: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-07-12T03:21:38.796+0530 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-07-12T03:21:38.796+0530 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-07-12T03:21:38.812+0530 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x563bc54f0c01 0x563bc54efe19 0x563bc54f02fd 0x7fbb808b8390 0x7fbb80512428 0x7fbb8051402a 0x563bc479f5b3 0x563bc52048e6 0x563bc47a984c 0x563bc47a9a71 0x563bc47a9cc9 0x563bc5df98e5 0x563bc5e1366b 0x563bc5e1a7c5 0x563bc5e3b670 0x563bc5e086ec 0x563bc5e58176 0x563bc5ee9232 0x563bc5ee96d5 0x563bc5e4eb47 0x563bc5e46dc6 0x563bc51e89e9 0x563bc51e1a1c 0x563bc50d54c0 0x563bc478b7de 0x563bc47ab076 0x7fbb804fd830 0x563bc48098d9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"563BC3F86000","o":"156AC01","s":"_ZN5mongo15printStackTraceERSo"},{"b":"563BC3F86000","o":"1569E19"},{"b":"563BC3F86000","o":"156A2FD"},{"b":"7FBB808A7000","o":"11390"},{"b":"7FBB804DD000","o":"35428","s":"gsignal"},{"b":"7FBB804DD000","o":"3702A","s":"abort"},{"b":"563BC3F86000","o":"8195B3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"563BC3F86000","o":"127E8E6"},{"b":"563BC3F86000","o":"82384C","s":"__wt_eventv"},{"b":"563BC3F86000","o":"823A71","s":"__wt_err"},{"b":"563BC3F86000","o":"823CC9","s":"__wt_panic"},{"b":"563BC3F86000","o":"1E738E5","s":"__wt_bm_read"},{"b":"563BC3F86000","o":"1E8D66B","s":"__wt_bt_read"},{"b":"563BC3F86000","o":"1E947C5","s":"__wt_page_in_func"},{"b":"563BC3F86000","o":"1EB5670","s":"__wt_row_search"},{"b":"563BC3F86000","o":"1E826EC","s":"__wt_btcur_search_near"},{"b":"563BC3F86000","o":"1ED2176"},{"b":"563BC3F86000","o":"1F63232"},{"b":"563BC3F86000","o":"1F636D5","s":"__wt_txn_recover"},{"b":"563BC3F86000","o":"1EC8B47","s":"__wt_connection_workers"},{"b":"563BC3F86000","o":"1EC0DC6","s":"wiredtiger_open"},{"b":"563BC3F86000","o":"12629E9","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"563BC3F86000","o":"125BA1C"},{"b":"563BC3F86000","o":"114F4C0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"563BC3F86000","o":"8057DE"},{"b":"563BC3F86000","o":"825076","s":"main"},{"b":"7FBB804DD000","o":"20830","s":"__libc_start_main"},{"b":"563BC3F86000","o":"8838D9","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.5", "gitVersion" : "520b8f3092c48d934f0cd78ab5f40fe594f96863", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.8.0-58-generic", "version" : "#63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "563BC3F86000", "elfType" : 3, "buildId" : "7CA488310DEBB5E4D2FD6653DD93385F07B61791" }, { "b" : "7FFCC4CEB000", "elfType" : 3, "buildId" : "7A29FD6E058E0D42E9E6ED28D7DAC177F8401815" }, { "b" : "7FBB81833000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FBB813EF000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FBB811E7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FBB80FE3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FBB80CDA000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FBB80AC4000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FBB808A7000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FBB804DD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FBB81A9C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x563bc54f0c01]
 mongod(+0x1569E19) [0x563bc54efe19]
 mongod(+0x156A2FD) [0x563bc54f02fd]
 libpthread.so.0(+0x11390) [0x7fbb808b8390]
 libc.so.6(gsignal+0x38) [0x7fbb80512428]
 libc.so.6(abort+0x16A) [0x7fbb8051402a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x563bc479f5b3]
 mongod(+0x127E8E6) [0x563bc52048e6]
 mongod(__wt_eventv+0x3D7) [0x563bc47a984c]
 mongod(__wt_err+0x9D) [0x563bc47a9a71]
 mongod(__wt_panic+0x24) [0x563bc47a9cc9]
 mongod(__wt_bm_read+0x135) [0x563bc5df98e5]
 mongod(__wt_bt_read+0x1FB) [0x563bc5e1366b]
 mongod(__wt_page_in_func+0x1275) [0x563bc5e1a7c5]
 mongod(__wt_row_search+0x660) [0x563bc5e3b670]
 mongod(__wt_btcur_search_near+0x95C) [0x563bc5e086ec]
 mongod(+0x1ED2176) [0x563bc5e58176]
 mongod(+0x1F63232) [0x563bc5ee9232]
 mongod(__wt_txn_recover+0x3D5) [0x563bc5ee96d5]
 mongod(__wt_connection_workers+0x37) [0x563bc5e4eb47]
 mongod(wiredtiger_open+0x1666) [0x563bc5e46dc6]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xA59) [0x563bc51e89e9]
 mongod(+0x125BA1C) [0x563bc51e1a1c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x563bc50d54c0]
 mongod(+0x8057DE) [0x563bc478b7de]
 mongod(main+0x966) [0x563bc47ab076]
 libc.so.6(__libc_start_main+0xF0) [0x7fbb804fd830]
 mongod(_start+0x29) [0x563bc48098d9]
-----  END BACKTRACE  -----
Aborted (core dumped)



 Comments   
Comment by Mark Agarunov [ 19/Jul/17 ]

Hello KesavMR,

Thank you for the response. Unfortunately the error you are seeing indicates that there is underlying corruption of the database which cannot be repaired. My recommendation would be a clean resync from a node that is not affected if possible or restoring from a backup.

To prevent this type of problem in the future, we recommend implementing regular backups and/or replication to mitigate any issues related to unreliable storage layers or server failures.

Thanks,
Mark

Comment by Kesava Raju Mudunuri [X] [ 12/Jul/17 ]

Hi Mark Agarunov,

Complete story is like this

#Original file are in AWS EC2 instance
#Attached a volume to this instance and added ssd
#mounted voulume to the path mnt/vbd
#later installed mongo using docker(docker run -d -v /mnt/vdb:/data/db -p 27017:27017 -e MONGODB_PASS="25742S#pijf" --net=ts-net --name=mongo3 tutum/mongodb:3.0).

  1. it run happly for 6 month. Now it reached dat content to near to 90 gb. So, having trouble with queries due to indexing. So, i try to add indexing.
  2. When adding index it stopped. so, i used repair command on it.
  3. Event after repair it not working. So, i try to restart docker. But, stopped working.
  4. I though it may wrong with new docker, i have updated the docker.
  5. Later i want to preserve the data. so, i left it alone and copied my local had disk and try wt utility. it did worked. But, files i have sent are original file with out run of wt utility.
  6. orginal version of mongo is 3.0.2. now i am trying on 3.4.5

If you need any further information let me know.

Comment by Kesava Raju Mudunuri [X] [ 12/Jul/17 ]

Hi Mark Agarunov,

I am not able to connect after extracting files you provided. Following the complete log of it.

root@kesav-Latitude-3550:/data/db# mongod
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] MongoDB starting : pid=2828 port=27017 dbpath=/data/db 64-bit host=kesav-Latitude-3550
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] db version v3.4.5
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] git version: 520b8f3092c48d934f0cd78ab5f40fe594f96863
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] modules: none
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] build environment:
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-07-12T22:34:11.087+0530 I CONTROL  [initandlisten] options: {}
2017-07-12T22:34:11.115+0530 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-07-12T22:34:11.115+0530 I STORAGE  [initandlisten] 
2017-07-12T22:34:11.115+0530 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-07-12T22:34:11.115+0530 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-07-12T22:34:11.115+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3432M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-07-12T22:34:11.476+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1499879051:476958][2828:0x7f46540f6d00], file:_mdb_catalog.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 12288: block header checksum of 4092924646 doesn't match expected checksum of 3842992612
2017-07-12T22:34:11.476+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1499879051:476994][2828:0x7f46540f6d00], file:_mdb_catalog.wt, WT_SESSION.open_cursor: _mdb_catalog.wt: encountered an illegal file format or internal value
2017-07-12T22:34:11.477+0530 E STORAGE  [initandlisten] WiredTiger error (-31804) [1499879051:477005][2828:0x7f46540f6d00], file:_mdb_catalog.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-07-12T22:34:11.477+0530 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-07-12T22:34:11.477+0530 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-07-12T22:34:11.499+0530 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x55a1f6acac01 0x55a1f6ac9e19 0x55a1f6aca2fd 0x7f4652d10390 0x7f465296a428 0x7f465296c02a 0x55a1f5d795b3 0x55a1f67de8e6 0x55a1f5d8384c 0x55a1f5d83a71 0x55a1f5d83cc9 0x55a1f73d04cf 0x55a1f73d0a1b 0x55a1f73cd00d 0x55a1f73d1be7 0x55a1f73eb7db 0x55a1f7424a53 0x55a1f74af1cb 0x55a1f74af70d 0x55a1f74af9cc 0x55a1f7433221 0x55a1f74a5598 0x55a1f744cc17 0x55a1f74a5334 0x55a1f74a5b65 0x55a1f67d8007 0x55a1f67d5edc 0x55a1f67d2524 0x55a1f67d0d4c 0x55a1f67be0eb 0x55a1f6704349 0x55a1f67bbac7 0x55a1f66af4c0 0x55a1f5d657de 0x55a1f5d85076 0x7f4652955830 0x55a1f5de38d9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55A1F5560000","o":"156AC01","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55A1F5560000","o":"1569E19"},{"b":"55A1F5560000","o":"156A2FD"},{"b":"7F4652CFF000","o":"11390"},{"b":"7F4652935000","o":"35428","s":"gsignal"},{"b":"7F4652935000","o":"3702A","s":"abort"},{"b":"55A1F5560000","o":"8195B3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55A1F5560000","o":"127E8E6"},{"b":"55A1F5560000","o":"82384C","s":"__wt_eventv"},{"b":"55A1F5560000","o":"823A71","s":"__wt_err"},{"b":"55A1F5560000","o":"823CC9","s":"__wt_panic"},{"b":"55A1F5560000","o":"1E704CF","s":"__wt_block_extlist_read"},{"b":"55A1F5560000","o":"1E70A1B","s":"__wt_block_extlist_read_avail"},{"b":"55A1F5560000","o":"1E6D00D","s":"__wt_block_checkpoint_load"},{"b":"55A1F5560000","o":"1E71BE7"},{"b":"55A1F5560000","o":"1E8B7DB","s":"__wt_btree_open"},{"b":"55A1F5560000","o":"1EC4A53","s":"__wt_conn_btree_open"},{"b":"55A1F5560000","o":"1F4F1CB","s":"__wt_session_get_btree"},{"b":"55A1F5560000","o":"1F4F70D","s":"__wt_session_get_btree"},{"b":"55A1F5560000","o":"1F4F9CC","s":"__wt_session_get_btree_ckpt"},{"b":"55A1F5560000","o":"1ED3221","s":"__wt_curfile_open"},{"b":"55A1F5560000","o":"1F45598"},{"b":"55A1F5560000","o":"1EECC17","s":"__wt_curtable_open"},{"b":"55A1F5560000","o":"1F45334"},{"b":"55A1F5560000","o":"1F45B65"},{"b":"55A1F5560000","o":"1278007","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"55A1F5560000","o":"1275EDC","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"55A1F5560000","o":"1272524","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"55A1F5560000","o":"1270D4C","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"55A1F5560000","o":"125E0EB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"55A1F5560000","o":"11A4349","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"55A1F5560000","o":"125BAC7"},{"b":"55A1F5560000","o":"114F4C0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55A1F5560000","o":"8057DE"},{"b":"55A1F5560000","o":"825076","s":"main"},{"b":"7F4652935000","o":"20830","s":"__libc_start_main"},{"b":"55A1F5560000","o":"8838D9","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.5", "gitVersion" : "520b8f3092c48d934f0cd78ab5f40fe594f96863", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.8.0-58-generic", "version" : "#63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55A1F5560000", "elfType" : 3, "buildId" : "7CA488310DEBB5E4D2FD6653DD93385F07B61791" }, { "b" : "7FFE4E1F0000", "elfType" : 3, "buildId" : "7A29FD6E058E0D42E9E6ED28D7DAC177F8401815" }, { "b" : "7F4653C8B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F4653847000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F465363F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F465343B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F4653132000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F4652F1C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F4652CFF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F4652935000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F4653EF4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a1f6acac01]
 mongod(+0x1569E19) [0x55a1f6ac9e19]
 mongod(+0x156A2FD) [0x55a1f6aca2fd]
 libpthread.so.0(+0x11390) [0x7f4652d10390]
 libc.so.6(gsignal+0x38) [0x7f465296a428]
 libc.so.6(abort+0x16A) [0x7f465296c02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55a1f5d795b3]
 mongod(+0x127E8E6) [0x55a1f67de8e6]
 mongod(__wt_eventv+0x3D7) [0x55a1f5d8384c]
 mongod(__wt_err+0x9D) [0x55a1f5d83a71]
 mongod(__wt_panic+0x24) [0x55a1f5d83cc9]
 mongod(__wt_block_extlist_read+0x8F) [0x55a1f73d04cf]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x55a1f73d0a1b]
 mongod(__wt_block_checkpoint_load+0x26D) [0x55a1f73cd00d]
 mongod(+0x1E71BE7) [0x55a1f73d1be7]
 mongod(__wt_btree_open+0xB3B) [0x55a1f73eb7db]
 mongod(__wt_conn_btree_open+0x163) [0x55a1f7424a53]
 mongod(__wt_session_get_btree+0xFB) [0x55a1f74af1cb]
 mongod(__wt_session_get_btree+0x63D) [0x55a1f74af70d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55a1f74af9cc]
 mongod(__wt_curfile_open+0x161) [0x55a1f7433221]
 mongod(+0x1F45598) [0x55a1f74a5598]
 mongod(__wt_curtable_open+0x827) [0x55a1f744cc17]
 mongod(+0x1F45334) [0x55a1f74a5334]
 mongod(+0x1F45B65) [0x55a1f74a5b65]
 mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x77) [0x55a1f67d8007]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x55a1f67d5edc]
 mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x64) [0x55a1f67d2524]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x47C) [0x55a1f67d0d4c]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x23B) [0x55a1f67be0eb]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x679) [0x55a1f6704349]
 mongod(+0x125BAC7) [0x55a1f67bbac7]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55a1f66af4c0]
 mongod(+0x8057DE) [0x55a1f5d657de]
 mongod(main+0x966) [0x55a1f5d85076]
 libc.so.6(__libc_start_main+0xF0) [0x7f4652955830]
 mongod(_start+0x29) [0x55a1f5de38d9]
-----  END BACKTRACE  -----
Aborted (core dumped)
root@kesav-Latitude-3550:/data/db# ^C
root@kesav-Latitude-3550:/data/db# ^C
root@kesav-Latitude-3550:/data/db# 

Comment by Mark Agarunov [ 12/Jul/17 ]

Hello KesavMR,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Comment by Kesava Raju Mudunuri [X] [ 12/Jul/17 ]

Hi Mark Agarunov,

Thank you for response. I have uploaded the files.

Comment by Mark Agarunov [ 12/Jul/17 ]

Hello KesavMR,

Thank you for the report. To further troubleshoot this issue, I would like to request a couple of files. Please upload the following:

  • WiredTiger.wt
  • WiredTiger.turtle

With these files, we can attempt a repair, although please note that there is no guarantee that the repair will be successful.

Thanks,
Mark

Comment by Kesava Raju Mudunuri [X] [ 12/Jul/17 ]

As mongo size reached 90 gb. We are getting timeout errors for non index fields. So, i try to index those fields. When i am on it, mongo got corrupted and not able run anymore. So, try to repair it. But, later it stopped working.Then it showing error in WiredTiger.wt file.

Generated at Thu Feb 08 04:22:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.