[SERVER-32471] Mongo db crashes when start. Please help! Created: 28/Dec/17  Updated: 27/Jul/18  Resolved: 31/Dec/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.18
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Pitstopper [X] Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File _mdb_catalog.wt     File repair_attempt.tar.gz     File sizeStorer.wt    
Operating System: Linux
Steps To Reproduce:

Shutdown and start server

Participants:

 Description   

2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] MongoDB starting : pid=12340 port=27017 dbpath=/var/lib/mongodb 64-bit host=vds.tarasov.co
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] db version v3.2.18
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] git version: 4c1bae566c0c00f996a2feb16febf84936ecaf6f
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] modules: none
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] build environment:
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten]     distmod: debian81
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten]     distarch: x86_64
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2017-12-28T12:33:28.817+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/$
2017-12-28T12:33:28.840+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-12-28T12:33:28.840+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-12-28T12:33:28.840+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-12-28T12:33:28.840+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=2G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=$
2017-12-28T12:33:29.049+0300 E STORAGE  [initandlisten] WiredTiger (0) [1514453609:49326][12340:0x7f84ba3e6cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1481462559 doesn't match expected checksum of 3$
2017-12-28T12:33:29.049+0300 E STORAGE  [initandlisten] WiredTiger (0) [1514453609:49414][12340:0x7f84ba3e6cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-12-28T12:33:29.049+0300 E STORAGE  [initandlisten] WiredTiger (-31804) [1514453609:49444][12340:0x7f84ba3e6cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-12-28T12:33:29.049+0300 I -        [initandlisten] Fatal Assertion 28558
2017-12-28T12:33:29.049+0300 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-12-28T12:33:29.539+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1359142 0x1358279 0x1358a82 0x7f84b9047890 0x7f84b8cc2067 0x7f84b8cc3448 0x12d9002 0x10cc403 0x978b31 0x978d2e 0x978ef2 0x19ee08c 0x19ee5f3 0x19eae89 0x19ef889 0x1a08aac 0x1a42e93 0x1ac8fcd 0x1ac94e9 0x1ac970b 0x1a506d6 0x1ac5ecd 0x1a8a38f 0x1a8a48e 0x1a3f62a 0x10b38$
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F59142","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F58279"},{"b":"400000","o":"F58A82"},{"b":"7F84B9038000","o":"F890"},{"b":"7F84B8C8D000","o":"35067","s":"gsignal"},{"b":"7F84B8C8D000","o":"36448","s":"abort"},{"b":"40000$
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1359142]
 mongod(+0xF58279) [0x1358279]
 mongod(+0xF58A82) [0x1358a82]
 libpthread.so.0(+0xF890) [0x7f84b9047890]
 libc.so.6(gsignal+0x37) [0x7f84b8cc2067]
 libc.so.6(abort+0x148) [0x7f84b8cc3448]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12d9002]
 mongod(+0xCCC403) [0x10cc403]
 mongod(__wt_eventv+0x440) [0x978b31]
 mongod(__wt_err+0x8D) [0x978d2e]
 mongod(__wt_panic+0x24) [0x978ef2]
 mongod(__wt_block_extlist_read+0x6C) [0x19ee08c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19ee5f3]
 mongod(__wt_block_checkpoint_load+0x359) [0x19eae89]
 mongod(+0x15EF889) [0x19ef889]
 mongod(__wt_btree_open+0xBDC) [0x1a08aac]
 mongod(__wt_conn_btree_open+0x153) [0x1a42e93]
 mongod(__wt_session_get_btree+0xDD) [0x1ac8fcd]
 mongod(__wt_session_get_btree+0x5F9) [0x1ac94e9]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1ac970b]
 mongod(__wt_curfile_open+0x86) [0x1a506d6]
 mongod(+0x16C5ECD) [0x1ac5ecd]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a8a38f]
 mongod(__wt_metadata_cursor+0x7E) [0x1a8a48e]
 mongod(wiredtiger_open+0x191A) [0x1a3f62a]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x5B6) [0x10b3836]
 mongod(+0xCAFA73) [0x10afa73]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x59E) [0xfd7bae]
 mongod(+0x5C3BC0) [0x9c3bc0]
 mongod(main+0x15D) [0x9797fd]
 libc.so.6(__libc_start_main+0xF5) [0x7f84b8caeb45]
 mongod(+0x5BF597) [0x9bf597]



 Comments   
Comment by Kelsey Schubert [ 31/Dec/17 ]

Thanks for your response. I'm glad to hear that this fixed the issue and everything is working again. To prevent this type of problem in the future, please take note of the following guidelines to help mitigate any issues related to unreliable storage layers:

Kind regards,
Kelsey

Comment by Pitstopper [X] [ 31/Dec/17 ]

It works! Thanks!!!

Comment by Kelsey Schubert [ 29/Dec/17 ]

Hello Pitstopper,

Thank you for your report. I've attached a repair attempt of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node(s) so that we can further investigate.

Thanks,
Kelsey

Generated at Thu Feb 08 04:30:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.