[SERVER-31258] Mongod doesn't restart after a 100% disk full Created: 26/Sep/17  Updated: 21/Mar/18  Resolved: 12/Feb/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Benjamin [X] Assignee: Kelsey Schubert
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello,

After a 100% disk full on my server, mongod doesn't restart.

I don't kwno how mongod works, it was implemented with another solution.

Here the logs :

2017-09-26T11:08:51.952+0200 I CONTROL [main] ***** SERVER RESTARTED *****
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] MongoDB starting : pid=7502 port=27017 dbpath=/var/lib/mongodb 64-bit host=a01senone001
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] db version v3.2.6
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] allocator: tcmalloc
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] modules: none
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] build environment:
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] distmod: ubuntu1404
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] distarch: x86_64
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] target_arch: x86_64
2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "127.0.0.1", port: 27017 }

, security:

{ authorization: "enabled" }

, storage: { dbPath: "/var/lib/mongodb", journal:

{ enabled: true }

}, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }

}
2017-09-26T11:08:51.978+0200 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-09-26T11:08:51.978+0200 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-09-26T11:08:51.978+0200 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2017-09-26T11:08:51.978+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (0) [1506416931:985915][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 2810340744 doesn't match expected checksum of 882284897
2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (0) [1506416931:985978][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (-31804) [1506416931:985997][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-09-26T11:08:51.986+0200 I - [initandlisten] Fatal Assertion 28558
2017-09-26T11:08:51.986+0200 I - [initandlisten]

***aborting after fassert() failure

Can you help me please ?

It is a production server so it is very urgent !

Thank you.



 Comments   
Comment by Kelsey Schubert [ 26/Sep/17 ]

Hi Dumon,

Would you please upload the WiredTiger.wt and WiredTiger.turtle files from your $dbpath? These files contain metadata, and we will be able to attempt a repair which may resolve this issue.

Kind regards,
Kelsey

Generated at Thu Feb 08 04:26:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.