Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31258

Mongod doesn't restart after a 100% disk full

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major - P3 Major - P3
    • None
    • None
    • WiredTiger
    • None
    • ALL

    Description

      Hello,

      After a 100% disk full on my server, mongod doesn't restart.

      I don't kwno how mongod works, it was implemented with another solution.

      Here the logs :

      2017-09-26T11:08:51.952+0200 I CONTROL [main] ***** SERVER RESTARTED *****
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] MongoDB starting : pid=7502 port=27017 dbpath=/var/lib/mongodb 64-bit host=a01senone001
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] db version v3.2.6
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] allocator: tcmalloc
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] modules: none
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] build environment:
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] distmod: ubuntu1404
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] distarch: x86_64
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] target_arch: x86_64
      2017-09-26T11:08:51.959+0200 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

      { bindIp: "127.0.0.1", port: 27017 }

      , security:

      { authorization: "enabled" }

      , storage: { dbPath: "/var/lib/mongodb", journal:

      { enabled: true }

      }, systemLog:

      { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }

      }
      2017-09-26T11:08:51.978+0200 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-09-26T11:08:51.978+0200 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
      2017-09-26T11:08:51.978+0200 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      2017-09-26T11:08:51.978+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (0) [1506416931:985915][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 2810340744 doesn't match expected checksum of 882284897
      2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (0) [1506416931:985978][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2017-09-26T11:08:51.985+0200 E STORAGE [initandlisten] WiredTiger (-31804) [1506416931:985997][7502:0x7faf87ea2cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-09-26T11:08:51.986+0200 I - [initandlisten] Fatal Assertion 28558
      2017-09-26T11:08:51.986+0200 I - [initandlisten]

      ***aborting after fassert() failure

      Can you help me please ?

      It is a production server so it is very urgent !

      Thank you.

      Attachments

        Activity

          People

            kelsey.schubert@mongodb.com Kelsey Schubert
            Dumon Benjamin [X]
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: