Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40904

Mongo wont restart following disk full

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • ALL

    Description

      We are running Mongo 4.0.0 using a single node setup with a replica set enabled.

      Our data disk filled up and the mongo process stopped. We are now unable to connect to the DB and the process continually takes about between 500% and 800% CPU (We have 8 CPUs available).

      I've increased the verbosity of the system log (to 5) as nothing is emitted otherwise.

      Apologies but I can't share the log files so I have tried to provide a summary below:

      We see lots of STORAGE messages for a while with out data records then printed out under with the text "fetched CCE metadata". This runs for a while.

      We then see Debug messages from RECOVERY where it runs through "Applying op x of 5000") through to 5000 of 5000.

      We then see lots of WT set timestamp of future write operations messages followed by WT begin transaction for snapshot id X and then commit_transaction for snapshot X.

      Eventually these logs message finish to be replaced by:

      WTJournalFlusher flushed Journal every 100ms

      and:

      STORAGE [ftdc] setting timestamp read source: 1, provided timestamp: none

      • [ftdc] User Assertion: NotYetInitialized: no replset config has been received src/mongo/db/repl/repl_set_get_status_cmd.cpp 54

      STROAGE  [ftdc]NamesplaceUUIDCache: registered namespace local.oplog.rs with UUID uuid

      (which happen every second)

       

      Please let me know if you need any more information but our database instance is currently unable to start.

       

      Thanks in advance.

       

       

       

      Attachments

        Activity

          People

            kelsey.schubert@mongodb.com Kelsey Schubert
            leigh.jones@ripjar.com Leigh Jones
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: