Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34403

The journal files grow out of disk size

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • 3.6.3
    • None
    • None
    • OS: Windows Server 2008 R2 Enterprise with Service Pack 1 (64 bit)
      CPU: Intel Xeon CPU E5-26xx v2 2.6GHz
      RAM: 4 GB
      disk: 100 GB disk partition for MongoDB.
      MongoDB Replica Set: 3 instances, this instance is the Primary.
    • Storage Execution

    Description

      Mongod crashed because of no space left on disk. This is the log when crash happened:

      2018-04-09T16:13:33.568+0800 I COMMAND [conn484] command sessions.session_common command: findAndModify { findAndModify: "session_common", query:

      Unknown macro: { sessionid}

      , update: { $inc:

      Unknown macro: { maxmsgid}

      , $set:

      Unknown macro: { activetime}

      }, new: true, $db: "sessions" } planSummary: IXSCAN

      Unknown macro: { sessionid}

      keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keysInserted:1 keysDeleted:1 numYields:0 reslen:417 locks:{ Global: { acquireCount:

      Unknown macro: { r}

      }, Database: { acquireCount:

      Unknown macro: { w}

      }, Collection: { acquireCount:

      Unknown macro: { w}

      }, oplog: { acquireCount:

      Unknown macro: { w}

      } } protocol:op_query 120ms 2018-04-09T16:13:33.876+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:876714][3376:1995256704], log-server: D:\mongodb\data\journal\WiredTigerTmplog.0000001761: handle-set-end: SetEndOfFile: 磁盘空间不足。

      : No space left on device 2018-04-09T16:13:33.876+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:876714][3376:1995256704], log-server: log pre-alloc server error: No space left on device 2018-04-09T16:13:33.877+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:877691][3376:1995256704], log-server: log server error: No space left on device 2018-04-09T16:13:33.877+0800 E STORAGE [thread3841] WiredTiger error (-31804) [1523261613:877691][3376:1995256704], log-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-04-09T16:13:33.886+0800 F - [conn486] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn475] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn483] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn478] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [WTOplogJournalThread] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn487] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn3774] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [thread3841] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361 2018-04-09T16:13:33.886+0800 F - [conn470] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn486]

      ***aborting after fassert() failure

      2018-04-09T16:13:33.886+0800 F - [conn466] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn475]

      ***aborting after fassert() failure

      2018-04-09T16:13:33.886+0800 F - [conn483]

      ***aborting after fassert() failure

      When I restarted the mongod the disk space was free.
      It seems that journal files fill in the whole disk space. But my real dataset is not very big.
      This is the database status in MongoDB:

      Unknown macro: { "db" }
      Unknown macro: { "db" }
      Unknown macro: { "db" }

      I notice that the number of WiredTigerPreplog file in journal directory always grows and doesn't stop. The total size of these files is even the majority of MongoDB data size.
      This is the screenshot on the machine. Note number of WiredTigerPreplog file:
      screenshot

      Attachments

        Issue Links

          Activity

            People

              backlog-server-execution Backlog - Storage Execution Team
              jinzhao jinwuzhao
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: