Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34403

The journal files grow out of disk size

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.3
    • Component/s: None
    • Labels:
    • Environment:
      OS: Windows Server 2008 R2 Enterprise with Service Pack 1 (64 bit)
      CPU: Intel Xeon CPU E5-26xx v2 2.6GHz
      RAM: 4 GB
      disk: 100 GB disk partition for MongoDB.
      MongoDB Replica Set: 3 instances, this instance is the Primary.
    • Storage Execution

      Mongod crashed because of no space left on disk. This is the log when crash happened:

      2018-04-09T16:13:33.568+0800 I COMMAND [conn484] command sessions.session_common command: findAndModify { findAndModify: "session_common", query:

      Unknown macro: { sessionid}

      , update: { $inc:

      Unknown macro: { maxmsgid}

      , $set:

      Unknown macro: { activetime}

      }, new: true, $db: "sessions" } planSummary: IXSCAN

      Unknown macro: { sessionid}

      keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keysInserted:1 keysDeleted:1 numYields:0 reslen:417 locks:{ Global: { acquireCount:

      Unknown macro: { r}

      }, Database: { acquireCount:

      Unknown macro: { w}

      }, Collection: { acquireCount:

      Unknown macro: { w}

      }, oplog: { acquireCount:

      Unknown macro: { w}

      } } protocol:op_query 120ms 2018-04-09T16:13:33.876+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:876714][3376:1995256704], log-server: D:\mongodb\data\journal\WiredTigerTmplog.0000001761: handle-set-end: SetEndOfFile: 磁盘空间不足。

      : No space left on device 2018-04-09T16:13:33.876+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:876714][3376:1995256704], log-server: log pre-alloc server error: No space left on device 2018-04-09T16:13:33.877+0800 E STORAGE [thread3841] WiredTiger error (28) [1523261613:877691][3376:1995256704], log-server: log server error: No space left on device 2018-04-09T16:13:33.877+0800 E STORAGE [thread3841] WiredTiger error (-31804) [1523261613:877691][3376:1995256704], log-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-04-09T16:13:33.886+0800 F - [conn486] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn475] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn483] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn478] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [WTOplogJournalThread] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn487] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn3774] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [thread3841] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361 2018-04-09T16:13:33.886+0800 F - [conn470] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn486]

      ***aborting after fassert() failure

      2018-04-09T16:13:33.886+0800 F - [conn466] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64 2018-04-09T16:13:33.886+0800 F - [conn475]

      ***aborting after fassert() failure

      2018-04-09T16:13:33.886+0800 F - [conn483]

      ***aborting after fassert() failure

      When I restarted the mongod the disk space was free.
      It seems that journal files fill in the whole disk space. But my real dataset is not very big.
      This is the database status in MongoDB:

      Unknown macro: { "db" }
      Unknown macro: { "db" }
      Unknown macro: { "db" }

      I notice that the number of WiredTigerPreplog file in journal directory always grows and doesn't stop. The total size of these files is even the majority of MongoDB data size.
      This is the screenshot on the machine. Note number of WiredTigerPreplog file:
      screenshot

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            jinzhao jinwuzhao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: