[SERVER-33187] Journal data are not cleared by WiredTiger Created: 08/Feb/18  Updated: 06/Dec/22  Resolved: 09/Feb/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sandeep [X] Assignee: Backlog - Storage Execution Team
Resolution: Duplicate Votes: 0
Labels: Bug, Cleanup
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoServer:3.4 version
OS:Windows 7 Professional ,Version 6.1
Processor:Intel(r) Core(TM) i5-4430 CPU @3.00GHz 3.00GHz
RAM:8.00GB
Systemtype:64-bit OS


Attachments: File _mdb_catalog.wt     File diagnostic.data.rar     File ftdc.tar     Text File journal_dir_listfiles.txt     PNG File logfiles.png     File mongod-logs.7z     File sizeStorer.wt    
Issue Links:
Related
related to WT-3904 Reconsider error path in log server t... Closed
Assigned Teams:
Storage Execution
Participants:

 Description   

Hi Team,

Thanks for your continuous support !!
*We have a production issue as in , the journal data is not cleared till mongoServer is restarted!! This folder has reached approximately 75GB!!!!
and occupied almost entire Drive !!!!
Query:
*
We would like to here from you that*_,
If MongoDB Server is never restarted , will the binary log data under data/db/journal will not be cleared ?
What is the Max number of journal files or max sixe of journal folder allowed to be created under Journal folder ?
[We undertood from journal file sixe is 100MB]
_*

Reference links
We have gone through the site:*https://docs.mongodb.com/manual/core/journaling/*

It has mentioned that
+"A clean shutdown removes all the files in the journal directory. A dirty shutdown (crash) leaves files in the journal directory;
these are used to automatically recover the database to a consistent state when the mongod process is restarted"+

Above statement implies that journal folder will keep on growing and never gets cleared ,
until ,

  • Mongo server is not restarted or
  • One has to perform clean shutdown
    Please Clarify and address the above mentioned query

Thanks & Regards
Sandeep KS



 Comments   
Comment by Eric Milkie [ 09/Feb/18 ]

Please follow the linked WT ticket for updates.

Comment by Sandeep [X] [ 09/Feb/18 ]

Dear Team,

I've attached file list of journal directory journal_dir_listfiles.txt to further aid your investigation.

Thanks & Regards
Sandeep KS

Comment by Susan LoVerso [ 08/Feb/18 ]

I opened WT-3904 to discuss possible future changes to how internal threads handle errors.

Comment by Susan LoVerso [ 08/Feb/18 ]

There are no known issues that I'm aware of (at least looking back on changes/fixes to relevant files). Archiving log files no longer needed is determined by checkpoints. Looking at FTDC checkpoints are happening regularly and nothing looks out of the ordinary there. bruce.lucas's analysis provides a clue.

Looking at the mongod logs, the log server thread that is responsible for both pre-allocating log files and archiving old ones got an error and exited:

2018-02-01T20:41:07.083-0500 E STORAGE  [thread2] WiredTiger error (0) [1517535667:83051][4752:140716872831104], log-server: C:\SRDM\SRDM2.5.0.11094.31\SRDM\Svr\DBService\bin64\..\data\db\journal/WiredTigerLog.0000000023: file-remove: DeleteFileW: Access is denied
MongoDbLog-2018-02-01_1132 AM.log:2018-02-01T20:41:07.083-0500 E STORAGE  [thread2] WiredTiger error (13) [1517535667:83051][4752:140716872831104], log-server: log server error: Permission denied

Therefore once that happened and the thread exited with an error, it never archived nor pre-allocated log files. I'll open a ticket to consider whether that should be a fatal error, but in 3.4.0 it is and the only way to restart that internal thread is a restart.

Comment by Susan LoVerso [ 08/Feb/18 ]

Can you attach an ls -l of the journal directory please?

Comment by Bruce Lucas (Inactive) [ 08/Feb/18 ]

It looks like up until about A we are recycling old log files, but at A we stop doing that and so starting at B we are having to create new log files. This condition persists throughout the rest of the FTDC data, and as expected the number of "pre-allocated log files not ready" matches the number of existing log files. There are one or two other things that change at about the same time, relevance TBD.

sue.loverso does this match any known issued in 3.4.0?

Comment by Sandeep [X] [ 08/Feb/18 ]

Sorry Team,
By mistake i mentioned Affected Versions as :3.4.12, But the MongoDB Server Version we are using is :3.4.0

Comment by Sandeep [X] [ 08/Feb/18 ]

Dear Team,
Further to aid your investigation we are attaching below data logs also

  1. _mdb_catalog.wt
  2. sizeStorer.wt
  3. mongod.logs
  4. diagnostic.data [already shared above]

Thanks & Regards
Sandeep KS

Comment by Sandeep [X] [ 08/Feb/18 ]

diagnostic.data.rar
Hi Team,

Also i have attached the* "$dbpath/diagnostic.data* for your analysis
There are around *686 files under <<data\db\journal>> folder *
*Files=> WiredTigerLog.0000000024 -to -WiredTigerLog.0000000709*
Attachment Name::<<diagnostic.data.rar>>

Thanks & awaiting for your reply

Thanks & Regards
Sandeep KS

Generated at Thu Feb 08 04:32:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.