[SERVER-40488] MongoDB WiredTigerLAS.wt file growth unexpected Created: 05/Apr/19  Updated: 25/Apr/19  Resolved: 25/Apr/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Yang Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

 
Hi guys
We have a MongoDB cluster with 1 primary server and 2 secondary servers.
All three servers are using using MongoDB version 3.6.11.

Recently, we found both secondary servers down and it is caused by the space being full. We found the file WiredTigerLAS.wt has grown very big to over 20GB. The whole mongodb data folder is supposed to be below 4GB. We tried to remove the WiredTigerLAS.wt and restarted the secondary servers, but the WiredTigerLAS.wt got created and started growing to be full of the disk again. The primary server has been OK, no impact is found.

Can someone please help and advice what we shall do now? If you know the reason behind the unexpected file growth, please let us know.

rs.conf:

// code placeholder
rs0:PRIMARY> rs.conf()
{
    "_id" : "rs0",
    "version" : 7,
    "protocolVersion" : NumberLong(1),
    "members" : [
        {
            "_id" : 0,
            "host" : "primary:50001",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {
 
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        },
        {
            "_id" : 1,
            "host" : "replica1:50001",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 0,
            "tags" : {
 
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 0
        },
        {
            "_id" : 2,
            "host" : "replica2:50001",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 0,
            "tags" : {
 
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 0
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatIntervalMillis" : 2000,
        "heartbeatTimeoutSecs" : 10,
        "electionTimeoutMillis" : 10000,
        "catchUpTimeoutMillis" : -1,
        "catchUpTakeoverDelayMillis" : 30000,
        "getLastErrorModes" : {
 
        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        },
        "replicaSetId" : ObjectId("5b52ac682b4bd7ae7913b1cf")
    }
}



 Comments   
Comment by Danny Hatcher (Inactive) [ 15/Apr/19 ]

The use of the WiredTigerLAS.wt file is working as designed though we may add some optimizations in the future. The way to prevent growth in the file is to work on the underlying issues that triggered the situation which may or may not have been resolved in an updated release. If you upload the "diagnostic.data" folder (located under the $dbpath) for every node in the replica set to our Secure Upload Portal I can take a look.

Thanks,

Danny

Comment by Yang [ 14/Apr/19 ]

Hello Danny,  we have removed these files as to keep the cluster running again.  I notice there is some recent upgrade on the version, is it a fix to this? 

Comment by Danny Hatcher (Inactive) [ 05/Apr/19 ]

Hello,

The WiredTigerLAS.wt file is used to store information when the WiredTiger cache is getting full. Typically a growing WiredTigerLAS.wt file indicates that the amount of memory allocated to the system is not sufficient to sustain the workload being performed. However, there are possibly some other things that can cause the effect. Could you please upload the "diagnostic.data" folder (located under the $dbpath) for every node in the replica set to our Secure Upload Portal? The files uploaded to that location will only be visible to MongoDB Engineers and will be automatically deleted after 90 days.

Thank you,

Danny

Generated at Thu Feb 08 04:55:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.