[SERVER-27903] Out of memory (mogodb) with WiredTiger Created: 03/Feb/17  Updated: 21/Feb/17  Resolved: 21/Feb/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: kangwanu Assignee: Mark Agarunov
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2017-02-06 at 5.30.40 PM.png    
Operating System: ALL
Steps To Reproduce:

==================================

engineConfig:
            cacheSizeGB: 6
            journalCompressor: zlib
        collectionConfig:
            blockCompressor: zlib
        indexConfig:
            prefixCompression: true

========================

{
        "tracked dirty bytes in the cache" : 747865945,
        "tracked bytes belonging to internal pages in the cache" : 57440243,
        "bytes currently in the cache" : 5153539327,
        "tracked bytes belonging to leaf pages in the cache" : 5096099084,
        "maximum bytes configured" : 6442450944,
        "tracked bytes belonging to overflow pages in the cache" : 0,
        "bytes read into cache" : 156686381017,
        "bytes written from cache" : 188905355048,
        "pages evicted by application threads" : 0,
        "checkpoint blocked page eviction" : 1013,
        "unmodified pages evicted" : 144545,
        "page split during eviction deepened the tree" : 0,
        "modified pages evicted" : 12889650,
        "pages selected for eviction unable to be evicted" : 334417,
        "pages evicted because they exceeded the in-memory maximum" : 0,
        "pages evicted because they had chains of deleted items" : 136,
        "failed eviction of pages that exceeded the in-memory maximum" : 69,
        "hazard pointer blocked page eviction" : 18872,
        "internal pages evicted" : 231729,
        "maximum page size at eviction" : 8388808,
        "eviction server candidate queue empty when topping up" : 59348,
        "eviction server candidate queue not empty when topping up" : 610688,
        "eviction server evicting pages" : 0,
        "eviction server populating queue, but not evicting pages" : 670035,
        "eviction server unable to reach eviction goal" : 0,
        "internal pages split during eviction" : 1,
        "leaf pages split during eviction" : 50675,
        "pages walked for eviction" : 20848176968,
        "eviction worker thread evicting pages" : 13368457,
        "in-memory page splits" : 126,
        "in-memory page passed criteria to be split" : 333,
        "lookaside table insert calls" : 0,
        "lookaside table remove calls" : 0,
        "percentage overhead" : 8,
        "tracked dirty pages in the cache" : 15513,
        "pages currently held in the cache" : 120186,
        "pages read into cache" : 13154136,
        "pages read into cache requiring lookaside entries" : 0,
        "pages written from cache" : 15897182,
        "page written requiring lookaside records" : 0,
        "pages written requiring in-memory restoration" : 0
}

Participants:

 Description   

hello
I have problem about WiredTiger , I found the Out of memory on Linux environment (15 GB memory), and Process was killed by OS. Please help me to investigate this case



 Comments   
Comment by kangwanu [ 21/Feb/17 ]

Hello Mark ,
I upgrade to MongoDB 3.4.2,It resolved problem

thanks so much sir.
kangwanu

Comment by Mark Agarunov [ 07/Feb/17 ]

Hello kangwanu,

The recommended version of MongoDB would be the latest, which at this point in time is version 3.4.2. Please note that once you upgrade to MongoDB 3.4.2, you will only be able to downgrade to 3.2.8 or later.

Thanks,
Mark

Comment by kangwanu [ 07/Feb/17 ]

Hello Mark,
Right now, I use mongoDB version 3.2.0, And I have plan to change mongoDB to new version but I'm not sore
Which version for your recommend 3.2.12, 3.4.2 or other?
BTW, I will send mongod log to you soon

Thanks,
kangwanu

Comment by Mark Agarunov [ 06/Feb/17 ]

Hello kangwanu,

Looking over the diagnostic data, There appears to be an operation starting just as the cache size starts to quickly increase. In the attached screenshot, this starts at points B and C.

To better determine the cause of this issue, please upload the full mongod logs.

Please note that the version on mongod being used is quite outdated, and there have been a number of critical bug fixes since its release. Additionally, we have spent significant effort improving WiredTiger performance. These fixes and improvements may correct this issue in more recent versions. Would you please upgrade to 3.2.12, the latest version point release, and let us know if the issue is resolved?

Thanks,
Mark

Comment by kangwanu [ 04/Feb/17 ]

Hi Thomas ,
I already uploaded the diagnostic data directory. Please kindly help verify and this case happened again in today as "var/log/message" as below.

Feb 4 12:43:29 kernel: Out of memory: Kill process 22421 (mongod) score 958 or sacrifice child

Thank you,
kangwanu

Comment by Kelsey Schubert [ 04/Feb/17 ]

Hi kangwanu,

Please note that the diagnostic.data directory does not contain any user data. It periodically collects the output of the following commands, which you are welcome to execute yourself to examine the output.

serverStatus: db.serverStatus({tcmalloc: true})
replSetGetStatus: rs.status()
collStats for local.oplog.rs: db.getSiblingDB('local').oplog.rs.stats()
getCmdLineOpts: db.adminCommand({getCmdLineOpts: true})
buildInfo: db.adminCommand({buildInfo: true})
hostInfo: db.adminCommand({hostInfo: true})

Additionally, you can examine the source code to confirm. However, since you have expressed some concern, I've gone ahead and created a secure upload portal for you to upload the logs and diagnostic.data. Files uploaded to this portal are only visible to MongoDB employees and are routinely deleted after some time.

Thank you,
Thomas

Comment by kangwanu [ 04/Feb/17 ]

Hi Thomas ,
Thanks for adding the comment.
The files in diagnostic.data directory look like binary ?
How upload an archive? ,I don't want to unknowingly share any secret data.

thanks,
kangwanu

Comment by Kelsey Schubert [ 03/Feb/17 ]

Hi kangwanu,

Thank you for reporting this issue. So we can continue to investigate, would you please upload an archive of the diagnostic.data directory and the complete logs for the affected mongod?

Thanks,
Thomas

Generated at Thu Feb 08 04:16:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.