[SERVER-52596] Detect memLimitMB in K8S pod and inform WiredTigerCacheSize calculation for pod RAM instead of total system RAM Created: 03/Nov/20  Updated: 29/Oct/23  Resolved: 05/Jan/21

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: 4.9.0, 4.4.4

Type: Improvement Priority: Major - P3
Reporter: Hemant Jain Assignee: Henrik Edin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-16571 Use Actual Memory Constraint vs. Tota... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: Execution Team 2020-12-28, Execution Team 2021-01-11
Participants:

 Description   

I am deploying mongo db 4.4.1 (windows servercore 1809 image) container on K8S. No operators are used, using the community edition mongo db container.

The memory limit on the ‘pod’ is set at 300M. However the container start fail with out of memory exception. Looking deeper in the logs, it turns out that wired tiger is trying to build a cache of size ~3GB which is resulting in the exception.

relevant log lines -

_{“t”:\{"$date":“2020-11-02T12:49:55.088+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}_
_{“t”:\{"$date":“2020-11-02T12:50:01.066+00:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}_
_{“t”:\{"$date":“2020-11-02T12:50:01.067+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648602, “ctx”:“main”,“msg”:“Implicit TCP FastOpen in use.”}_
_{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:\{“pid”:6496,“port”:27017,“dbPath”:“C:/PluginDataFolder”,“architecture”:“64-bit”,“host”:“documents-0”}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23398, “ctx”:“initandlisten”,“msg”:“Target operating system minimum version”,“attr”:\{“targetMinOS”:“Windows 7/Windows Server 2008 R2”}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:\{“buildInfo”:{“version”:“4.4.1”,“gitVersion”:“ad91a93a5a31e175f5cbf8c69561e788bbc55ce1”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distmod”:“windows”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:\{“os”:{“name”:“Microsoft Windows Server 2019”,“version”:“10.0 (build 17763)”}}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:\{“options”:{“net”:{“bindIp”:"*"},“security”:\{“authorization”:“enabled”},“storage”:\{“dbPath”:“C:\PluginDataFolder”}}}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.089+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,*+{color:#de350b}cache_size=3071M{color}+*,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}_
_{“t”:\{"$date":“2020-11-02T12:50:01.093+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:\{“error”:12,“message”:"[1604321401:93157][6496:140717780783712], wiredtiger_open: __wt_calloc, 52: memory allocation of 32237280 bytes failed: Not enough space"}}_

 



 Comments   
Comment by Githook User [ 12/Jan/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-52596 Set memory and CPU limits when running in Windows Container using process isolation

(cherry picked from commit d2e4a1601475f7c7f72d11cfe1b554963214a0e5)
Branch: v4.4
https://github.com/mongodb/mongo/commit/917e3e22cf21d71b1fb2a6649577c92a1beda78f

Comment by Ian Whalen (Inactive) [ 07/Jan/21 ]

Author:

{'username': u'evrg-bot-webhook', 'name': u'Henrik Edin', 'email': u'henrik.edin@mongodb.com'}

Message:SERVER-52596 Set memory and CPU limits when running in Windows Container using process isolation
Branch:master
https://github.com/mongodb/mongo/commit/d2e4a1601475f7c7f72d11cfe1b554963214a0e5

Comment by Henrik Edin [ 05/Jan/21 ]

https://github.com/mongodb/mongo/commit/d2e4a1601475f7c7f72d11cfe1b554963214a0e5

Comment by Eric Sedor [ 05/Nov/20 ]

Thanks mail2h.jain@gmail.com, I'll pass this on to an appropriate team for consideration as a possible improvement.

In case you have not already done so, you'll want to configure the storage.wiredTiger.engineConfig.cacheSizeGB setting manually using this guidance in our Production Notes. Some discussion about containers is included at the end of the WiredTiger section:

If you run mongod in a container (e.g. lxc, cgroups, Docker, etc.) that does not have access to all of the RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container. See memLimitMB.

Generated at Thu Feb 08 05:28:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.