Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52596

Detect memLimitMB in K8S pod and inform WiredTigerCacheSize calculation for pod RAM instead of total system RAM

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.9.0, 4.4.4
    • Affects Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Fully Compatible
    • v4.4
    • Execution Team 2020-12-28, Execution Team 2021-01-11

      I am deploying mongo db 4.4.1 (windows servercore 1809 image) container on K8S. No operators are used, using the community edition mongo db container.

      The memory limit on the ‘pod’ is set at 300M. However the container start fail with out of memory exception. Looking deeper in the logs, it turns out that wired tiger is trying to build a cache of size ~3GB which is resulting in the exception.

      relevant log lines -

      _{“t”:\{"$date":“2020-11-02T12:49:55.088+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.066+00:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.067+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648602, “ctx”:“main”,“msg”:“Implicit TCP FastOpen in use.”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:\{“pid”:6496,“port”:27017,“dbPath”:“C:/PluginDataFolder”,“architecture”:“64-bit”,“host”:“documents-0”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23398, “ctx”:“initandlisten”,“msg”:“Target operating system minimum version”,“attr”:\{“targetMinOS”:“Windows 7/Windows Server 2008 R2”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:\{“buildInfo”:{“version”:“4.4.1”,“gitVersion”:“ad91a93a5a31e175f5cbf8c69561e788bbc55ce1”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distmod”:“windows”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:\{“os”:{“name”:“Microsoft Windows Server 2019”,“version”:“10.0 (build 17763)”}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:\{“options”:{“net”:{“bindIp”:"*"},“security”:\{“authorization”:“enabled”},“storage”:\{“dbPath”:“C:\PluginDataFolder”}}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.089+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,*+{color:#de350b}cache_size=3071M{color}+*,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.093+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:\{“error”:12,“message”:"[1604321401:93157][6496:140717780783712], wiredtiger_open: __wt_calloc, 52: memory allocation of 32237280 bytes failed: Not enough space"}}_
      

       

            Assignee:
            henrik.edin@mongodb.com Henrik Edin
            Reporter:
            mail2h.jain@gmail.com Hemant Jain
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: