Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52596

Detect memLimitMB in K8S pod and inform WiredTigerCacheSize calculation for pod RAM instead of total system RAM

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9.0, 4.4.4
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v4.4
    • Sprint:
      Execution Team 2020-12-28, Execution Team 2021-01-11

      Description

      I am deploying mongo db 4.4.1 (windows servercore 1809 image) container on K8S. No operators are used, using the community edition mongo db container.

      The memory limit on the ‘pod’ is set at 300M. However the container start fail with out of memory exception. Looking deeper in the logs, it turns out that wired tiger is trying to build a cache of size ~3GB which is resulting in the exception.

      relevant log lines -

      _{“t”:\{"$date":“2020-11-02T12:49:55.088+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.066+00:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.067+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648602, “ctx”:“main”,“msg”:“Implicit TCP FastOpen in use.”}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:\{“pid”:6496,“port”:27017,“dbPath”:“C:/PluginDataFolder”,“architecture”:“64-bit”,“host”:“documents-0”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23398, “ctx”:“initandlisten”,“msg”:“Target operating system minimum version”,“attr”:\{“targetMinOS”:“Windows 7/Windows Server 2008 R2”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:\{“buildInfo”:{“version”:“4.4.1”,“gitVersion”:“ad91a93a5a31e175f5cbf8c69561e788bbc55ce1”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distmod”:“windows”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:\{“os”:{“name”:“Microsoft Windows Server 2019”,“version”:“10.0 (build 17763)”}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.076+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:\{“options”:{“net”:{“bindIp”:"*"},“security”:\{“authorization”:“enabled”},“storage”:\{“dbPath”:“C:\PluginDataFolder”}}}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.089+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,*+{color:#de350b}cache_size=3071M{color}+*,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}_
      _{“t”:\{"$date":“2020-11-02T12:50:01.093+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:\{“error”:12,“message”:"[1604321401:93157][6496:140717780783712], wiredtiger_open: __wt_calloc, 52: memory allocation of 32237280 bytes failed: Not enough space"}}_
      

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              henrik.edin Henrik Edin
              Reporter:
              mail2h.jain@gmail.com Hemant Jain
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: