Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32701

Mongod with MMAPV1 SE does not use all free memory and cause disk IO

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 3.4.10
    • Fix Version/s: None
    • Component/s: MMAPv1, Performance
    • Labels:
      None
    • Operating System:
      ALL

      Description

      MongoDB server version: 3.4.10
      OS: Debian 4.9.65-3+deb9u1~bpo8+1 (2017-12-23) x86_64 GNU/Linux
      RAM: 32GB
      CPU: 12 cores
      Database size: 61GB (no replication, no sharding)
      Collection records: 32 Millions (only one collection in the database)

      On server peak hours something hinky happens. Some queries on the database takes from 10s to 50s! When I get htop, all of a sudden can see that mongod has used ONLY 40% of server RAM and about 14GB of memory is free, cpu usage is used about 10% to 20%! When I see those queries I can see that it has used `IXSCAN` as its winning plan and fetched like 4000 records which belongs to the user, which is normal.

      Why `mongod` has not used all of the memory and causes disk IO? It has been mentioned in the documentation that mongod will consume all the memory on the server!

      I should note that I have benchmarked disk IO using `hdparm` and result was like below:

      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17388 MB in  2.00 seconds = 8706.52 MB/sec
       Timing buffered disk reads: 2722 MB in  3.00 seconds = 907.05 MB/sec
       
      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17022 MB in  2.00 seconds = 8523.85 MB/sec
       Timing buffered disk reads: 2054 MB in  3.01 seconds = 683.17 MB/sec
       
      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17732 MB in  2.00 seconds = 8879.00 MB/sec
       Timing buffered disk reads: 3298 MB in  3.00 seconds = 1099.23 MB/sec
      
      

      This misbehaviour seems as a bug not normal behaviour.

        Attachments

        1. diagnostic_data.tar.gz
          133.81 MB
        2. diagnostic.data.zip
          12.20 MB
        3. mongod_logs.tar.gz
          630 kB

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: