Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32701

Mongod with MMAPV1 SE does not use all free memory and cause disk IO

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.10
    • Component/s: MMAPv1, Performance
    • Labels:
      None
    • ALL

      MongoDB server version: 3.4.10
      OS: Debian 4.9.65-3+deb9u1~bpo8+1 (2017-12-23) x86_64 GNU/Linux
      RAM: 32GB
      CPU: 12 cores
      Database size: 61GB (no replication, no sharding)
      Collection records: 32 Millions (only one collection in the database)

      On server peak hours something hinky happens. Some queries on the database takes from 10s to 50s! When I get htop, all of a sudden can see that mongod has used ONLY 40% of server RAM and about 14GB of memory is free, cpu usage is used about 10% to 20%! When I see those queries I can see that it has used `IXSCAN` as its winning plan and fetched like 4000 records which belongs to the user, which is normal.

      Why `mongod` has not used all of the memory and causes disk IO? It has been mentioned in the documentation that mongod will consume all the memory on the server!

      I should note that I have benchmarked disk IO using `hdparm` and result was like below:

      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17388 MB in  2.00 seconds = 8706.52 MB/sec
       Timing buffered disk reads: 2722 MB in  3.00 seconds = 907.05 MB/sec
       
      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17022 MB in  2.00 seconds = 8523.85 MB/sec
       Timing buffered disk reads: 2054 MB in  3.01 seconds = 683.17 MB/sec
       
      root@hs1.example.com:~# hdparm -tT /dev/mapper/BDRS--CH01--001--vg-VAR
      /dev/mapper/BDRS--CH01--001--vg-VAR:
       Timing cached reads:   17732 MB in  2.00 seconds = 8879.00 MB/sec
       Timing buffered disk reads: 3298 MB in  3.00 seconds = 1099.23 MB/sec
      
      

      This misbehaviour seems as a bug not normal behaviour.

        1. diagnostic_data.tar.gz
          133.81 MB
        2. diagnostic.data.zip
          12.20 MB
        3. mongod_logs.tar.gz
          630 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            alireza_stack Ali
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: