Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17495

Stand alone mongod throughout dropped by 40% after 17 hours of insert only workload

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.0.0-rc11, 3.1.6
    • Component/s: Storage, WiredTiger
    • Labels:
    • Fully Compatible
    • Windows
    • Platform 4 06/05/15, Platform 5 06/26/16, Platform 6 07/17/15, Platform 7 08/10/15

      Version –
      Mongod RC11binaries

      Environment:

      • Single mongod with wiredtiger as storage engine
      • Windows 2012
      • WriedTiger configured with 1GB cache
      • EC2 machine c3.large

      Workload:

      • Used hammer.mongo to do insert only workload
      • The machine throughput dropped by 40% after 17 hours of execution

      Will continue and investigate this issue

      At this stage

      • This is the 2nd repro of the same problem, when I analyzed the dump, I found a huge internal heap fragmentation (see more details on the comment)
      • No indication that client stress tool slow down (I observed steady amount of incoming TCP connection per/sec and TCP connection established remained stable)

      Next plan – Keep it running to death and debug it when the throughput is significant low.

        1. 317profiler.png
          317profiler.png
          58 kB
        2. mongodb_08172211.csv
          3.35 MB
        3. perf_test_data.csv
          550 kB
        4. Process_IO_Data_Operations_sec0.png
          Process_IO_Data_Operations_sec0.png
          63 kB
        5. Process_IO_Write_Operations_sec0.png
          Process_IO_Write_Operations_sec0.png
          54 kB
        6. Processor_Information_Percent_Processor_Time0.png
          Processor_Information_Percent_Processor_Time0.png
          53 kB
        7. profile-output.jpg
          profile-output.jpg
          143 kB
        8. Re httpsjira.mongodb.orgbrowseSERVER-17495.htm
          141 kB
        9. repro-317.png
          repro-317.png
          94 kB
        10. repro-317.png
          repro-317.png
          68 kB
        11. ServerStats.txt
          11 kB
        12. stat.txt
          11 kB

            Assignee:
            backlog-server-platform DO NOT USE - Backlog - Platform Team
            Reporter:
            eitan.klein Eitan Klein
            Votes:
            1 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: