Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17495

Stand alone mongod throughout dropped by 40% after 17 hours of insert only workload

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Incomplete
    • Affects Version/s: 3.0.0-rc11, 3.1.6
    • Fix Version/s: None
    • Component/s: Storage, WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      Windows
    • Sprint:
      Platform 4 06/05/15, Platform 5 06/26/16, Platform 6 07/17/15, Platform 7 08/10/15

      Description

      Version –
      Mongod RC11binaries

      Environment:

      • Single mongod with wiredtiger as storage engine
      • Windows 2012
      • WriedTiger configured with 1GB cache
      • EC2 machine c3.large

      Workload:

      • Used hammer.mongo to do insert only workload
      • The machine throughput dropped by 40% after 17 hours of execution

      Will continue and investigate this issue

      At this stage

      • This is the 2nd repro of the same problem, when I analyzed the dump, I found a huge internal heap fragmentation (see more details on the comment)
      • No indication that client stress tool slow down (I observed steady amount of incoming TCP connection per/sec and TCP connection established remained stable)

      Next plan – Keep it running to death and debug it when the throughput is significant low.

        Attachments

        1. 317profiler.png
          317profiler.png
          58 kB
        2. mongodb_08172211.csv
          3.35 MB
        3. perf_test_data.csv
          550 kB
        4. Process_IO_Data_Operations_sec0.png
          Process_IO_Data_Operations_sec0.png
          63 kB
        5. Process_IO_Write_Operations_sec0.png
          Process_IO_Write_Operations_sec0.png
          54 kB
        6. Processor_Information_Percent_Processor_Time0.png
          Processor_Information_Percent_Processor_Time0.png
          53 kB
        7. profile-output.jpg
          profile-output.jpg
          143 kB
        8. Re httpsjira.mongodb.orgbrowseSERVER-17495.htm
          141 kB
        9. repro-317.png
          repro-317.png
          94 kB
        10. repro-317.png
          repro-317.png
          68 kB
        11. ServerStats.txt
          11 kB
        12. stat.txt
          11 kB

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: