[SERVER-20426] mongoperf is producing questionable results Created: 15/Sep/15  Updated: 06/Dec/22  Resolved: 15/Mar/16

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Steven Hand Assignee: Backlog - Performance Team
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File C_fio-test.fio     Text File C_fio_32T_rand_rw.txt     PNG File C_vs_U_32_T_IOps.png     File CentOS_32_data.csv     File U_fio-test.fio     Text File U_fio_32T_rand_rw.txt     File Ubuntu_32_data.csv    
Issue Links:
Related
related to TOOLS-979 Extend mongoperf to simulate a WiredT... Closed
is related to SERVER-18973 Rename mongoperf to something more us... Closed
is related to SERVER-18907 mongoperf behavior is confusing Closed
Assigned Teams:
Product Performance
Operating System: ALL
Steps To Reproduce:

1) Setup CentOS 6.5 and Ubuntu 14.04 VMs on Azure
2) Install MongoDB 3.0.6 on both VMs
3) run mongoperf tests
4) run fio test
5) compare results

Participants:

 Description   

I found a problem with the output with mongperf. In comparing the performance of CentOS 6.5 and Ubuntu 14.04 on Azure, I discovered that mongoperf's results were questionable. With 32 threads, mongperf was reporting that Ubuntu 14.04 was handling 120X mote IOps than CentOS 6.5. However, a like test using FIO does not reach the same results.



 Comments   
Comment by Ian Whalen (Inactive) [ 15/Mar/16 ]

Apologies for the delay in responding on this issue, but we've decided not to proceed with any further enhancements to mongoperf in its current form - as such we're closing this as Won't Fix. We aren't moving the existing code or removing the binary, but any changes going forward would almost certainly involve first moving the existing functionality to a newly written tool. We made a similar decision with our other tools in 3.0.0 and have been very happy with the results.

We don't have a specific timeline for this work, but you can vote on and add yourself as a watcher to TOOLS-1103 to be informed of updates as we weigh its priority against other projects that our Tools team are working on.

Comment by Ramon Fernandez Marina [ 22/Sep/15 ]

steven.hand, the future of mongoperf is still being debated internally, so I'm going to put this ticket in "Needs Further Definition" so we can discuss what to do.

Comment by Steven Hand [ 15/Sep/15 ]

I have done some testing using two D2 VMs

  1. Ubuntu 14.04
    • FIO 2.1.3
    • two block devices
    • MongoDB 3.0.6
  2. CentOS 6.6
    • FIO 2.1.4 (v2.1.3 is not available independently of distro)
    • two block devices
    • MongoDB 3.0.6

The output of running mongperf with 32 threads,

echo "{ nThreads: 32, fileSizeMB: 2, sleepMicros: 0, mmf: false, r: true, w:false, syncDelay: 0 }" | mongoperf

on both OSes produced the following diagram

As you can see from the graph, the IOps performance for Ubuntu reported by mongoperf was much higher than CentOS. The MB/s was much higher as well. See Ubuntu_32_data.csv and CentOS_32_data.csv.

FIO is a widely used block and file IO testing application. The difference between the two OSes was too dramatic to simply accept. I don't know what IO pattern that mongoperf imposes on the disks. I assumed 4K block size random read and write, but collected results for sequential IO as well. I didn't know what queue depth is used by mongoperf. But in any case, after several test runs with FIO, with various parameters, the difference in IO performance between these two OSes were not nearly as large as mongoperf indicated. In fact, FIO reported that CentOS performs slightly better than Ubuntu. See enclosed FIO results, C_fio_32T_rand_rw.txt and U_fio_32T_rand_rw.txt, and fio parameter files, U_fio-test.fio and C_fio-test.fio.

I also tested document insert performance using MongoDB 3.0.6 on the two OSes and captured 7212 inserts per second on Ubuntu and 7618 on CentOS, again indicating a slight performance advantage with CentOS.

In summary, the performance results that mongoperf reports are not in line with other tools and appear to be suspect.

Generated at Thu Feb 08 03:54:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.