[SERVER-10503] High iowait on secondary but don't see many writes in MongoDB web interface Created: 14/Aug/13 Updated: 10/Dec/14 Resolved: 19/Aug/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.4.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Nic Cottrell (Personal) | Assignee: | Stennie Steneker (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
rhel6, 40GB ram, RAID5 |
||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
This server is the secondary of a replica set which is the primary of a sharding cluster. There is iowait >10% constantly and I see writes of about 1.5Mb/s with iotop. In the mongo web interface I don't see many writes - in fact one lists the collection as blank. Any idea why the iowait is so high? Or a way to track down which collection is causing the writes? |
| Comments |
| Comment by Stennie Steneker (Inactive) [ 19/Aug/13 ] |
|
Hi Nic, The SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-users group (http://groups.google.com/group/mongodb-user) or Stack Overflow / ServerFault. In terms of looking at current activity on your server, in addition to mongostat (which you are already using) you could use db.currentOp() to report in-progress operations. The currentOp output is the top part of the screenshot you attached from the admin interface. I had a look at this host in MMS, and it seems like your I/O is saturated. The background flushes are consistently taking ~35s and the iostat for sdb is hovering around 6,000 writes per second. You might be able to better distribute some of the I/O by moving the journal to a separate drive or reducing the syncdelay from the default of 60 seconds. You also note that you are using RAID5; this is definitely not the best choice for performance and will have significant overhead. As per our Linux production notes, RAID-10 is recommended. It also looks like your inserts & queries are significantly weighted toward shard1, so you may want to reconsider your choice of shard key. For further discussion on this, I would suggest posting in one of the support forums. Regards, |