[SERVER-10503] High iowait on secondary but don't see many writes in MongoDB web interface Created: 14/Aug/13  Updated: 10/Dec/14  Resolved: 19/Aug/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.4.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nic Cottrell (Personal) Assignee: Stennie Steneker (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

rhel6, 40GB ram, RAID5


Attachments: PNG File Screen Shot 2013-08-14 at 10.34.27 AM.png     Text File mongostat.txt    
Operating System: ALL
Participants:

 Description   

This server is the secondary of a replica set which is the primary of a sharding cluster. There is iowait >10% constantly and I see writes of about 1.5Mb/s with iotop. In the mongo web interface I don't see many writes - in fact one lists the collection as blank.

Any idea why the iowait is so high? Or a way to track down which collection is causing the writes?



 Comments   
Comment by Stennie Steneker (Inactive) [ 19/Aug/13 ]

Hi Nic,

The SERVER project is for reporting bugs or feature suggestions for the MongoDB server.

For MongoDB-related support discussion please post on the mongodb-users group (http://groups.google.com/group/mongodb-user) or Stack Overflow / ServerFault.

In terms of looking at current activity on your server, in addition to mongostat (which you are already using) you could use db.currentOp() to report in-progress operations. The currentOp output is the top part of the screenshot you attached from the admin interface.

I had a look at this host in MMS, and it seems like your I/O is saturated. The background flushes are consistently taking ~35s and the iostat for sdb is hovering around 6,000 writes per second. You might be able to better distribute some of the I/O by moving the journal to a separate drive or reducing the syncdelay from the default of 60 seconds. You also note that you are using RAID5; this is definitely not the best choice for performance and will have significant overhead. As per our Linux production notes, RAID-10 is recommended. It also looks like your inserts & queries are significantly weighted toward shard1, so you may want to reconsider your choice of shard key.

For further discussion on this, I would suggest posting in one of the support forums.

Regards,
Stephen

Generated at Thu Feb 08 03:23:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.