I have a system setup that is very similar to the architecture described in http://www.snailinaturtleneck.com/blog/2010/08/30/return-of-the-mongo-mailbag/
? 4 servers, 2 shards.
A disturbing fact is that the number of open connections to the two masters seems to be constantly increasing over time (see the attached ganglia graph).

So here are a few weird observations that I made:
1) One mongos has less than 20 client connections but over 30 connections to each mongod
2) The number of open connections reported by netstat is much less than the number of connections reported in db.serverStatus(): 50 vs. 101
3) db.currentOps() shows a few active "writebacklisten" operations that are running for 3+ days (might be related to SERVER-2434)
4) the mongos log contains a couple of client connection timeout errors like: "MessagingPort recv() errno:110 Connection timed out..."

The clients use the Java driver in version 2.4 and the client instance is kept open and reused for each query.