Mongos consistently crashes under high loads (~2500 concurrent requests). This is easily reproducible on our system. Stack traces and logs at links below.

1) First crash stack traces, two out of three mongos died at the same time.
https://gist.github.com/a70ccfbdd5e58430f68c

2) mongos log: http://sprunge.us/OEiU - we noticed in this one: "ERROR: Out of file descriptors. Waiting one second before trying to accept more connections." So we increased the file descriptors and that one went away for now. Regardless, mongos should not crash like this.

Somewhere in here, we increased the number of mongos from 3 to 12 and it seemed to work a bit better, but still easy to make it crash.

3) And this just goes on and on, we'll restart them and easily kill them with a bit of load.

http://sprunge.us/IfUR
http://sprunge.us/gFSS
http://sprunge.us/dLCj
http://sprunge.us/XSOb
http://sprunge.us/TZZD
http://sprunge.us/ZNUO