SERVER-27796 (still unsolved) indicates that there may be a bug related to concurrency control in the cluster cursor manager. While we have a JS concurrency test (also this one) which target mongos cursor manager, the bug has never been reproduced in our own test environment.
We should consider adding a unit test which stresses the concurrency control of the cluster cursor manager directly, in similar spirit to d_concurrency_test, which directly stresses the mongod locking code with 32 threads. If the cause of
SERVER-27796 is a concurrency control issue in the CCC, this would give us a better chance of reproducing it.
The existing JS test also runs with a mere 3 collections. The mongos code which triggers the invariant failure has to do with all cursors for a collection being destroyed, meaning that the JS test likely does not exercise this path very often. We should investigate whether this is indeed the case, and allow the test to run with more collections (assuming it won't make the existing workload useless) if so.