By default, mongos invalidates its user cache every 10 minutes. It's possible to change this interval at runtime using the "userCacheInvalidationIntervalSecs" option to setParameter. The problem is that changing this value with setParameter doesn't wake up the invalidation thread if its already sleeping. This means that if you start up a new mongos and then immediately change its cache invalidation interval to something shorter than the 10 minute default, it is likely that the cache still won't be invalidated for almost 10 minutes, since the invalidation thread is already in the middle of a 10 minute sleep when you change the sleep interval.
This could be fixed without too much work by changing the invalidation thread to sleep on a condition variable and by signaling that condvar whenever the "userCacheInvalidationIntervalSecs" value is changed.