We recommend that customers use NTP (or similar where NTP does not apply) to keep their servers in sync. There are vague descriptions of the consequences to having out of sync hosts, but beyond making log correlation difficult, I am not aware of a concrete list of potential consequences.

When we make the recommendation that users sync their hosts, that is often the first question: "What will happen if I don't?" Usually followed by something along the lines of: "Could this cause corruption/crashes/<insert other problem here>?"

If we could outline the potential pitfalls, that would be great. Something like:

mongos needs to be in sync to properly obtain and manage locks on the config databases (note: I have no idea if this is the case, just giving an example)