The background key generator thread is what generates signature keys for cluster time validation. The lifetime of the key generator is like this:

1. Unconditionally enabled when a node initializes as a shard
2. Disabled on step-down
3. Enabled on step-up
4. Disabled on FCV downgrade from 3.6 to 3.4

The following problems exist with these transitions:

• Because of (4) above, an FCV change sequence from 3.6 -> 3.4 -> 3.6 will not re-enable the key generator, so it will not generate new keys and will cause the router to fail starting-up
  • This is not a major problem, because keys typically last for months and if a stall happens on router start-up, this can be worked around by stepping down the config server primary
• Because of (1) above, a secondary replica set node will end up with the key generator running
  • This is mitigated because when that key generator tries to insert a new key, it will fail with a NotMaster error