From the comments of BF-41134:

I was thinking that inserts preceding calls to analyzeShardKey like this might be the culprit? I know that we ultimately send w: <nodesInTheReplSet>, but the description of SERVER-91796 (if I'm understanding it correctly) referencing SPM-3489 and my reading of the repl README's timestamp glossary make me think that that write concern isn't enough to prevent anlyzeShardKey from hitting the "gap" on a secondary between `lastApplied` and `lastWritten.`

I'm trying to get an idea of how often this situation happens by re-running some of the failing tasks. I think a pragmatic thing to try might be to just insert some awaitReplication calls after the writes in this test and see if the issue recurs.