The evergreen set-module command does not update the manifest/pinned commit for modules. It seems that
EVG-5560 is broken.
Steps to reproduce:
Expected: The genny module is checked out at revision 9ae96a.
Actual: The genny module is checked out at master (31373926 as of 2019-04-26).
This isn't a problem with fetch, it's just an easy way to demonstrate it. It's a problem with set-module. It seems the set-module command does not update the manifest.
This makes it hard to see if different combinations of projects and modules reproduce errors.
It was suggested on slack that the solution is to hard-code the revision that's checked out in the evergreen yaml. This is tedious and requires submitting a whole new patch when you may wan to re-run the same patch with a different module revision.