Startup a mongod instance with plan ranker mode automaticCE. Load data into the mongod instance. Create indexes that are relevant to your queries. Restart the mongod instance. I have a harder time reproducing the issue if indexes are created just before running the queries. Run one or a set of relatively cheap queries such that the total number of runtimes is 100. Plot the runtimes from the explains. By relatively cheap, I mean a query that has an expected runtime of <3 ms. I could not reproduce the issue when I swapped the 1 million document 371 MB dataset with a 10,000 document 4MB dataset, so it seems like the collection needs to have a minimum size. A script that to reproduce the issue is here: https://github.com/10gen/mongo/blob/spm-2658-local-analysis/src/mongo/db/query/benchmark/analysis/unstable_query.ipynb