Many of our CPU-bound performance workloads get gradually slower over time even when there are no significant changes to the workload's hot code paths. One possibility is that the steadily increasing size of the the mongos and mongod executables make their performance more sensitive to i-cache efficiency.

One good way to test this effect (if it is real) would be to run some of these workloads on a mongod that has been stripped down to remove functionality that they do not use and see if it improves any of the benchmark numbers.

If there is a significant effect, it may be possible to get the same benefit in practice by breaking large features into separate dynamic libraries, which will never get loaded if a user does not use the feature.