In https://github.com/mongodb/mongoid/pull/4953 it can be seen that the app tests include their own list of Rails versions that are then tested in each configuration. This creates problems when additional Rails configurations are added in Evergreen since these Rails versions wouldn't be tested by the app tests.

Instead the app tests should use the Rails version configured in Evergreen, and there should be Evergreen configurations running app tests for each supported Rails version.