MongoDB replica secondaries crashing and not-starting
I have a 3 node replica set with mongodb version of 3.4.13 running on ubuntu 16.
I updated a secondary node from 3.4.13 to 3.6.5 and switched it to become primary. This node became primary and running fine with no issues.
Soon after that, the rest two secondary nodes crashed with a stack trace given below.
I tried to restart them but i am still getting the same errors and the mongo is aborting.
1. Return error status from SessionsCollection derived ::setupSessionsCollection method if the featureCompatibility is not fully upgraded to 3.6 this status will be checked in the _refresh method https://github.com/mongodb/mongo/blob/r3.6.6/src/mongo/db/logical_session_cache_impl.cpp#L275-L279 . This is the main defense against setting up sessions table too early
2. return from _refresh if FCV is not fully upgraded to 3.6 in https://github.com/mongodb/mongo/blob/r3.6.6/src/mongo/db/logical_session_cache_impl.cpp#L262 i.e. after computing statistics but before calling setupSessionCollections. This way the logs will not be polluted with the messages. The full downgrade support is tracked in
3. This is not required for this fix but nice to have while on it: Change return type https://github.com/mongodb/mongo/blob/r3.6.6/src/mongo/db/initialize_operation_session_info.cpp#L41 from void to Status and consequently do not uassert inside initializeOperationSessionInfo
and uassert in the callers.