We have a MongoDB cluster host on-premise on AWS, containing 1 primary node and 2 secondary nodes, on 3 r5 EC2 instances. Due to some heavy workloads, the primary node's memory utilization reached 100% and then the instance crashed.

After rebooting the instance, we restart the MongoDB, one of the secondary nodes became the primary as expected. Then the former primary node turned into ROLLBACK state. We noticed the docs on https://docs.mongodb.com/manual/core/replica-set-rollbacks/ that this is because secondaries can not keep up with the throughput of operations on the former primary. However, it stuck at the state after several rollback files were created under the rollback folder, and after that, we did not notice any new rollback activities on the log.

In the end, we stopped MongoDB, cleared all data on the node, and started again to sync data from the replica set.