Our initial mongo configuration:
1 Primary: serverA
1 Secondary: serverB
1 Hidden Secondary: serverC

Application is configured for w:majority.

Today we ran into a scenario where we were adding nodes to our environment. serverD and serverE, with an equal configuration as serverA and serverB, were added to the replica set and had begun replicating. It was noted through an rs.status() that they were both syncing from serverB. Soon afterwards (during the sync) a long running query was executed. Performance of the application began to decline. Even after the long running query had completed our application eventually appeared down. We decided to restart the application which performs a couple of harmless writes to the mongo database before starting, but this step hung.

During this period of time we noted that serverC's replication lag was high. The performance of serverC is much less than that of all other servers, but he's a hidden secondary host with the hopes it doesn't matter.

To get our app back up and running, we removed serverD and serverE from the replica set and then we were able to successfully restart our application. ServerC's replication lag continued to be high until he finally caught up later in the day.

The question I have is regarding how does one determine the write concern. With our initial setup of only 3 servers, one secondary being hidden, does this hidden host count towards that majority? If it's invisible to the application, will mongo wait for an ack from this node?

When we added 2 more hosts to the environment, and during the time we were syncing data, did the write majority value change? Originally it was 2/3 servers, did it change to 3/5 despite those two new servers not being ready to serve requests?

We are trying to better understand how write concern is calculated as configured using majority and have not been able to find the answer we are looking for in the documentation available.

Due to the issues today we are planning on changing our configuration of write concern to w:2. If we send a write to the replica set, which secondary will we wait for an ack? Knowing we only need to wait for the ack from the non-hidden secondary, could there be a situation where we need to wait for the hidden secondary to send an ack as well? Does this configuration change make sense per our problem that occurred today?