It seems that currently any change for the replica set config with rs.reconfig() will cause a voting and the cluster will be down for a moment. This should be change so that the primary won't drop to secondary unless it's really necessary.

For example I changed one slave from hidden to non hidden and the rs.reconfig() caused the primary to drop into secondary for a few seconds. I see no reason why this is needed, because it adds downtime for no valid reason.