One of secondary mongod host has crushed recently complaining "..too many open files..." and when we did try to do initial sync it also failed with the same error message.  

This the full information about mongod version and server :

db version v3.4.21
git version: ba2e818de44c9a331ce09a624780a2bd841cf6e4
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: enterprise
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64