I am running MongoDB version 2.6.9 on three Amazon Web Services Linux boxes, one running as a primary node, the other two as secondary nodes. On one of the secondary nodes I am running a CRON job every hour that locks the database, runs mongodump, then unlocks the database.

However, every so often (perhaps once every few weeks, e.g. one in 500 or more times) mongodump will hang while writing out a collection. The collection it hangs on and the point at which it hangs vary, so it would indicate it's not related to the data in a specific collection.

This is an example of the script that runs:

mongo admin --eval --quiet "printjson(db.fsyncLock())" >> my-database-backup.log 2>&1
mongodump -d my-database >> my-database-backup.log 2>&1
mongo admin --eval "printjson(db.fsyncUnlock())" >> my-database-backup.log 2>&1

And here is an example of the log at the point at which mongodump hangs:

MongoDB shell version: 2.6.9
connecting to: admin

{ "info" : "now locked against writes, use db.fsyncUnlock() to unlock", "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand", "ok" : 1 }

connected to: 127.0.0.1
2015-11-11T10:00:02.221+0000 DATABASE: my-database to dump/my-database
2015-11-11T10:00:02.221+0000 my-database.system.indexes to dump/my-database/system.indexes.bson
2015-11-11T10:00:02.222+0000 120 documents
2015-11-11T10:00:02.222+0000 my-database.UserAuthCounters to dump/my-database/UserAuthCounters.bson
2015-11-11T10:00:02.223+0000 1 documents
2015-11-11T10:00:02.223+0000 Metadata for my-database.UserAuthCounters to dump/my-database/UserAuthCounters.metadata.json
2015-11-11T10:00:02.223+0000 my-database.UserAuths to dump/my-database/UserAuths.bson
2015-11-11T10:00:02.365+0000 47791 documents
2015-11-11T10:00:02.365+0000 Metadata for my-database.UserAuths to dump/my-database/UserAuths.metadata.json

...

2015-11-11T10:04:18.141+0000 my-database.AuditLogModels to dump/my-database/AuditLogModels.bson
2015-11-11T10:04:18.143+0000 500 documents
2015-11-11T10:04:18.143+0000 Metadata for my-database.AuditLogModels to dump/my-database/AuditLogModels.metadata.json
2015-11-11T10:04:18.143+0000 my-database.TransactionModels to dump/my-database/TransactionModels.bson
2015-11-11T10:04:21.141+0000 Collection File Writing Progress: 1900/52571 3% (documents)
2015-11-11T10:04:40.479+0000 Collection File Writing Progress: 3600/52571 6% (documents)

10 minutes later I killed the mongodump process as it hadn't done anything further, it just seemed to hang at 6%.