Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-986

mongodump (2.6.9) hangs intermittently

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mongodump
    • Labels:
    • Environment:
      AWS (r3.xlarge) GNU/Linux, 1 PRIMARY node, 2 SECONDARY nodes
    • Backport Requested:
      v2.6

      Description

      I am running MongoDB version 2.6.9 on three Amazon Web Services Linux boxes, one running as a primary node, the other two as secondary nodes. On one of the secondary nodes I am running a CRON job every hour that locks the database, runs mongodump, then unlocks the database.

      However, every so often (perhaps once every few weeks, e.g. one in 500 or more times) mongodump will hang while writing out a collection. The collection it hangs on and the point at which it hangs vary, so it would indicate it's not related to the data in a specific collection.

      This is an example of the script that runs:

      mongo admin --eval --quiet "printjson(db.fsyncLock())" >> my-database-backup.log 2>&1
      mongodump -d my-database >> my-database-backup.log 2>&1
      mongo admin --eval "printjson(db.fsyncUnlock())" >> my-database-backup.log 2>&1

      And here is an example of the log at the point at which mongodump hangs:

      MongoDB shell version: 2.6.9
      connecting to: admin

      { "info" : "now locked against writes, use db.fsyncUnlock() to unlock", "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand", "ok" : 1 }

      connected to: 127.0.0.1
      2015-11-11T10:00:02.221+0000 DATABASE: my-database to dump/my-database
      2015-11-11T10:00:02.221+0000 my-database.system.indexes to dump/my-database/system.indexes.bson
      2015-11-11T10:00:02.222+0000 120 documents
      2015-11-11T10:00:02.222+0000 my-database.UserAuthCounters to dump/my-database/UserAuthCounters.bson
      2015-11-11T10:00:02.223+0000 1 documents
      2015-11-11T10:00:02.223+0000 Metadata for my-database.UserAuthCounters to dump/my-database/UserAuthCounters.metadata.json
      2015-11-11T10:00:02.223+0000 my-database.UserAuths to dump/my-database/UserAuths.bson
      2015-11-11T10:00:02.365+0000 47791 documents
      2015-11-11T10:00:02.365+0000 Metadata for my-database.UserAuths to dump/my-database/UserAuths.metadata.json

      ...

      2015-11-11T10:04:18.141+0000 my-database.AuditLogModels to dump/my-database/AuditLogModels.bson
      2015-11-11T10:04:18.143+0000 500 documents
      2015-11-11T10:04:18.143+0000 Metadata for my-database.AuditLogModels to dump/my-database/AuditLogModels.metadata.json
      2015-11-11T10:04:18.143+0000 my-database.TransactionModels to dump/my-database/TransactionModels.bson
      2015-11-11T10:04:21.141+0000 Collection File Writing Progress: 1900/52571 3% (documents)
      2015-11-11T10:04:40.479+0000 Collection File Writing Progress: 3600/52571 6% (documents)

      10 minutes later I killed the mongodump process as it hadn't done anything further, it just seemed to hang at 6%.

        Attachments

          Activity

            People

            Assignee:
            gabriel.russell Gabriel Russell (Inactive)
            Reporter:
            AnturGyffrous Owain Williams [X]
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: