Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1590

mongodump with --oplog should not count the amount of entries

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.2.11
    • Fix Version/s: None
    • Component/s: mongodump
    • Backport Requested:
      v3.4, v3.2
    • Documentation Changes:
      Not Needed

      Description

      Currently, oplog dumping uses the dumpQueryToIntent (previously dumpQueryToWriter) that checks the command-line query to be non-empty. I believe that when oplog is being dumped, this check should be skipped and the total oplog entry count should not be obtained, because it effectively scans the oplog in full (since there is on index on "ts" field and the query is something along the lines of {ts:{$gt:...}}).
      While having the progress bar is certainly helpful, forcing an extra (possibly 100+GB) oplog scan to output it properly seems a bit overkill, since this leads to excessive disk utilization on databases with large historical oplogs (not to mention killing the cache on MMAPv1 engines, straining the database).
      Also, although I'm not sure, maybe the "query" instead of "dump.query" should be checked, then it will work for oplog queries automatically.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              backlog-server-tools Backlog - Server Tools Team
              Reporter:
              onyxmaster Aristarkh Zagorodnikov
              Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: