Core Server
  1. Core Server
  2. SERVER-4273

Allow mongodump --oplog to work for a single db/collection

    Details

    • Backport:
      No
    • # Replies:
      2
    • Last comment by Customer:
      true

      Description

      Filter out oplog entries not for the db, or collection, being backed up.

        Activity

        Hide
        Scott Hernandez
        added a comment -

        I should note that with the current implementation of replication (using the oplog, with s single increasing ts across all namespaces) restoring data like this will cause replication to break, and skip all filtered out oplog entries forever. While this feature is logically desirable it is not technically possible without changing how replication works; it might make more sense if filter replication is ever supported.

        Show
        Scott Hernandez
        added a comment - I should note that with the current implementation of replication (using the oplog, with s single increasing ts across all namespaces) restoring data like this will cause replication to break, and skip all filtered out oplog entries forever. While this feature is logically desirable it is not technically possible without changing how replication works; it might make more sense if filter replication is ever supported.
        Hide
        Jay Feldblum
        added a comment -

        +1 for this feature.

        I understand that it would be a major change to implement it.

        But it would be very useful to have completely live/online backups from a secondary of just one database - including, specifically, NOT backing up the admin/local databases.

        A case where we don't want to back up the admin database is where that is completely driven by configuration management and so does not need to be backed up (we can simply re-run the configuration management script to re-generate all admin data automatically, including replica set peers and admin users).

        A case where we don't want to back up the local database is where we back up data from one secondary and use it to restore the primary or any other secondary.

        And then there are the normal cases where we're only interested in backing up the data in database X every 24 hours, and we're only interested in backing up the data in database Y very 12 hours, and we're interested in storing these dumps in separate archives, etc.

        Question. Could mongodump apply the oplog to the dumped bson files (as mongod would apply the oplog), then delete the oplog from the dump? If so, restoring from the dump wouldn't break replication.

        Show
        Jay Feldblum
        added a comment - +1 for this feature. I understand that it would be a major change to implement it. But it would be very useful to have completely live/online backups from a secondary of just one database - including, specifically, NOT backing up the admin/local databases. A case where we don't want to back up the admin database is where that is completely driven by configuration management and so does not need to be backed up (we can simply re-run the configuration management script to re-generate all admin data automatically, including replica set peers and admin users). A case where we don't want to back up the local database is where we back up data from one secondary and use it to restore the primary or any other secondary. And then there are the normal cases where we're only interested in backing up the data in database X every 24 hours, and we're only interested in backing up the data in database Y very 12 hours, and we're interested in storing these dumps in separate archives, etc. Question. Could mongodump apply the oplog to the dumped bson files (as mongod would apply the oplog), then delete the oplog from the dump? If so, restoring from the dump wouldn't break replication.

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Days since reply:
              2 years, 20 weeks, 5 days ago
              Date of 1st Reply: