Uploaded image for project: 'MongoDB Tools'
  1. MongoDB Tools
  2. TOOLS-26

Allow mongodump --oplog to work for a single db/collection

    Details

      Description

      Filter out oplog entries not for the db, or collection, being backed up.

        Issue Links

          Activity

          Hide
          scotthernandez Scott Hernandez added a comment -

          I should note that with the current implementation of replication (using the oplog, with s single increasing ts across all namespaces) restoring data like this will cause replication to break, and skip all filtered out oplog entries forever. While this feature is logically desirable it is not technically possible without changing how replication works; it might make more sense if filter replication is ever supported.

          Show
          scotthernandez Scott Hernandez added a comment - I should note that with the current implementation of replication (using the oplog, with s single increasing ts across all namespaces) restoring data like this will cause replication to break, and skip all filtered out oplog entries forever. While this feature is logically desirable it is not technically possible without changing how replication works; it might make more sense if filter replication is ever supported.
          Hide
          y_feldblum Jay Feldblum added a comment -

          +1 for this feature.

          I understand that it would be a major change to implement it.

          But it would be very useful to have completely live/online backups from a secondary of just one database - including, specifically, NOT backing up the admin/local databases.

          A case where we don't want to back up the admin database is where that is completely driven by configuration management and so does not need to be backed up (we can simply re-run the configuration management script to re-generate all admin data automatically, including replica set peers and admin users).

          A case where we don't want to back up the local database is where we back up data from one secondary and use it to restore the primary or any other secondary.

          And then there are the normal cases where we're only interested in backing up the data in database X every 24 hours, and we're only interested in backing up the data in database Y very 12 hours, and we're interested in storing these dumps in separate archives, etc.

          Question. Could mongodump apply the oplog to the dumped bson files (as mongod would apply the oplog), then delete the oplog from the dump? If so, restoring from the dump wouldn't break replication.

          Show
          y_feldblum Jay Feldblum added a comment - +1 for this feature. I understand that it would be a major change to implement it. But it would be very useful to have completely live/online backups from a secondary of just one database - including, specifically, NOT backing up the admin/local databases. A case where we don't want to back up the admin database is where that is completely driven by configuration management and so does not need to be backed up (we can simply re-run the configuration management script to re-generate all admin data automatically, including replica set peers and admin users). A case where we don't want to back up the local database is where we back up data from one secondary and use it to restore the primary or any other secondary. And then there are the normal cases where we're only interested in backing up the data in database X every 24 hours, and we're only interested in backing up the data in database Y very 12 hours, and we're interested in storing these dumps in separate archives, etc. Question. Could mongodump apply the oplog to the dumped bson files (as mongod would apply the oplog), then delete the oplog from the dump? If so, restoring from the dump wouldn't break replication.
          Hide
          akira.kurogane Akira Kurogane added a comment -

          I have linked this to TOOLS-646, which would be a more comprehensive enhancement. Rather than just filtering the oplog to include the operations in one db's namespace, it asks for that to be a list of dbs. Furthermore it asks if an --exclude option is possible.

          Show
          akira.kurogane Akira Kurogane added a comment - I have linked this to TOOLS-646 , which would be a more comprehensive enhancement. Rather than just filtering the oplog to include the operations in one db's namespace, it asks for that to be a list of dbs. Furthermore it asks if an --exclude option is possible.

            People

            • Assignee:
              Unassigned
              Reporter:
              scotthernandez Scott Hernandez
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: