[SERVER-14724] How to take oplog backup, so that we can do point in time recovery Created: 29/Jul/14  Updated: 10/Dec/14  Resolved: 30/Jul/14

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Sarbamangal Choudhury Assignee: Stennie Steneker (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

We have over 200GB database. We are taking mongodump and restore. But it is taking long time( backup 3-4 hours, restore 5-6 hours) and putting lots of load on server. We are planning to take everyday full backup and oplog backup in every hour. So that we can restore any point in time in case of any failure. We like to know how we can do this. We can't do on MMS cloud due to security issue. 10gen is asking $5000 per replica set. We are looking for any open source backup utility which can do this without spending too much money.

Full Backup : 10 AM daily
OPlog : every hour.

Restore any time like 11AM, 9PM applying oplog on top on Full backup.



 Comments   
Comment by Sarbamangal Choudhury [ 05/Aug/14 ]

I tried this. But it did not work. I am getting this error during oplog apply:

[root@mongo-dba-test oplogR]# mongorestore -h 10.98.141.231 --port 27017 --oplogReplay --oplogLimit 1407127315:363 /data/test/oplogR
connected to: 10.98.141.231:27017
Tue Aug 5 13:33:56.785 The oplogLimit is not newer than the last oplog entry on the server.
[root@mongo-dba-test oplogR]# mongorestore -h 10.98.141.231 --port 27017 --oplogReplay /data/test/oplogR
connected to: 10.98.141.231:27017
Tue Aug 5 13:34:26.386 Replaying oplog
assertion: 13106 nextSafe():

{ $err: "Invalid ns [.$cmd]", code: 16256 }

[root@mongo-dba-test oplogR]#

Also I posed this question. But I am not getting good response.

Comment by Stennie Steneker (Inactive) [ 30/Jul/14 ]

Hi Sarbamangal,

The replication oplog (http://docs.mongodb.org/manual/core/replica-set-oplog/) is stored as a capped collection in MongoDB: oplog.rs in the local database.

You can back the oplog up using a command line like:

mongodump --db local --collection oplog.rs --out oplog.bson

To do a point in time restore you can replay a newer oplog against a full data backup that includes an oplog with a point in common.

An example of finding a point in time entry and replaying the oplog is included in Asya's StackOverflow answer that I suggested earlier: http://stackoverflow.com/questions/15444920/modify-and-replay-mongodb-oplog/15451297#15451297

I recommend you try out the backup & restore process in a dev/UAT environment, and if you have any questions please follow up on an appropriate forum like the mongodb-user discussion group: https://groups.google.com/forum/#!forum/mongodb-user.

If you're interested in learning more about MongoDB operations, I would highly recommend our free online MongoDB University courses: https://university.mongodb.com. A point-in-time restore is one of the exercises in the Advanced Deployment & Operations (M202) course.

Regards,
Stephen

Comment by Sarbamangal Choudhury [ 30/Jul/14 ]

Still I am not able to get my answer. I am looking how we can take oplog backup ( in every hour and save as file) and apply it when we need to. Say for example: Mysql , we can take full , incremental and bin log ( like oplog) backup and apply bin log( file) on top of full or incremental backup. I like to know how I can do this on Mongodb.

Comment by Stennie Steneker (Inactive) [ 30/Jul/14 ]

Hi Sarbamangal,

The SERVER project is for reporting bugs or feature suggestions for the MongoDB server.

For MongoDB-related community support discussion please post on the mongodb-users group (http://groups.google.com/group/mongodb-user) or Stack Overflow / ServerFault.

I'm not sure what sort of deployment you are backing up (eg. replica set vs sharded cluster) but the MongoDB documentation has more information on different Backup Methods.

Backing up with mongodump will create the smallest backup file because you are only exporting the data and index definitions, but also will take the longest time for a full restore as mongod needs to rebuild all of the data files and then the indexes.

The typical (and fastest) approach for taking a backup would be to use a filesystem or EC2 snapshot. For example, see: Backup and Restore with Filesystem Snapshots or Backup a Sharded Cluster with Filesystem Snapshots.

For an example of point-in-time recovery using the oplog, see Asya's answer on StackOverflow: Modify and replay MongoDB oplog.

Regards,
Stephen

Generated at Thu Feb 08 03:35:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.