[SERVER-14724] How to take oplog backup, so that we can do point in time recovery Created: 29/Jul/14 Updated: 10/Dec/14 Resolved: 30/Jul/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Sarbamangal Choudhury | Assignee: | Stennie Steneker (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
We have over 200GB database. We are taking mongodump and restore. But it is taking long time( backup 3-4 hours, restore 5-6 hours) and putting lots of load on server. We are planning to take everyday full backup and oplog backup in every hour. So that we can restore any point in time in case of any failure. We like to know how we can do this. We can't do on MMS cloud due to security issue. 10gen is asking $5000 per replica set. We are looking for any open source backup utility which can do this without spending too much money. Full Backup : 10 AM daily Restore any time like 11AM, 9PM applying oplog on top on Full backup. |
| Comments |
| Comment by Sarbamangal Choudhury [ 05/Aug/14 ] | |
|
I tried this. But it did not work. I am getting this error during oplog apply: [root@mongo-dba-test oplogR]# mongorestore -h 10.98.141.231 --port 27017 --oplogReplay --oplogLimit 1407127315:363 /data/test/oplogR [root@mongo-dba-test oplogR]# Also I posed this question. But I am not getting good response. | |
| Comment by Stennie Steneker (Inactive) [ 30/Jul/14 ] | |
|
Hi Sarbamangal, The replication oplog (http://docs.mongodb.org/manual/core/replica-set-oplog/) is stored as a capped collection in MongoDB: oplog.rs in the local database. You can back the oplog up using a command line like:
To do a point in time restore you can replay a newer oplog against a full data backup that includes an oplog with a point in common. An example of finding a point in time entry and replaying the oplog is included in Asya's StackOverflow answer that I suggested earlier: http://stackoverflow.com/questions/15444920/modify-and-replay-mongodb-oplog/15451297#15451297 I recommend you try out the backup & restore process in a dev/UAT environment, and if you have any questions please follow up on an appropriate forum like the mongodb-user discussion group: https://groups.google.com/forum/#!forum/mongodb-user. If you're interested in learning more about MongoDB operations, I would highly recommend our free online MongoDB University courses: https://university.mongodb.com. A point-in-time restore is one of the exercises in the Advanced Deployment & Operations (M202) course. Regards, | |
| Comment by Sarbamangal Choudhury [ 30/Jul/14 ] | |
|
Still I am not able to get my answer. I am looking how we can take oplog backup ( in every hour and save as file) and apply it when we need to. Say for example: Mysql , we can take full , incremental and bin log ( like oplog) backup and apply bin log( file) on top of full or incremental backup. I like to know how I can do this on Mongodb. | |
| Comment by Stennie Steneker (Inactive) [ 30/Jul/14 ] | |
|
Hi Sarbamangal, The SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related community support discussion please post on the mongodb-users group (http://groups.google.com/group/mongodb-user) or Stack Overflow / ServerFault. I'm not sure what sort of deployment you are backing up (eg. replica set vs sharded cluster) but the MongoDB documentation has more information on different Backup Methods. Backing up with mongodump will create the smallest backup file because you are only exporting the data and index definitions, but also will take the longest time for a full restore as mongod needs to rebuild all of the data files and then the indexes. The typical (and fastest) approach for taking a backup would be to use a filesystem or EC2 snapshot. For example, see: Backup and Restore with Filesystem Snapshots or Backup a Sharded Cluster with Filesystem Snapshots. For an example of point-in-time recovery using the oplog, see Asya's answer on StackOverflow: Modify and replay MongoDB oplog. Regards, |