Overview ======== The MongoDB Management Service (MMS) is a service for monitoring and backing up MongoDB infrastructure. MMS Backup provides backup and restore functionality for MongoDB instances being monitored. Engineered specifically for MongoDB, BRS features scheduled snapshots and point in time recovery. It can take a consistent snapshot across multiple shards. Once the service is up and running, backup and restoration can be managed by the end user from an easy to use web interface. BRS can easily be scaled horizontally to the desired level of availability and durability for the backups. This document will cover MMS where necessary, but will focus on BRS. Details about MMS can be found at http://mms.mongodb.com/help-hosted. How it works ============ A lightweight agent runs within your infrastructure and connects to the configured MongoDB instances. Using the same mechanism as replication the agent will perform an initial sync and then tail the oplog of a replica set’s primary. Instead of acting as another local secondary the agent ships the data over HTTP back to the BRS service. The service will recreate every replica set being backed up and apply oplogs when the data comes in from the agents. This will result in the service having a mongodb database on disk for every replica set being backed up (referred to as the "heads"). Each “head” is consistent with the primary up to the last oplog supplied by the agent. The service will take scheduled snapshots of all heads and retain those snapshots based on a user-defined policy. Snapshot storage is very efficient. The data is compressed and each snapshot only needs to store the data that has changed from the previous snapshots. When a restore is necessary the data can be retrieved from the snapshot storage. A point in time restore is accomplished by pulling a snapshot and applying stored oplogs to reach the desired time. Components ========== MMS --- The MMS service contains the web interface for managing MongoDB monitoring and backups. Users can sign up, creating new accounts or joining an existing group. The MMS service also contains endpoints used by the MMS Agent to report back information on monitored MongoDB instances. MMSDB* Databases ---------------- The MMS service will create a set of databases in its configured mongodb instance. These databases will hold user/group metadata and monitoring data. In a production setup, the MMS databases should be configured to a replica set to provide durability and failover. Backup HTTP Server --------------------------- The Backup HTTP Server contains a set of web services used by the backup agent. The agent retrieves its configuration from this service. The agent also sends back initial sync and oplog data through this interface. Backup Alert Service ------------------------------ The Backup Alert Service watches the state of all agents, heads, and snapshots. It will send email alerts when problems are found. Backup Daemon --------------------- The Backup Daemon manages all heads and snapshots. Jobs are queued based on data coming in to the Backup HTTP server from the agents. The Daemon acts on those jobs. No client applications talk directly to the daemon. Its state and job queues come from the backup database. The daemon creates the heads on its local disk. If there are multiple servers running the daemon, new incoming backups will be grabbed by an idle daemon and that backup's head will live with that daemon. The daemon also takes snapshots and stores those snapshots in the Blockstore. Backup* and Oplog Databases --------------------------- The Backup* and Oplog Databases (called backupdb in the configuration) are a set of databases used by the Backup HTTP Server and Daemon to manage all backup metadata. The Backup databases will hold job state, snapshot metadata, and temporary storage for the initial sync of a new replica set. There will also be a database for every MMS group that contains the last 24 hours of oplogs for every replica set being backed up. The database name will be the name of the group. In a production setup, the backup database should be configured to a replica set to provide durability and failover. Blockstore Databases ------------------- The Blockstore holds all snapshots broken down into blocks. The Blockstore will contain a database for every replica set. The database name will be the replica set id. The durability of your snapshots increased for every replica secondary added to the Blockstore. Hardware Requirements ===================== The front-end installation contains MMS, the Backup HTTP Server, and the Backup Alert Service. A server running this package should have at minimum 2 CPU cores and 8GB of RAM to get started. As more servers are monitored and backed up the requirements will increase. The back-end installation contains the Daemon. The resource requirements for the Daemon vary greatly. The server will need enough RAM, CPU, and IOPs to handle being a replica secondary for the replica sets it will manage. The daemon will also need enough storage to hold the heads for all replica sets it will manage. The MMS and Backup databases should have at minimum 4 CPU cores, 8GB of RAM, although this could vary based on the data set. SSDs or fast HDDs are recommended. MMS will need at least 200GB of storage for the first 400 MongoDB instances. The Backup database will need enough storage to hold 24 hours of oplogs for all replica sets being backed up and temporarily the initial sync of any new backup. The Blockstore database should have at minimum 4 CPU cores and 8GB of RAM. The storage requirements will vary greatly based on the size of the data backed up and the churn of that data. The Blockstore database can be placed on traditional HDDs. Software Requirements ===================== MMS requires 64-bit Linux. The following distributions are supported: * CentOS 5 or later * Red Hat Enterprise Linux 5 or later * Amazon Linux (latest version only) * SLES 11 or later * Ubuntu 12.04 or later The MMS service requires MongoDB 2.4.6 or later. The MongoDB replica sets and sharded clusters to be backuped up must be running MongoDB 2.4.3 or later. OS Configuration ============= Ulimit ------ It is extremely important that ulimits be configured correctly. The MMS/Backup HTTP server under load and the Backup Daemon launching many mongods will quickly hit most linux distributions default ulimits. The following settings are recommended for the /etc/security/limits.conf file on all servers running the front-end or back-end package. mongodb-mms soft nofile 64000 mongodb-mms hard nofile 64000 mongodb-mms soft nproc 32000 mongodb-mms hard nproc 32000 mongod soft nofile 64000 mongod hard nofile 64000 mongod soft nproc 32000 mongod hard nproc 32000 Be sure to check for a /etc/security/limits.d/90-nproc.confls file that may override the configured limits. Firewall -------- The front-end package will default to running web servers on ports 8080 and 8081. Installation ============ The following instructions must be executed as root. 1. Install a MongoDB server (or replica set) for the MMS databases. Detailed instructions on getting started with a MongoDB installation can be found at http://www.mongodb.org/downloads 2. Install a MongoDB server (or replica set) for the Backup databases. 3. Install a MongoDB server (or replica set) for the Blockstore database. 4. Install the MMS/BRS software. The software consists of two packages, the front-end package (mongodb-mms-brs-X.X-X_X.x86_64.rpm / mongodb-mms-brs-X.X.X.x86_64.deb) and the back-end package (mongodb-backup-daemon-X.X-X_X.x86_64.rpm / mongodb-backup-daemon-X.X.X.x86_64.deb). To install the front-end package on RHEL/CentOS/Amazon Linux/SLES run: rpm -ivh mongodb-mms-brs-X.X-X_X.x86_64.rpm To install the front-end package on Ubuntu run: dpkg -i mongodb-mms-brs-X.X.X.x86_64.deb The software will be installed to /opt/mongodb/mms. Open the application configuration file at /opt/mongodb/mms/conf/conf-mms.properties and fill in all of the required configuration. All of the MongoDB connections default to localhost. If you are not running the MMS and Backup databases on this server change these values to the correct host. The backup database configuration setting (called backupdb) should point to the Backup database, setup in step 2. All other database connections in the front-end config file are for the MMS database and should point to the MMS database from step. More detailed information on MMS configuration can be found at http://mms.mongodb.com/help-hosted. To start the front-end package run: service mongodb-mms start If everything worked the following should be displayed: Start MMS server Instance 0 starting... [ OK ] Start Backup HTTP Server Instance 0 starting... [ OK ] Start Backup Alert Process [ OK ] You should now be able to view MMS by going to the configured central URL in a web browser. The default port of MMS is 8080. If you run into any problems, the log files can be found at /opt/mongodb/mms/logs To install the back-end package on RHEL/CentOS/Amazon Linux/SLES run: rpm -ivh mongodb-backup-daemon-X.X-X_X.x86_64.rpm To install the back-end backup on Ubuntu run: dpkg -i mongodb-backup-daemon-X.X.X.x86_64.deb The software will be installed to /opt/mongodb/backup-daemon. Open the application configuration file at /opt/mongodb/backup-daemon/conf/conf-daemon.properties and fill in all of the required configuration. Same as the front-end package the MMS settings should point to the database setup in step 1, the Backup setting to the database in step 2, and the Blockstore setting to the database setup in step 3. To start the back-end package run: service mongodb-backup-daemon start If everything worked the following will be displayed: Start Backup Daemon [ OK ] If you run into any problems, the log files can be found at /opt/mongodb/backup-daemon/logs Usage ===== To begin backing up a replica set or sharded cluster the MongoDB instances must first be monitored by MMS. For instructions on installing the MMS agent and adding instances refer to the mms-manual.pdf. After your instances are monitored by MMS, the instructions for the hosted version of BRS can be used to get started backing up your first replica set or sharded cluster. Enable Backup for a Replica Set https://mms.mongodb.com/help/backup/tutorial/enable-backup-for-replica-set/ Enable Backup for a Sharded Cluster https://mms.mongodb.com/help/backup/tutorial/enable-backup-for-sharded-cluster/ Restore from a Stored Snapshot https://mms.mongodb.com/help/backup/tutorial/restore-from-snapshot/ Restore from a Point in the Last 24 Hours https://mms.mongodb.com/help/backup/tutorial/restore-from-point-in-time-snapshot/ Upgrades ======== Version 1.2.1 ------------- The MMS service now requires MongoDB 2.4.6 or later. Maintenance ================== As Snapshots change, the Blockstore will accumulate dead blocks over time. To reclaim this space a garage collection must be run. This is a very resource intensive process and will temporarily double the Blockstore space used by a replica set. This is currently a manual process, although it will be automated in the future. The are two utilities installed with the back-end package to help with garbage collection called backup-blockstore-usage and backup-blockstore-repo. They can be found in the bin directory (/opt/mongodb/backup-daemon/bin). Running backup-blockstore-usage will print a table with all replica sets, the number of live bytes, and the number of dead bytes. Dead bytes is space that can be reclaimed by running a garbage collection. The backup-blockstore-repo will run a garbage collection on a replicate set.