[SERVER-32562] Importing local database into standalone server causes update to replset.minvalid Created: 05/Jan/18  Updated: 19/Jan/18  Resolved: 19/Jan/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Arnie Listhaus Assignee: Spencer Brody (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File 00466802 - Split Replica Set    
Operating System: ALL
Sprint: Repl 2018-01-29
Participants:
Case:

 Description   

TL;DR
When importing the local database into a standalone server, the document inside of the replset.minvalid collection is modified with an updated timestamp. Since the server is a standalone server and not a member of a replica set, this should not occur. This unexpected behavior made the process described below a bit more difficult to work through.

Background

I wanted to move a large database (1TB) to a new replica set without having any significant application downtime.

Only one of my applications targeted the specific database in the replica set which was 1TB in size. The total size of all of the databases in the replica set was 6TB.

The Initial Plan

The plan was to add two secondary nodes to the existing replica set, allow the initial sync to populate the new nodes and then remove the new nodes from the replica set to create a new one with all of the data in tact. The removal of the new nodes and spinning up a new replica set (with an arbiter as the third node) could be done in minutes vs the many hours it would take for a backup/restore. Once the new replica set was up the application that targeted the 1TB database would be switched to the new replica set. At that point, he would drop the 1TB database in the original replica set and drop the 5TB databases in the new replica set.

My requirement was that he would have at least 2 available data bearing nodes of each replica set so he would not lose high availability.

The Steps

  1. add 2 new nodes to the existing replica set and wait for an initial sync to complete
  2. remove the 2 new members from the replica set
  3. start each node as a standalone and remove the local database from each
  4. shutdown one of the nodes and restart with the new --replSet name. This node will become the primary.
  5. run rs.initiate() on this node. We now have a single node replica set with all of the data.
  6. export the local database from the new replica set with the mongodump command e.g.
    mongodump --port primary-node-port-number --db local
  7. import the local database dump into the second node using the mongorestore command e.g. mongorestore --port second-node-port-number --db local dump/local
  8. shut down the second node and restart with the new --replSet name
  9. run the rs.add() command from your primary (created in step #5) to add the second node to the replica set
  10. an initial sync process will be initiated but since the oplog shows both to be at the same point, the second node should transition to SECONDARY almost immediately.

The Issues

  1. mongodump of the local database does not include the collection system.replset. This is a known issue recorded in TOOLS-1387. Although I think that should be reconsidered it is not the topic of this SERVER ticket.
  1. The replset.minvalid collection is updated by the server with a new timestamp that will not match any timestamp in the primary's oplog.

Workarounds

  1. In order to workaround the issue of system.replset being missed, I tried to dump that collection separately using:

    mongodump --port port-number --db local --collection system.replset

  1. In order to reverse the effect of the replset.minvalid collection being changed, I run:

    use local
    db.replset.minvalid.update({}, {$set: {ts:Timestamp(0,0)}})

Summary

The steps above allowed me to achieve my goals. The fact that the server modified data in the local database when the local database was imported into a standalone server was unexpected and should be fixed albeit as a low priority.

Note: a script to reproduce this issue is attached



 Comments   
Comment by Spencer Brody (Inactive) [ 19/Jan/18 ]

arnie.listhaus, agreed that this is weird behavior, however this also isn't a supported way to start a new replica set, so I'm reluctant to spend any more time on this issue currently.

Another way to do what you're trying to do (although this also isn't technically a supported procedure, so no promises that this won't break in the future), I believe, would be to change step 3 from dropping the local database to dropping all collections in the local database except the oplog. Although I haven't actually tested this procedure, so it's possible that that may not work. If that doesn't work, another idea (also untested) would be to just seed the oplog on the second node with the most recent oplog entry from the other node, rather than trying to dump/restore the entire local database.

Comment by Judah Schvimer [ 10/Jan/18 ]

********************Finding minValid on source before restore
MongoDB shell version v0.0.0
connecting to: mongodb://127.0.0.1:28017/
MongoDB server version: 0.0.0
{ "_id" : ObjectId("5a5677e48235942dd0e804ea"), "t" : NumberLong(-1), "ts" : Timestamp(0, 0) }
********************Found minValid on source before restore
********************Finding minValid before restore
MongoDB shell version v0.0.0
connecting to: mongodb://127.0.0.1:28018/
MongoDB server version: 0.0.0
********************Found minValid before restore
2018-01-10T15:30:40.367-0500	the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2018-01-10T15:30:40.368-0500	building a list of collections to restore from dump/local dir
2018-01-10T15:30:40.386-0500	reading metadata for local.startup_log from dump/local/startup_log.metadata.json
2018-01-10T15:30:40.386-0500	reading metadata for local.oplog.rs from dump/local/oplog.rs.metadata.json
2018-01-10T15:30:40.386-0500	restoring local.startup_log from dump/local/startup_log.bson
2018-01-10T15:30:40.389-0500	reading metadata for local.system.replset from dump/local/system.replset.metadata.json
2018-01-10T15:30:40.389-0500	reading metadata for local.replset.election from dump/local/replset.election.metadata.json
2018-01-10T15:30:40.395-0500	no indexes to restore
2018-01-10T15:30:40.395-0500	finished restoring local.startup_log (1 document)
2018-01-10T15:30:40.395-0500	reading metadata for local.replset.minvalid from dump/local/replset.minvalid.metadata.json
2018-01-10T15:30:40.708-0500	restoring local.system.replset from dump/local/system.replset.bson
2018-01-10T15:30:40.791-0500	restoring local.replset.election from dump/local/replset.election.bson
2018-01-10T15:30:40.833-0500	restoring local.oplog.rs from dump/local/oplog.rs.bson
2018-01-10T15:30:40.905-0500	restoring local.replset.minvalid from dump/local/replset.minvalid.bson
2018-01-10T15:30:40.906-0500	no indexes to restore
2018-01-10T15:30:40.907-0500	finished restoring local.system.replset (1 document)
2018-01-10T15:30:40.907-0500	no indexes to restore
2018-01-10T15:30:40.907-0500	finished restoring local.replset.election (1 document)
2018-01-10T15:30:40.907-0500	reading metadata for local.me from dump/local/me.metadata.json
2018-01-10T15:30:40.907-0500	no indexes to restore
2018-01-10T15:30:40.907-0500	finished restoring local.oplog.rs (6 documents)
2018-01-10T15:30:40.909-0500	no indexes to restore
2018-01-10T15:30:40.909-0500	finished restoring local.replset.minvalid (1 document)
2018-01-10T15:30:40.993-0500	restoring local.me from dump/local/me.bson
2018-01-10T15:30:40.997-0500	no indexes to restore
2018-01-10T15:30:40.997-0500	finished restoring local.me (1 document)
2018-01-10T15:30:40.997-0500	done
********************Finding minValid after restore
MongoDB shell version v0.0.0
connecting to: mongodb://127.0.0.1:28018/
MongoDB server version: 0.0.0
{ "_id" : ObjectId("5a5677e48235942dd0e804ea"), "t" : NumberLong(-1), "ts" : Timestamp(1515616240, 1) }
********************Found minValid after restore

Comment by Judah Schvimer [ 10/Jan/18 ]

I've done a bit of debugging on this and I see what's happening by adding the following around the restore call:

mongo --port 28018 --eval " db.setLogLevel(3, 'replication'); db.getSiblingDB('local').replset.minvalid.find();"
mongorestore --port 28018 --db local dump/local
mongo --port 28018 --eval "db.getSiblingDB('local').replset.minvalid.find();"

The minvalid certainly is getting set but with some extra log lines I don't see anything explicitly setting it. It's definitely strange.

Comment by Vick Mena (Inactive) [ 08/Jan/18 ]

An alternate workflow would be to use mongo-connector to mirror the database in question only and allow the application to seamlessly transition from one replica to another with zero downtime. A better approach would be to have mongomirror support non-Atlas targets and improve it's mirroring capabilities.

Generated at Thu Feb 08 04:30:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.