[SERVER-11540] db.repairDatabase() doesn't work on Secondarys Created: 04/Nov/13  Updated: 10/Dec/14  Resolved: 07/Mar/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Steffen Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 12.04.3 LTS
MongoDB 2.4.6


Issue Links:
Duplicate
duplicates SERVER-11391 repairDatabase should be disallowed o... Closed
duplicates SERVER-2264 make repairDatabase work on secondary... Closed
Operating System: ALL
Steps To Reproduce:

On primary create DB with test collection.

> use testDB
> db.testCollection.insert({ test : 1})

On secondary go to DB an try to run Repair command

> use testDB
switched to db testDB
> rs.slaveOk()
> db.repairDatabase()
{
    "ok" : 0,
    "errmsg" : "clone failed for testDB with error: namespace query failed testDB.system.namespaces"
}

Participants:

 Description   

We try to run Repair in our ReplicaSet to improve disk usage.
We want to repair each secondary and then after stepDown the last primary.

If we try to run repair on a secondary we receive an error:

> db.repairDatabase()
{
    "ok" : 0,
    "errmsg" : "clone failed for testDB with error: namespace query failed testDB.system.namespaces"
}

We used the mongo shell and set rs.slaveOk(). We also wrote a python script to do this task, which fails with the same Error.
RepairDatabase() works on a single mongod without replicatset config.

Some more debugging:

> use testDB
> db.system.namespaces.find();
{ "name" : "testDB.system.indexes" }
{ "name" : "testDB.testCollection.$_id_" }
{ "name" : "testDB.testCollection" }
> db.createCollection( "testDB.system.namespaces" );
{ "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" }
> rs.slaveOk()
> db.createCollection( "testDB.system.namespaces" );
{ "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" }

We ran the createCollection command on the Primary before as well, but still don't see the namespace in the listing.
If we run the command on the primary again

> db.createCollection( "testDB.system.namespaces" );
{ "ok" : 0, "errmsg" : "collection already exists" }



 Comments   
Comment by Kevin J. Rice [ 14/Mar/14 ]

I'd like to upvote what Eric Milkie said. That is, the concept of making repairDatabase() work like compact(). I'm having to script a solution to repairDatabase() so I can reclaim disk space.

Doing this via a script requires, for each of our 48 shards running on 4 different boxes:

  • connecting to the right box name for a specific shard
  • actually finding and killing the correct mongod process
  • restarting a mongod with the correct db path, etc., but without --replSet xxxx and on a different port
  • connecting to that port
  • running db.repairDatabase(),
  • detecting that it's done
  • killing the mongod process I just started;
  • restarting the original mongod process correctly with the right port/replset/etc.,

Obviously, just being able to connect to a secondary and run db.repairDatabase() would be FAR easier.

Comment by Eric Milkie [ 05/Mar/14 ]

We could make repairDatabase() work the same way that compact() does on secondaries, by setting maintenance mode.

Comment by Steffen [ 04/Nov/13 ]

We would like to be able to run repair on secondarys.
This is a maintenance task for us, which we want to do frequently and automated (via a python script).

If we run the repair on the primary, the primary would go into maintenance and a fail over will happen. Another secondary will be elected as primary. Thus the old primary becomes a secondary repairing the database.

Comment by J Rassi [ 04/Nov/13 ]

> But I don't see the point to be able to do it just on a primary in a repset, which will become a secondary. Thus, how can I change this to a feature request?

Can you explain more the behavior you'd like to see? I'm not able to understand exactly what you mean.

Comment by Steffen [ 04/Nov/13 ]

Hello.
Thanks for clarifying.

I see that it makes sense for a single host. But I don't see the point to be able to do it just on a primary in a repset, which will become a secondary.
Thus, how can I change this to a feature request?

Thanks

Comment by J Rassi [ 04/Nov/13 ]

Running write operations on secondaries is disallowed. See the repairDatabase documentation: "to run repair on a secondary/slave restart the instance in standalone mode without the --replSet or --slave options." See the rolling index build tutorial for more detailed instructions on temporarily bringing a replica set member into standalone mode.

Generated at Thu Feb 08 03:26:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.