[SERVER-3866] reIndex() on secondary drops all indexes Created: 14/Sep/11  Updated: 11/Jul/16  Resolved: 15/Sep/11

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.0.0
Fix Version/s: 1.8.4, 2.0.1, 2.1.0

Type: Bug Priority: Blocker - P1
Reporter: Mitchell Hashimoto Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 1
Labels: indexing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

This is huge issue.

I ran reIndex on one of our collections and this was the output:

{ 
        "nIndexesWas" : 2, 
        "msg" : "indexes dropped for collection", 
        "errmsg" : "exception: no index name specified", 
        "code" : 12523, 
        "ok" : 0 
} 

After doing this and calling getIndexes and empty list is returned. Our indexes are indeed gone.

Luckily we ran this on a secondary machine, but this is a huge problem.

This issue does not affect reindexing on a primary. Of course one typically does not reindex on a primary.

Workaround: Start the target mongod without "--replSet" and on a different port, reindex, and then restart mongod again with its normal --replSet command line. Be sure to use a different port number so no traffic other than administrative hits the machine during this maintenace procedure.



 Comments   
Comment by auto [ 14/Oct/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: backport of SERVER-3866
Branch: v1.8
https://github.com/mongodb/mongo/commit/58a3ee1f9e39877ffee65564697c43ac647b001d

Comment by auto [ 15/Sep/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: fix reIndex on secondaries SERVER-3866
Branch: v2.0
https://github.com/mongodb/mongo/commit/692f4ec8dfbb75c5d2f5a28b70231f8f95d4b95e

Comment by Scott Hernandez (Inactive) [ 14/Sep/11 ]

If you have lost your indexes you can do a "mongodump -d <dbname>" to create a dump of the index definitions from the primary, and then restore them on the secondary. In order to do a mongorestore of the indexes you will need to restart the secondary without --replSet and on a different port (so it does not confuse the other replica set members). Before you restore the indexes please make sure that you have enough oplog to last the time it takes for all index creation. You may want to restore the indexes in batches so that replication can catch up between the index builds.

To avoid the problem all together, make sure to start up without "--replSet" and on a different port, before you do the reIndex on the secondary node.

Comment by auto [ 14/Sep/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Add test for reindexing on secondaries: SERVER-3866.
Branch: master
https://github.com/mongodb/mongo/commit/19a94b8d2db9d2d028cdedba965dc4cae127a692

Comment by auto [ 14/Sep/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: fix reIndex on secondaries SERVER-3866
Branch: master
https://github.com/mongodb/mongo/commit/b88b810d28fc0060e3405702dcb492dfe9355af2

Comment by Spencer Brody (Inactive) [ 14/Sep/11 ]

I have successfully reproduced the problem. It happens whenever calling a reindex on a secondary. This problem is not new to 2.0, but is just being discovered now as people are trying to reindex all their databases to upgrade to the new index format.

Generated at Thu Feb 08 03:04:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.