System:

The system in place includes a Rails based application using MongoMapper as the interface to the db. The mongo instances include a Primary and Secondary with an arbiter on small VM.

Background:

We are using mongo to maintain a two lists of people that is the base of a checkout system. One collection named "campaign_ID" is the source of people the comprises the list of people who are available for a checkout. A second collection named "available_ID" maintains the same list of people but this collection is actually modified using a findAndRemove() call to remove the person who has been checked out.

We were experiencing "double checkouts" caused by the same person being placed into the available_ID collection multiple times. The solution executed was to change the index on the collection to include :unique and :dropDups on the person_id attribute of the document.

The solution appears to work properly has the count() and length of a distinct() match on both the available_ID and campaign_ID collections.
However, at some point during replication we are receiving a duplicate key error on the person_id index. The current work around we have employed is to reIndex() the collections on the Secondary, which results in dropping all indexes. Not an ideal situation for fail-over but it does allow the replication to begin working again and will eventually complete it's sync.

A second situation that may have a part in producing the error may occur when these collections are "refreshed" on a nightly basis. This refresh happens through a rake task kicked off by a request to the Rails application. This task creates a temp_ID collection and then uses mongo_import to to fill the collection with fresh data. Indexes are created on this temp collection. The current collections are then renamed campaign_old_ID and available_old_ID and the temp are renamed the standard campaign_ID and available_ID.

The Question:

How can we eliminate the duplicate key error when replicating to the Secondary? What could cause an index to exist but not be respected on the Secondary server?

The error message:

syncThread: 11000 E11000 duplicate key error index: vcs.available_NV2301.$person_id_1 dup key: { : "28FCCB8F-E05A-4B2A-9418-66F8194BBC2A" }
syncThread: 11000 E11000 duplicate key error index: vcs.available_MD1898.$person_id_1 dup key: { : "2A0F7EC6-B6EB-4A0B-AC56-F9BFC261EF40" }

The indexes on the collections:

vcsrep:PRIMARY> db.available_NV2301.getIndexes();
[
{
"name" : "id",
"ns" : "vcs.available_NV2301",
"key" :

{ "_id" : 1 }

,
"v" : 0
},
{
"name" : "loc_2d_after_-1",
"ns" : "vcs.available_NV2301",
"key" :

{ "loc" : "2d", "after" : -1 }

},
{
"name" : "id_1_after-1",
"ns" : "vcs.available_NV2301",
"key" :

{ "_id" : 1, "after" : -1 }

,
"v" : 0
},
{
"name" : "person_id_1",
"ns" : "vcs.available_NV2301",
"key" :

{ "person_id" : 1 }

,
"unique" : true,
"drop_dups" : true,
"dropDups" : true,
"v" : 0
}
]