[SERVER-17473] Deleting index on primary doesnt delete on secondary Created: 05/Mar/15  Updated: 16/Mar/15  Resolved: 16/Mar/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.6.8
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: prashant deva Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File diffindexes.js    
Operating System: ALL
Participants:

 Description   

I have an index of type unqiue=true. I know for sure we deleted it on a primary in our 3 node cluster.
Recently our primary changed and we saw the deleted index was still present on the new primary (which was previously a secondary)



 Comments   
Comment by Ramon Fernandez Marina [ 16/Mar/15 ]

pdeva, I was wondering whether your primary changed before the index drop operation got replicated to the secondary. I tried to reproduce that scenario on my end, unsuccessfully. And without detailed logs from your setup there's not enough information for us to investigate further.

What I would suggest is that, if this happens again, you collect the indexes from both primary and secondaries as you did above and upload full logs for all the nodes involved. Until then, I'm going to resolve this ticket.

Regards,
Ramón.

Comment by prashant deva [ 12/Mar/15 ]

ok rs.slaveOk() worked.

output:

dsmongo-5 (secondary)

dsmongo:SECONDARY> db.accounts.getIndexes()
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "dripstat.accounts"
	},
	{
		"v" : 1,
		"unique" : true,
		"key" : {
			"licenseKey" : 1
		},
		"name" : "licenseKey",
		"ns" : "dripstat.accounts",
		"dropDups" : true,
		"sparse" : false
	}
]
dsmongo:SECONDARY> 

dsmongo-6 - secondary

dsmongo:SECONDARY> db.accounts.getIndexes()
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "dripstat.accounts"
	},
	{
		"v" : 1,
		"unique" : true,
		"key" : {
			"licenseKey" : 1
		},
		"name" : "licenseKey",
		"ns" : "dripstat.accounts",
		"dropDups" : true,
		"sparse" : false
	}
]
dsmongo:SECONDARY> 

dsmongo-4 - primary

dsmongo:PRIMARY> db.accounts.getIndexes()
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "dripstat.accounts"
	},
	{
		"v" : 1,
		"unique" : true,
		"key" : {
			"licenseKey" : 1
		},
		"name" : "licenseKey",
		"ns" : "dripstat.accounts",
		"dropDups" : true,
		"sparse" : false
	}
]
dsmongo:PRIMARY> 

so at the moment, the indexes do seem fine.
but i can guarantee we an index on field 'email' that was unique=true.
when we swtiched primary, we found that the secondary (which is now primary) had that index even though we deleted it from previous primary.
we found lost data, cause the accounts collection in this secondarry (now primary) couldnt sync the new account documents we created in previous primary.

Comment by Eliot Horowitz (Inactive) [ 12/Mar/15 ]

Call rs.slaveOk() first.

Comment by prashant deva [ 12/Mar/15 ]

i even tried logging in as root user in the mongo shell, but this is what i get:

dsmongo:SECONDARY> db.accounts.getIndexes()
2015-03-12T06:17:00.008+0000 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131
dsmongo:SECONDARY> 

Comment by prashant deva [ 12/Mar/15 ]

also it seems the reason your script doesnt work is cause rs.status() results in the following:

{
	"ok" : 0,
	"errmsg" : "not authorized on admin to execute command { replSetGetStatus: 1.0 }",
	"code" : 13
}

Comment by prashant deva [ 12/Mar/15 ]

i am not sure how to run that query on a secondary.

i tried using the mongoshell but got this:

dsmongo:SECONDARY> use dripstat
switched to db dripstat
dsmongo:SECONDARY> db.accounts.getIndexes()
2015-03-12T06:00:48.106+0000 error: {
	"$err" : "not authorized for query on dripstat.system.indexes",
	"code" : 13
} at src/mongo/shell/query.js:131
dsmongo:SECONDARY> 

Comment by Ramon Fernandez Marina [ 10/Mar/15 ]

Given the issues with diffindex.js can you please run:

db.collection.getIndexes()

for the database and collection affected by this issue? Please run it on all nodes and post the output. Please also post full logs as requested above.

Thanks,
Ramón.

Comment by prashant deva [ 10/Mar/15 ]

please can we get a reply on this issue

Comment by prashant deva [ 07/Mar/15 ]

can you please recommend what to do.
we are seeing data inconsistency issues due to this since the deleted index was unique

Comment by prashant deva [ 06/Mar/15 ]

ok this is what i get now:

ubuntu@dsmongo-5:/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.6.8/bin$ ./mongo --port 27000 < ~/diff.js 
MongoDB shell version: 2.6.8
connecting to: 127.0.0.1:27000/test
2015-03-06T18:20:58.929+0000 TypeError: Cannot call method 'forEach' of undefined
bye
ubuntu@dsmongo-5:/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.6.8/bin$ 

Comment by Ramon Fernandez Marina [ 06/Mar/15 ]

You can use mongo --port <number> to specify the port your mongod is running on. If it's 27000, can you please run

mongo --port 27000 < ~/diff.js

and post the output?

Comment by prashant deva [ 06/Mar/15 ]

this is what i get:

 
ubuntu@dsmongo-5:/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.6.8/bin$ ./mongo < ~/diff.js 
MongoDB shell version: 2.6.8
connecting to: test
2015-03-06T12:02:16.057+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-03-06T12:02:16.058+0000 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
exception: connect failed

i think its cause this instance is deployed using mms service and is using port 27000

Comment by Ramon Fernandez Marina [ 05/Mar/15 ]

You can run it on any node, and it's safe to run it in a production system. If you have any concerns then please run it against one of your secondaries.

Thanks,
Ramón.

Comment by prashant deva [ 05/Mar/15 ]

The version is 2.6.8.

Do I run it on a primary or secondary?
Is it safe to run it on our production db?


Prashant

Comment by Ramon Fernandez Marina [ 05/Mar/15 ]

pdeva, can you please run the attached diffindexes.js script on your system and post the resulting output?

mongo < diffindexes.js

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 05/Mar/15 ]

pdeva, what version of MongoDB are you using? Also, can you please upload the logs of the former and the new primary nodes?

Thanks,
Ramón.

Generated at Thu Feb 08 03:44:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.