[SERVER-7888]  E11000 duplicate key error index while stopping the Balancer Created: 10/Dec/12  Updated: 08/Mar/13  Resolved: 24/Feb/13

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Michael Tewner Assignee: David Hows
Resolution: Incomplete Votes: 0
Labels: DuplicateKey, balancer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 64-bit
2 shards of 3 servers each ( Primary/Secondary/Arbiter )


Attachments: Text File mongos.log    
Issue Links:
Related
related to SERVER-4532 GetLastError on sharded cluster can r... Closed
Operating System: ALL
Participants:

 Description   

Our MongoDB Backup script (python) disables the balancer using:

self.config_db.set_lasterror_options(fsync=True)
self.config_db.settings.update(

{ '_id' : "balancer" }

, { '$set' :

{ 'stopped' : True}

}, True, False, True)

About 30% of the time, the following exception is thrown:

06:41:18 Traceback (most recent call last):
06:41:18 File "/opt/XXXX/bin/mongo.backup.py", line 398, in <module>
06:41:18 balancer.disableBalancer()
06:41:18 File "/opt/XXXX/bin/mongo.backup.py", line 47, in disableBalancer
06:41:18 self.__setBalancer (True)
06:41:18 File "/opt/XXXX/bin/mongo.backup.py", line 43, in __setBalancer
06:41:18 self.config_db.settings.update(

{ '_id' : "balancer" }

, { '$set' :

{ 'stopped' : bol }

}, True, False, True)
06:41:18 File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 411, in update
06:41:18 check_keys, self._uuid_subtype), safe)
06:41:18 File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 831, in _send_message
06:41:18 rv = self.__check_response_to_last_error(response)
06:41:18 File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 778, in __check_response_to_last_error
06:41:18 raise DuplicateKeyError(details["err"])
06:41:18 pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: XXXX.m_generics.$id dup key: { : "11f1952c-55b0-474b-950e-f03c54d538b5" }

After a few attempts, the script succeeds in disabling the balancer.
Any ideas of how to handle this? Does this point to a bigger issue (are there really duplicate keys somehow?)



 Comments   
Comment by David Hows [ 18/Dec/12 ]

Hi Michael,

Given what you have said I would suspect SERVER-4532.

Especially given that the error is from an unrelated collection Database.m_sequences.

Can you upload the mongos logs and if possible the logs of both primaries? I would like to try and confirm that this is the case.

Cheers,

David

Comment by Michael Tewner [ 17/Dec/12 ]

I should add, though, that I was mistaken - The environment is made up of 2 shards of replica sets, but each replica set only has a Primary node. Could that be relevant?

Comment by Michael Tewner [ 16/Dec/12 ]

Is it possible that I'm getting

{ 'err': None }

?
i'm not sure this is the output you're looking for. How do get the required info using PyMongo?

11:34:20

{u'ok': 1.0, u'err': None, u'singleShard': u'devshare01srv02:20001,devshare02srv02:20001,devshare04:20001', u'connectionId': 11942, u'n': 1, u'updatedExisting': True, u'waited': 13}

11:34:20 Traceback (most recent call last):
11:34:20 File "/opt/XXXX/bin/mongo.backup.py", line 400, in <module>
11:34:20 balancer.disableBalancer()
11:34:20 File "/opt/XXXX/bin/mongo.backup.py", line 49, in disableBalancer
11:34:20 self.__setBalancer (True)
11:34:20 File "/opt/XXXX/bin/mongo.backup.py", line 43, in __setBalancer
11:34:20 result = self.config_db.settings.update(

{ '_id' : "balancer" }

, { '$set' :

{ 'stopped' : bol }

}, True, False, True)
11:34:20 File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 411, in update
11:34:20 check_keys, self._uuid_subtype), safe)
11:34:20 File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 831, in _send_message
11:34:20 rv = self.__check_response_to_last_error(response)
11:34:20 File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 778, in __check_response_to_last_error
11:34:20 raise DuplicateKeyError(details["err"])
11:34:20 pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: Database.m_sequences.$id dup key: { : "SHARED_CONTENT" }

Comment by David Hows [ 14/Dec/12 ]

Hi Michael,

Would you be able to output the whole structure of the getLastError object and attach to the ticket?

Cheers,

David

Generated at Thu Feb 08 03:15:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.