[SERVER-40061] Chunk move fails due to DuplicateKey error on the `config.chunks` collection at migration commit Created: 10/Mar/19  Updated: 09/Aug/19  Resolved: 09/Aug/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aristarkh Zagorodnikov Assignee: Kaloian Manassiev
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-8829 String representation for chunk id is... Closed
Operating System: ALL
Sprint: Sharding 2019-05-20, Sharding 2019-06-03, Sharding 2019-06-17, Sharding 2019-07-01, Sharding 2019-07-15, Sharding 2019-07-29, Sharding 2019-08-12
Participants:

 Description   

The ChunkType::genID method uses the BSONElement::toString method, which was changed to provide a better formatting for UUID BinData. Unfortunately, the ChunkType::genID is used all around sharding-related code as a value of "_id" field in "config.chunks" collection. When the chunk minimum field has a value that is an UUID, the value of "_id" for v3.6 and v3.4 (and previous versions) differ.

We've hit it when trying to move chunks manually in a cluser we recently moved from v3.4 to v3.6:

2019-03-10T16:54:39.264+0300 I COMMAND  [conn469729] command admin.$cmd appName: "MongoDB Shell" command: _configsvrMoveChunk { _configsvrMoveChunk: 1, _id: "a.fs.chunks-files_id_UUID("05660000-0000-e000-96c2-dc81ca6fa911")n_0", ns: "a.fs.chunks", min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, shard: "driveFS-2", lastmod: Timestamp(571033, 1), lastmodEpoch: ObjectId('51793868331d54dfcf8e0032'), toShard: "driveFS-17", maxChunkSizeBytes: 536870912, secondaryThrottle: {}, waitForDelete: false, writeConcern: { w: "majority", wtimeout: 15000 }, lsid: { id: UUID("605f316b-6296-4010-9f26-835b60f923ff"), uid: BinData(0, EE3A53D0CA965E6112DBEBF842D31DC81E8CE7E7548256DE28D08422B2C59D3B) }, $replData: 1, $clusterTime: { clusterTime: Timestamp(0, 0), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId:0 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.9" }, os: { type: "Windows", name: "Microsoft Windows 10", architecture: "x86_64", version: "10.0 (build 17134)" }, mongos: { host: "dorado2:27017",client: "10.254.3.70:1334", version: "3.6.10" } }, $configServerState: { opTime: { ts: Timestamp(1552225736, 38), t: 95 } }, $db: "admin" } exception: Chunk move was not successful due to E11000 duplicate key error collection: config.chunks index: ns_1_min_1 dup key: { : "a.fs.chunks", : { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 } } code:DuplicateKey numYields:0 reslen:562 locks:{ Global: { acquireCount: { r: 10, w: 6 } }, Database: { acquireCount: { r: 2, w: 6 } }, Collection: { acquireCount: { r: 2,w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_msg 340766ms

Of course, the "config.chunks" collection contains this:

> db.chunks.find({ns:"a.fs.chunks",min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 } })
{ "_id" : "a.fs.chunks-files_id_BinData(4, 056600000000E00096C2DC81CA6FA911)n_0", "lastmod" : Timestamp(539637, 1290), "lastmodEpoch" : ObjectId("51793868331d54dfcf8e0032"), "ns" : "a.fs.chunks", "min" : { "files_id" : UUID("05660000-0000-e000-96c2-dc81ca6fa911"), "n" : 0 }, "max" : { "files_id" : UUID("05666100-0000-e000-9252-7b82dea0b186"), "n" : 3 }, "shard" : "driveFS-2" }

Since I do not know what other operations are using the "_id" field, I cannot estimate the true potential of this problem, but cursory inspection of the codebase shows there are at least some places where the update is performed without checking the number of matched/modified documents, so it may be the case that the metadata (I mean the chunk structure) could be lost/damaged silently.



 Comments   
Comment by Kaloian Manassiev [ 09/Aug/19 ]

chenjian@tmxmall.com, let's continue the conversation in SERVER-42734.

Comment by Kaloian Manassiev [ 09/Aug/19 ]

We "fixed" the issue by stopping the balancer and rewriting the chunks collection manually, fixing the IDs.

onyxmaster, these are the correct workaround steps for the situation that your cluster ended up in. This is a side effect of SERVER-8829, which causes the _ids of the chunks collection to be string representation of the shard key's BSON, which is not always unique.

We recently fixed it in the current master branch (4.3 as of now) and will be exploring whether it is possible to backport it to earlier releases.

Apologies again for the inconvenience and for the time it took.

Comment by Chen Jian [ 09/Aug/19 ]

Thank you, here is the new ticket: SERVER-42734

Comment by Chen Jian [ 09/Aug/19 ]

Thank you for your reply, at the beginning, our problem like this Issue: SERVER-11421, we got the error log and it didn't stop, and make our disk out of memory.

Now one of the primary shard is down and can not start again, could tell me the method to start the stopped shard?

Comment by Aristarkh Zagorodnikov [ 09/Aug/19 ]

We "fixed" the issue by stopping the balancer and rewriting the chunks collection manually, fixing the IDs.

Comment by Kaloian Manassiev [ 09/Aug/19 ]

Hi chenjian@tmxmall.com,

My investigation so far has concluded that the issue originally reported here is related to SERVER-8829, which causes two chunks to have duplicated _id value, which fails the committing of the migrated chunk on the primary, but never makes it into the oplog. Also it seems to be specific to GridFS chunks which have UUIDs as shard keys and I am currently working on finding out a workaround.

The case that you reported seems to be happening on the routing metadata cache collections and during replication startup recovery, so it is likely a different issue. Would it be possible to create a separate ticket and to include the complete latest mongod log of the node you are trying to start?

Best regards,
-Kal.

Comment by Chen Jian [ 09/Aug/19 ]

Thank you for your reply, at the beginning, our promble like this Issue:https://jira.mongodb.org/browse/SERVER-11421, we got the error log and it didn't stop, and make our disk out of memory.
 
Now one of the primary shard is down and can not start again, could tell me the method to start the stopped shard?

Comment by Chen Jian [ 09/Aug/19 ]

I also find this promble in version 4.0.9. I can't start my shard, and I get the following error information:

[repl writer worker 5] writer worker caught exception: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: \{ : { files_id: -8792944871971349461 } } on: \{ op: "i", ns: "config.cache.chunks.pr_tmxbase.toffs_6.chunks", ui: UUID("03c48aff-f98b-4de3-bb08-3248ab7b588f"), o: { _id: { files_id: -8792944871971349461 }, max: \{ files_id: -8700326615160610030 }, shard: "prodShard1", lastmod: Timestamp(98, 0), history: [ \{ validAfter: Timestamp(1564563326, 26), shard: "prodShard1" } ] }, ts: Timestamp(1565335484, 137), t: 7, h: -6045188019592284924, v: 2, wall: new Date(1565335484859) }
2019-08-09T18:50:07.622+0800 F REPL [initandlisten] Failed to apply batch of operations. Number of operations in batch: 4229. First operation: \{ op: "n", ns: "", o: { msg: "new primary" }, ts: Timestamp(1565335398, 2), t: 7, h: -153190637679831346, v: 2, wall: new Date(1565335398497) }. Last operation: \{ op: "i", ns: "pr_tmxbase.stfs_2.chunks", ui: UUID("78d50f72-a565-4a31-be18-2354f730743f"), o: { _id: ObjectId('5d4d1fc44ef55e38430807eb'), files_id: "3f05ae47e29a0704c7407a8c3bdf9df4", n: 23, data: BinData(0, 693D22687474703A2F2F7777772E77332E6F72672F323030312F584D4C536368656D612D696E7374616E63652220786D6C6E733A7873643D22687474703A2F2F7777772E7733...) }, ts: Timestamp(1565335492, 11), t: 7, h: -791270771233656130, v: 2, wall: new Date(1565335492057) }. Oplog application failed in writer thread 11: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: \{ : { files_id: -8792944871971349461 } }
2019-08-09T18:50:07.624+0800 F REPL [initandlisten] Caught exception during replication recovery: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: \{ : { files_id: -8792944871971349461 } }
2019-08-09T18:50:07.625+0800 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information
2019-08-09T18:50:07.634+0800 F - [initandlisten] DBException::toString(): DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: \{ : { files_id: -8792944871971349461 } }
Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)11000, mongo::AssertionException>
 0x5634a5f505e1 0x5634a5f4ffc5 0x5634a6044a46 0x5634a6044a91 0x5634a4841c3f 0x5634a487bf1c 0x5634a487c7e7 0x5634a45be20c 0x5634a45c09ff 0x5634a45522e9 0x7fb3ab98e830 0x5634a45bb3d9
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"5634A3B34000","o":"241C5E1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"5634A3B34000","o":"241BFC5"},\{"b":"5634A3B34000","o":"2510A46","s":"_ZN10__cxxabiv111__terminateEPFvvE"},\{"b":"5634A3B34000","o":"2510A91"},\{"b":"5634A3B34000","o":"D0DC3F","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE"},\{"b":"5634A3B34000","o":"D47F1C","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},\{"b":"5634A3B34000","o":"D487E7","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},\{"b":"5634A3B34000","o":"A8A20C"},\{"b":"5634A3B34000","o":"A8C9FF","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},\{"b":"5634A3B34000","o":"A1E2E9","s":"main"},\{"b":"7FB3AB96E000","o":"20830","s":"__libc_start_main"},\{"b":"5634A3B34000","o":"A873D9","s":"_start"}],"processInfo":\{ "mongodbVersion" : "4.0.9", "gitVersion" : "fc525e2d9b0e4bceff5c2201457e564362909765", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-91-generic", "version" : "#114-Ubuntu SMP Tue Aug 8 11:56:56 UTC 2017", "machine" : "x86_64" }, "somap" : [ \{ "b" : "5634A3B34000", "elfType" : 3, "buildId" : "D363637D8649A86A028650E1AE31DD0CCF80723C" }, \{ "b" : "7FFFA4FE6000", "elfType" : 3, "buildId" : "735D8C7F1005D2D0E443E551BEA48FD5091211DF" }, \{ "b" : "7FB3AD148000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5C1A06A89F89E1ADAAA507BC5580C0A7931B0AB2" }, \{ "b" : "7FB3ACF2D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, \{ "b" : "7FB3ACAE9000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B467957BF7E767DD5C0CA92213C2DAF0182765D2" }, \{ "b" : "7FB3AC880000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "5A9B9023B6E9ABCEE21DADDB95EF096F505E9E67" }, \{ "b" : "7FB3AC67C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, \{ "b" : "7FB3AC474000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, \{ "b" : "7FB3AC16B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, \{ "b" : "7FB3ABF55000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, \{ "b" : "7FB3ABD38000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, \{ "b" : "7FB3AB96E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, \{ "b" : "7FB3AD3B7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }, \{ "b" : "7FB3AB73B000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, \{ "b" : "7FB3AB51F000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, \{ "b" : "7FB3AB2D5000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54EB3E674DB58B5BAEF009CBC4AB02CCF5386CA7" }, \{ "b" : "7FB3AB0C6000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "213C3A6756470FF961EE7FD63BAC177BC6B72A49" }, \{ "b" : "7FB3AAE75000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "698C15E7BD2CD96618583CADC6EBCE07B293CEB2" }, \{ "b" : "7FB3AAC5B000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, \{ "b" : "7FB3AA92B000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "213041D960C027C8C451F53B9B3143A901322015" }, \{ "b" : "7FB3AA6F8000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, \{ "b" : "7FB3AA4C2000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, \{ "b" : "7FB3AA242000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, \{ "b" : "7FB3A9F70000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "D60D5A759577E5C477314520EF3D13BDBAED5111" }, \{ "b" : "7FB3A9D41000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "09FA442048F4AA6E701F0B504FEB688F09618E30" },\{ "b" : "7FB3A9B3D000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, \{ "b" : "7FB3A9932000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "00DFFD025B1AEBFD619FAE9DE01610BA6F209367" }, \{ "b" : "7FB3A9717000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, \{ "b" : "7FB3A94D6000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, \{ "b" : "7FB3A9271000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "2F170DCF36786FE91EB82D547A42B2DEE2E9DF81" }, \{ "b" : "7FB3A905E000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "A6EE7754728D688B04095ACB34F1A071F99ED6C3" }, \{ "b" : "7FB3A8E5A000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, \{ "b" : "7FB3A8C51000", "path": "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, \{ "b" : "7FB3A89C7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, \{ "b" : "7FB3A8725000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, \{ "b" : "7FB3A84F2000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, \{ "b" : "7FB3A82DC000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, \{ "b" : "7FB3A80D4000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, \{ "b" : "7FB3A7EAB000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, \{ "b" : "7FB3A7C9C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, \{ "b" : "7FB3A7A51000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, \{ "b" : "7FB3A777C000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, \{ "b" : "7FB3A7544000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7BDD51353D50310FFA1587E4AA01B40ABE32D582" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5634a5f505e1]
 mongod(+0x241BFC5) [0x5634a5f4ffc5]
 mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x5634a6044a46]
 mongod(+0x2510A91) [0x5634a6044a91]
 mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE+0xC7F) [0x5634a4841c3f]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x54C) [0x5634a487bf1c]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x237) [0x5634a487c7e7]
 mongod(+0xA8A20C) [0x5634a45be20c]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xEDF) [0x5634a45c09ff]
 mongod(main+0x9) [0x5634a45522e9]
 libc.so.6(__libc_start_main+0xF0) [0x7fb3ab98e830]
 mongod(_start+0x29) [0x5634a45bb3d9]
----- END BACKTRACE -----

Comment by Aristarkh Zagorodnikov [ 05/Jul/19 ]

It appears that automatic migrations of chunks that have UUIDs as keys doesn't work too. The chunk migrates, but commit fails with this:

Chunk move failed :: caused by :: DuplicateKey: Chunk move was not successful due to E11000 duplicate key error collection: config.chunks index: ns_1_min_1 dup key: { : "moderation.fs.chunks", : { files_id: UUID("00ce8000-0000-e000-bc9b-abac0c772d7d"), n: 0 } }

Is there any progress on this issue? Not being able to move chunks either manually or automatically will lead to shards running out of disk space. Is there any workaround available?

Comment by Aristarkh Zagorodnikov [ 14/May/19 ]

Kaloian, thank you for returning back to this issue. I've uploaded the fresh zipped dump of the cluster's config database to the location you provided.

Comment by Kaloian Manassiev [ 14/May/19 ]

Hi onyxmaster,

I apologise for the delay in looking into this issue. We started investigating it yesterday and found out that due to the time that has passed, the storage on which I saved the dump of the config server that you provided me has expired and the archive was deleted. I am going to have to ask you to please upload it again.

To make it easier, I’ve created a secure upload portal for you to use. Files uploaded to this portal are only visible to MongoDB employees investigating this issue and are routinely deleted after some time.

Apologies again and many thanks for helping us investigate this issue!

Best regards,
-Kal.

Comment by Danny Hatcher (Inactive) [ 16/Apr/19 ]

amardeepsg@gmail.com, we do not currently have a fix prepared but will update this ticket when we do. Regardless, we do not update patch versions like 3.6.7. If we do decide to backport a fix to 3.6, it would be to the latest 3.6.x version.

Also, what are the alternate ways for us to ensure shard data balancing, if empty chunks continue to exist and we cant merge them.

For advice on shard balancing, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Thanks,

Danny

Comment by amardeep singh [ 15/Apr/19 ]

Any change of this getting closed on 3.6.7?  Also, what are the alternate ways for us to ensure shard data balancing, if empty chunks continue to exist and we cant merge them.

Comment by Aristarkh Zagorodnikov [ 16/Mar/19 ]

Kaloian, thank you! I'm feeling relieved knowing that there is no data loss involved. Thank you for investigating this, looking forward to the fix. I hope this gets backported to 3.6 too, since we're not yet ready to move to 4.x (at least until this summer).

Comment by Kaloian Manassiev [ 15/Mar/19 ]

Hi onyxmaster,

Thank you very much for uploading your config server's metadata! Thanks to it I was able to reproduce the problem locally and now we have everything we need in order to continue investigating this issue, so feel free to remove the upload.

For now I don't have anything to update, but we will post the results of the investigation here.

Thanks again for your help!

Best regards,
-Kal.

PS: I also edited the ticket title to indicate that there is no actual data loss

Comment by Aristarkh Zagorodnikov [ 11/Mar/19 ]

There is nothing specifically sensitive in this data, so I uploaded it at -https://drive-public-eu.s3.amazonaws.com/mongodb/server-40061-config.zip- (10MB). Still, I would like to not to keep it up longer than needed, so please tell me when you no longer need it to be online.

Comment by Kaloian Manassiev [ 11/Mar/19 ]

Thank you very much for the prompt response. From the message, this looks like a duplicate key that was generated in the ns, min index, not the _id.

Would it be possible to attach a dump of your config database so I can recreate the cluster (without data) and try the move command that you are issuing? If attaching it to this ticket poses any concern on your side, we can provide a secure upload portal to which only MongoDB employees have access.

Comment by Aristarkh Zagorodnikov [ 11/Mar/19 ]

I'd also like to note that this problem reproduces in 100% of the cases for our cluster that was upgraded from v3.4 – all chunks that have UUID field in "min" key fail at migration commit every time. I believe splits are failing too, because I don't see any splits in logs after upgrade (although it might be I'm looking in the wrong place since IIRC v3.6 moved a lot of chunk management onto shards themselves).

Comment by Aristarkh Zagorodnikov [ 11/Mar/19 ]

As for my data loss concerns, the Balancer::_splitOrMarkJumbo will probably always silently fail marking chunks as jumbo (no data loss, yet suboptimal) and ShardingCatalogManager::commitChunkSplit will probably never be able to split chunk (not sure how the failure will be handled since it's an upsert).

Comment by Aristarkh Zagorodnikov [ 11/Mar/19 ]

No, I'm not using the internal command, I'm issuing the sh.moveChunk command from shell (or the "moveChunk" command from our C# application). Consider the following log from one of the shard primaries:

...
2019-03-10T16:54:38.231+0300 I SHARDING [conn788179] moveChunk data transfer progress: { waited: true, active: true, sessionId: "driveFS-2_driveFS-17_5c8515ca7bc16fcff631675b", ns: "a.fs.chunks", from: "driveFS-2/d1.s2.fs.drive.bru:27022,d2.s2.fs.drive.bru:27022,d3.s2.fs.drive.bru:27022", min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, shardKeyPattern: { files_id: 1.0, n: 1.0 }, state: "clone", counts: { cloned: 2576, clonedBytes: 346861214, catchup: 0, steady: 0 }, ok: 1.0, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff000000000000000f') }, $configServerState: { opTime: { ts: Timestamp(1552226074, 5), t: 95 } } } mem used: 0 documents remaining to clone: 0
2019-03-10T16:54:38.598+0300 I SHARDING [conn788179] moveChunk data transfer progress: { waited: true, active: true, sessionId: "driveFS-2_driveFS-17_5c8515ca7bc16fcff631675b", ns: "a.fs.chunks", from: "driveFS-2/d1.s2.fs.drive.bru:27022,d2.s2.fs.drive.bru:27022,d3.s2.fs.drive.bru:27022", min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, shardKeyPattern: { files_id: 1.0, n: 1.0 }, state: "steady", counts: { cloned: 2576, clonedBytes: 346861214, catchup: 0, steady: 0 }, ok: 1.0, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff000000000000000f') }, $configServerState: { opTime: { ts: Timestamp(1552226074, 5), t: 95 } } } mem used: 0 documents remaining to clone: 0
2019-03-10T16:54:38.635+0300 I SHARDING [conn788179] Migration successfully entered critical section
2019-03-10T16:54:39.011+0300 I SHARDING [conn788179] Error occurred while committing the migration. Performing a majority write against the config server to obtain its latest optime :: caused by :: DuplicateKey: E11000 duplicate key error collection: config.chunks index: ns_1_min_1 dup key: { : "a.fs.chunks", : { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 } }
2019-03-10T16:54:39.011+0300 I SHARDING [conn788179] about to log metadata event into changelog: { _id: "dorado3-2019-03-10T16:54:39.011+0300-5c85171f7bc16fcff6319a41", server: "dorado3", clientAddr: "10.3.1.11:51332", time: new Date(1552226079011), what: "moveChunk.validating", ns: "a.fs.chunks", details: { min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, from: "driveFS-2", to: "driveFS-17" } }
2019-03-10T16:54:39.187+0300 I SHARDING [conn788179] about to log metadata event into changelog: { _id: "dorado3-2019-03-10T16:54:39.187+0300-5c85171f7bc16fcff6319a4e", server: "dorado3", clientAddr: "10.3.1.11:51332", time: new Date(1552226079187), what: "moveChunk.error", ns: "a.fs.chunks", details: { min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, from: "driveFS-2", to: "driveFS-17" } }
2019-03-10T16:54:39.190+0300 W SHARDING [conn788179] Chunk move failed :: caused by :: DuplicateKey: Chunk move was not successful due to E11000 duplicate key error collection: config.chunks index: ns_1_min_1 dup key: { : "a.fs.chunks", : { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 } }
2019-03-10T16:54:39.190+0300 I SHARDING [conn788179] about to log metadata event into changelog: { _id: "dorado3-2019-03-10T16:54:39.190+0300-5c85171f7bc16fcff6319a50", server: "dorado3", clientAddr: "10.3.1.11:51332", time: new Date(1552226079190), what: "moveChunk.from", ns: "a.fs.chunks", details: { min: { files_id: UUID("05660000-0000-e000-96c2-dc81ca6fa911"), n: 0 }, max: { files_id: UUID("05666100-0000-e000-9252-7b82dea0b186"), n: 3 }, step 1 of 6: 0, step 2 of 6: 88, step 3 of 6: 90, step 4 of 6: 339694, step 5 of 6: 386, to: "driveFS-17", from: "driveFS-2", note: "aborted" } }

Comment by Kaloian Manassiev [ 11/Mar/19 ]

Hi onyxmaster,

Thank you very much for your report and concern.

First I would like to alleviate your concerns about data loss: The _id field in the config.chunks collection is not used for routing of queries and/or updates, but only for generating chunk modifications internally (SERVER-27735). So, even if there is change in how it is generated will not cause any operations to be mis-routed or for the balancer to move chunks to wrong shards. So rest assured that there is no data loss.

For the DuplicateKey error - I notice that the application name for the _configsvrMoveChunk command which encountered it is "MongoDB Shell". Are you issuing this command directly against the config server? Because this is an internal command which should not be used directly, and instead moveChunk against MongoS should be used.

Best regards,
-Kal.

Comment by Aristarkh Zagorodnikov [ 11/Mar/19 ]

The "changed" link in the description is wrong, it should be this one.

Generated at Thu Feb 08 04:53:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.