[SERVER-14431] Invalid chunk data after splitting on a key that's too large Created: 03/Jul/14  Updated: 11/Jul/16  Resolved: 23/Jul/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.3, 2.7.2
Fix Version/s: 2.6.4, 2.7.4

Type: Bug Priority: Critical - P2
Reporter: Kamran K. Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-14602 Update records document but not index... Closed
Related
related to SERVER-14763 Update jstests/sharding/split_large_k... Closed
Tested
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

var dbName = 'too_large';
var collName = 'foo';
var ns = dbName + '.' + collName;
 
var st = new ShardingTest({});
var coll = st.s0.getDB(dbName)[collName];
coll.drop();
 
st.s0.adminCommand({enableSharding: dbName});
st.s0.adminCommand({shardCollection: ns, key: {_id: 1}});
 
st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
// retry the split to trigger the 'Couldn't load a valid config' assertion
st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
 
st.c0.getDB('config').chunks.find();
st.stop();

Participants:

 Description   
Issue Status as of Jul 24, 2014

ISSUE SUMMARY
In a sharded cluster, splitting a chunk on a mid point that is larger than 1024 bytes (i.e. too large to be indexed), the higher of the two chunks can get lost in the split process which corrupts the metadata information.

This is only an issue for split points specified by the user through the split command. Internal splits (auto-split or initial split) are not affected by this issue.

USER IMPACT
A mongos process checks whether the entire range from MinKey to MaxKey is covered without gaps and overlaps during startup. For a sharded cluster affected by this issue, the range is no longer covered and therefore mongos refuses to start.

WORKAROUNDS
Make sure that the specified split point is never longer than 1024 bytes.

AFFECTED VERSIONS
All production release versions up 2.6.3 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
The size of the split point value is now checked before the split is executed.

Original description

It seems like chunk data becomes invalid if you attempt to split on a key that's too large to index.

I was able to reproduce the bug with 2.6.3, 2.7.2, and master.

// sharding has already been enabled and the collection has been sharded
 
 > st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
 m30999| 2014-07-02T20:09:55.012-0400 [conn1] splitting chunk [{ _id: MinKey },{ _id: MaxKey }) in collection too_large.foo on shard shard0001
 m30001| 2014-07-02T20:09:55.012-0400 [conn4] received splitChunk request: { splitChunk: "too_large.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "shard0001", splitKeys: [ { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } ], shardId: "too_large.foo-_id_MinKey", configdb: "localhost:30000" }
 m30000| 2014-07-02T20:09:55.013-0400 [initandlisten] connection accepted from 127.0.0.1:61633 #11 (10 connections now open)
 m30001| 2014-07-02T20:09:55.013-0400 [LockPinger] creating distributed lock ping thread for localhost:30000 and process Kams-MacBook-Pro.local:30001:1404346195:1781788995 (sleeping for 30000ms)
 m30001| 2014-07-02T20:09:55.014-0400 [LockPinger] cluster localhost:30000 pinged successfully at Wed Jul  2 20:09:55 2014 by distributed lock pinger 'localhost:30000/Kams-MacBook-Pro.local:30001:1404346195:1781788995', sleeping for 30000ms
 m30001| 2014-07-02T20:09:55.014-0400 [conn4] distributed lock 'too_large.foo/Kams-MacBook-Pro.local:30001:1404346195:1781788995' acquired, ts : 53b49f538013064737cade39
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] remotely refreshing metadata for too_large.foo based on current shard version 1|0||53b49f4afc63753e54c82028, current metadata version is 1|0||53b49f4afc63753e54c82028
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] metadata of collection too_large.foo already up to date (shard version : 1|0||53b49f4afc63753e54c82028, took 0ms)
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] splitChunk accepted at version 1|0||53b49f4afc63753e54c82028
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] ERROR: saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, max: { _id: MaxKey }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "too_large.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 1000|0 } } ] } result: { errmsg: "exception: insertDocument :: caused by :: 0 Btree::insert: key too large to index, failing config.chunks.$_id_ 1057 { : "too_large.foo-_id_"aaaaaaaaaa...", code: 8, ok: 0.0 }
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] Assertion: 13593:saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, max: { _id: MaxKey }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "too_large.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 1000|0 } } ] } result: { errmsg: "exception: insertDocument :: caused by :: 0 Btree::insert: key too large to index, failing config.chunks.$_id_ 1057 { : "too_large.foo-_id_"aaaaaaaaaa...", code: 8, ok: 0.0 }
 m30001| 2014-07-02T20:09:55.018-0400 [conn4] 
 m30001|  0x1007dcbfa 0x10078865d 0x100777840 0x1007776bc 0x10066b824 0x1001ed5ea 0x1001ee2fa 0x1001ef56d 0x100444c1c 0x1002d89ee 0x10000c1c9 0x100798c30 0x100818121 0x7fff8acfc899 0x7fff8acfc72a 0x7fff8ad00fc9
 m30001| ----- BEGIN BACKTRACE -----
 m30001| {"backtrace":[{"b":"100000000","o":"7DCBFA"},{"b":"100000000","o":"78865D"},{"b":"100000000","o":"777840"},{"b":"100000000","o":"7776BC"},{"b":"100000000","o":"66B824"},{"b":"100000000","o":"1ED5EA"},{"b":"100000000","o":"1EE2FA"},{"b":"100000000","o":"1EF56D"},{"b":"100000000","o":"444C1C"},{"b":"100000000","o":"2D89EE"},{"b":"100000000","o":"C1C9"},{"b":"100000000","o":"798C30"},{"b":"100000000","o":"818121"},{"b":"7FFF8ACFB000","o":"1899"},{"b":"7FFF8ACFB000","o":"172A"},{"b":"7FFF8ACFB000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "2.7.3-pre-", "gitVersion" : "13693a083ef299c2de0759ed34c53b9370c850f6", "uname" : { "sysname" : "Darwin", "release" : "13.2.0", "version" : "Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/kk/code/mongo/mongo/mongod", "machType" : 2, "b" : "100000000", "buildId" : "310AF0EE166E3A65B997A72AEA57DDCD" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF84E19000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libstdc++.6.dylib", "machType" : 6, "b" : "7FFF84EF8000", "buildId" : "0241E6A4136833BE950BD0A175C41F54" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8D75C000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF9156B000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF855C0000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF89FE5000", "buildId" : "5A881779D0D63029B371E3021C2DDA5E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8BFD3000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8ED07000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF90146000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF90D3E000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF88590000", "buildId" : "38D1AB2CA476385F8EA87AB604CA1F89" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF855BA000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF85F40000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF9052E000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF91A19000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF89F81000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF9014A000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF85385000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8FD54000", "buildId" : "A0B7CF19D9F233D48107A62184C9066E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF85592000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF86281000", "buildId" : "498AEBD741943CF2AA16D5D03FFBD8C0" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8AD03000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF9154F000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8F8D2000", "buildId" : "8B1E1F1DA5CC3BAE8B1EABC84337A364" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF89F83000", "buildId" : "52571EC3689437E4946E064B021ED44E" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF91548000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8ACFB000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF860F2000", "buildId" : "B46E4040A8C63EBC91F8F1CB01106614" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF87ED1000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8983D000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF904FE000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8625C000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF85A05000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8F1B5000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF85544000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8BEE8000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8DC82000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 m30001|  mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x1007dcbfa]
 m30001|  mongod(_ZN5mongo10logContextEPKc+0xCD) [0x10078865d]
 m30001|  mongod(_ZN5mongo11msgassertedEiPKc+0x180) [0x100777840]
 m30001|  mongod(_ZN5mongo11msgassertedEiRKSs+0xC) [0x1007776bc]
 m30001|  mongod(_ZN5mongo17SplitChunkCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x73A4) [0x10066b824]
 m30001|  mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3A) [0x1001ed5ea]
 m30001|  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x80A) [0x1001ee2fa]
 m30001|  mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x51D) [0x1001ef56d]
 m30001|  mongod(_ZN5mongo11newRunQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_+0x2EC) [0x100444c1c]
 m30001|  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x92E) [0x1002d89ee]
 m30001|  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xB9) [0x10000c1c9]
 m30001|  mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x390) [0x100798c30]
 m30001|  mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x100818121]
 m30001|  libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8acfc899]
 m30001|  libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8acfc72a]
 m30001|  libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ad00fc9]
 m30001| -----  END BACKTRACE  -----
 m30001| 2014-07-02T20:09:55.019-0400 [conn4] distributed lock 'too_large.foo/Kams-MacBook-Pro.local:30001:1404346195:1781788995' unlocked. 
 m30999| 2014-07-02T20:09:55.019-0400 [conn1] warning: splitChunk failed - cmd: { splitChunk: "too_large.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "shard0001", splitKeys: [ { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } ], shardId: "too_large.foo-_id_MinKey", configdb: "localhost:30000" } result: { errmsg: "exception: saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp...", code: 13593, ok: 0.0 }
{
	"cause" : "SplitFailed splitChunk failed - cmd: { splitChunk: \"too_large.foo\", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: \"shard0001\", splitKeys: [ { _id: \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...\" } ], shardId: \"too_large.foo-_id_MinKey\", configdb: \"localhost:30000\" } result: { errmsg: \"exception: saving chunks failed.  cmd: { applyOps: [ { op: \"u\", b: true, ns: \"config.chunks\", o: { _id: \"too_large.foo-_id_MinKey\", lastmod: Timestamp...\", code: 13593, ok: 0.0 }",
	"ok" : 0,
	"errmsg" : "split failed"
}
 
 
// retry the split
> st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.582-0400 [conn1] Assertion: 13282:Couldn't load a valid config for too_large.foo after 3 attempts. Please try again.
 m30999| 2014-07-02T20:09:56.584-0400 [conn1] 
 m30999|  0x10052cd1a 0x1004dd47d 0x1004cb1a0 0x1004cb01c 0x10033058d 0x1003afd13 0x10037028e 0x10041f3e0 0x1003870b6 0x100436bf9 0x10041e18a 0x100003f55 0x1004e7950 0x100568241 0x7fff8acfc899 0x7fff8acfc72a 0x7fff8ad00fc9
 m30999| ----- BEGIN BACKTRACE -----
 m30999| {"backtrace":[{"b":"100000000","o":"52CD1A"},{"b":"100000000","o":"4DD47D"},{"b":"100000000","o":"4CB1A0"},{"b":"100000000","o":"4CB01C"},{"b":"100000000","o":"33058D"},{"b":"100000000","o":"3AFD13"},{"b":"100000000","o":"37028E"},{"b":"100000000","o":"41F3E0"},{"b":"100000000","o":"3870B6"},{"b":"100000000","o":"436BF9"},{"b":"100000000","o":"41E18A"},{"b":"100000000","o":"3F55"},{"b":"100000000","o":"4E7950"},{"b":"100000000","o":"568241"},{"b":"7FFF8ACFB000","o":"1899"},{"b":"7FFF8ACFB000","o":"172A"},{"b":"7FFF8ACFB000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "2.7.3-pre-", "gitVersion" : "13693a083ef299c2de0759ed34c53b9370c850f6", "uname" : { "sysname" : "Darwin", "release" : "13.2.0", "version" : "Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/kk/code/mongo/mongo/mongos", "machType" : 2, "b" : "100000000", "buildId" : "3A1A686D504A373387A8AA2B4779A06D" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF84E19000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libstdc++.6.dylib", "machType" : 6, "b" : "7FFF84EF8000", "buildId" : "0241E6A4136833BE950BD0A175C41F54" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8D75C000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF9156B000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF855C0000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF89FE5000", "buildId" : "5A881779D0D63029B371E3021C2DDA5E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8BFD3000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8ED07000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF90146000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF90D3E000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF88590000", "buildId" : "38D1AB2CA476385F8EA87AB604CA1F89" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF855BA000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF85F40000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF9052E000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF91A19000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF89F81000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF9014A000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF85385000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8FD54000", "buildId" : "A0B7CF19D9F233D48107A62184C9066E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF85592000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF86281000", "buildId" : "498AEBD741943CF2AA16D5D03FFBD8C0" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8AD03000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF9154F000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8F8D2000", "buildId" : "8B1E1F1DA5CC3BAE8B1EABC84337A364" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF89F83000", "buildId" : "52571EC3689437E4946E064B021ED44E" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF91548000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8ACFB000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF860F2000", "buildId" : "B46E4040A8C63EBC91F8F1CB01106614" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF87ED1000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8983D000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF904FE000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8625C000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF85A05000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8F1B5000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF85544000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8BEE8000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8DC82000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 m30999|  mongos(_ZN5mongo15printStackTraceERSo+0x3A) [0x10052cd1a]
 m30999|  mongos(_ZN5mongo10logContextEPKc+0xCD) [0x1004dd47d]
 m30999|  mongos(_ZN5mongo11msgassertedEiPKc+0x180) [0x1004cb1a0]
 m30999|  mongos(_ZN5mongo11msgassertedEiRKSs+0xC) [0x1004cb01c]
 m30999|  mongos(_ZN5mongo12ChunkManager18loadExistingRangesERKSs+0xBBD) [0x10033058d]
 m30999|  mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0xD43) [0x1003afd13]
 m30999|  mongos(_ZN5mongo11dbgrid_cmds18SplitCollectionCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x65E) [0x10037028e]
 m30999|  mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x340) [0x10041f3e0]
 m30999|  mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x136) [0x1003870b6]
 m30999|  mongos(_ZN5mongo8Strategy15clientCommandOpERNS_7RequestE+0x5D9) [0x100436bf9]
 m30999|  mongos(_ZN5mongo7Request7processEi+0x3AA) [0x10041e18a]
 m30999|  mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x65) [0x100003f55]
 m30999|  mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x390) [0x1004e7950]
 m30999|  mongos(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x100568241]
 m30999|  libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8acfc899]
 m30999|  libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8acfc72a]
 m30999|  libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ad00fc9]
 m30999| -----  END BACKTRACE  -----
{
	"code" : 13282,
	"ok" : 0,
	"errmsg" : "exception: Couldn't load a valid config for too_large.foo after 3 attempts. Please try again."
}
 
 
 
// display the chunk data
> st.c0.getDB('config').chunks.find();
{ "_id" : "too_large.foo-_id_MinKey", "lastmod" : Timestamp(1, 1), "lastmodEpoch" : ObjectId("53b4a22af430925aa314290f"), "ns" : "too_large.foo", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, "shard" : "shard0001" }


Version: 13693a083ef299c2de0759ed34c53b9370c850f6



 Comments   
Comment by Githook User [ 24/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14431 Invalid chunk data after splitting on a key that's too large to index
(cherry picked from commit 0e8a436fdedc9a56ddd4e773d45d7963a8f358a6)
(cherry picked from commit 7174d92cc1ad3dd24430769b72c05b1ee6fbcd74)
(cherry picked from commit 4f01ffd9cbbb0dcb4ee2f9167ad2337178259ba9)
Branch: v2.6
https://github.com/mongodb/mongo/commit/d42d01afece8782738aa342a7ade891cee003c64

Comment by Githook User [ 23/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14431 Invalid chunk data after splitting on a key that's too large to index

Fix linking error
Branch: master
https://github.com/mongodb/mongo/commit/4f01ffd9cbbb0dcb4ee2f9167ad2337178259ba9

Comment by Githook User [ 23/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14431 Invalid chunk data after splitting on a key that's too large to index

Fix error message
Branch: master
https://github.com/mongodb/mongo/commit/7174d92cc1ad3dd24430769b72c05b1ee6fbcd74

Comment by Randolph Tan [ 23/Jul/14 ]

Decided to also fix the split command since it is easier to backport.

Comment by Githook User [ 23/Jul/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14431 Invalid chunk data after splitting on a key that's too large to index
Branch: master
https://github.com/mongodb/mongo/commit/0e8a436fdedc9a56ddd4e773d45d7963a8f358a6

Comment by Randolph Tan [ 18/Jul/14 ]

The single split problem as described by the ticket will be fixed by SERVER-14602. There is also a potential issue with multi-splits but we are able to avoid this since mongos does not expose that functionality. The internal sharding mechanism does however perform multi splits, but they are always derived from existing documents and will not have this problem since there is a 512 byte limit (half of index key limit) for shard key values.

Generated at Thu Feb 08 03:34:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.