Loading...

Type: Bug
Resolution: Done
Priority: Critical - P2
Fix Version/s: 2.6.4, 2.7.4
Affects Version/s: 2.6.3, 2.7.2
Component/s: Sharding
Labels:
- 28qa

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Backport Completed:

2.6.4
Steps To Reproduce:
Hide

var dbName = 'too_large'; var collName = 'foo'; var ns = dbName + '.' + collName; var st = new ShardingTest({}); var coll = st.s0.getDB(dbName)[collName]; coll.drop(); st.s0.adminCommand({enableSharding: dbName}); st.s0.adminCommand({shardCollection: ns, key: {_id: 1}}); st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}}); // retry the split to trigger the 'Couldn't load a valid config' assertion st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}}); st.c0.getDB('config').chunks.find(); st.stop();
Show
var dbName = 'too_large'; var collName = 'foo'; var ns = dbName + '.' + collName; var st = new ShardingTest({}); var coll = st.s0.getDB(dbName)[collName]; coll.drop(); st.s0.adminCommand({enableSharding: dbName}); st.s0.adminCommand({shardCollection: ns, key: {_id: 1}}); st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}}); // retry the split to trigger the 'Couldn't load a valid config' assertion st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}}); st.c0.getDB('config').chunks.find(); st.stop();

Issue Status as of Jul 24, 2014

ISSUE SUMMARY
In a sharded cluster, splitting a chunk on a mid point that is larger than 1024 bytes (i.e. too large to be indexed), the higher of the two chunks can get lost in the split process which corrupts the metadata information.

This is only an issue for split points specified by the user through the split command. Internal splits (auto-split or initial split) are not affected by this issue.

USER IMPACT
A mongos process checks whether the entire range from MinKey to MaxKey is covered without gaps and overlaps during startup. For a sharded cluster affected by this issue, the range is no longer covered and therefore mongos refuses to start.

WORKAROUNDS
Make sure that the specified split point is never longer than 1024 bytes.

AFFECTED VERSIONS
All production release versions up 2.6.3 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
The size of the split point value is now checked before the split is executed.

Original description

It seems like chunk data becomes invalid if you attempt to split on a key that's too large to index.

I was able to reproduce the bug with 2.6.3, 2.7.2, and master.

// sharding has already been enabled and the collection has been sharded

 > st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
 m30999| 2014-07-02T20:09:55.012-0400 [conn1] splitting chunk [{ _id: MinKey },{ _id: MaxKey }) in collection too_large.foo on shard shard0001
 m30001| 2014-07-02T20:09:55.012-0400 [conn4] received splitChunk request: { splitChunk: "too_large.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "shard0001", splitKeys: [ { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } ], shardId: "too_large.foo-_id_MinKey", configdb: "localhost:30000" }
 m30000| 2014-07-02T20:09:55.013-0400 [initandlisten] connection accepted from 127.0.0.1:61633 #11 (10 connections now open)
 m30001| 2014-07-02T20:09:55.013-0400 [LockPinger] creating distributed lock ping thread for localhost:30000 and process Kams-MacBook-Pro.local:30001:1404346195:1781788995 (sleeping for 30000ms)
 m30001| 2014-07-02T20:09:55.014-0400 [LockPinger] cluster localhost:30000 pinged successfully at Wed Jul  2 20:09:55 2014 by distributed lock pinger 'localhost:30000/Kams-MacBook-Pro.local:30001:1404346195:1781788995', sleeping for 30000ms
 m30001| 2014-07-02T20:09:55.014-0400 [conn4] distributed lock 'too_large.foo/Kams-MacBook-Pro.local:30001:1404346195:1781788995' acquired, ts : 53b49f538013064737cade39
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] remotely refreshing metadata for too_large.foo based on current shard version 1|0||53b49f4afc63753e54c82028, current metadata version is 1|0||53b49f4afc63753e54c82028
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] metadata of collection too_large.foo already up to date (shard version : 1|0||53b49f4afc63753e54c82028, took 0ms)
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] splitChunk accepted at version 1|0||53b49f4afc63753e54c82028
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] ERROR: saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, max: { _id: MaxKey }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "too_large.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 1000|0 } } ] } result: { errmsg: "exception: insertDocument :: caused by :: 0 Btree::insert: key too large to index, failing config.chunks.$_id_ 1057 { : "too_large.foo-_id_"aaaaaaaaaa...", code: 8, ok: 0.0 }
 m30001| 2014-07-02T20:09:55.015-0400 [conn4] Assertion: 13593:saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('53b49f4afc63753e54c82028'), ns: "too_large.foo", min: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }, max: { _id: MaxKey }, shard: "shard0001" }, o2: { _id: "too_large.foo-_id_"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "too_large.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 1000|0 } } ] } result: { errmsg: "exception: insertDocument :: caused by :: 0 Btree::insert: key too large to index, failing config.chunks.$_id_ 1057 { : "too_large.foo-_id_"aaaaaaaaaa...", code: 8, ok: 0.0 }
 m30001| 2014-07-02T20:09:55.018-0400 [conn4] 
 m30001|  0x1007dcbfa 0x10078865d 0x100777840 0x1007776bc 0x10066b824 0x1001ed5ea 0x1001ee2fa 0x1001ef56d 0x100444c1c 0x1002d89ee 0x10000c1c9 0x100798c30 0x100818121 0x7fff8acfc899 0x7fff8acfc72a 0x7fff8ad00fc9
 m30001| ----- BEGIN BACKTRACE -----
 m30001| {"backtrace":[{"b":"100000000","o":"7DCBFA"},{"b":"100000000","o":"78865D"},{"b":"100000000","o":"777840"},{"b":"100000000","o":"7776BC"},{"b":"100000000","o":"66B824"},{"b":"100000000","o":"1ED5EA"},{"b":"100000000","o":"1EE2FA"},{"b":"100000000","o":"1EF56D"},{"b":"100000000","o":"444C1C"},{"b":"100000000","o":"2D89EE"},{"b":"100000000","o":"C1C9"},{"b":"100000000","o":"798C30"},{"b":"100000000","o":"818121"},{"b":"7FFF8ACFB000","o":"1899"},{"b":"7FFF8ACFB000","o":"172A"},{"b":"7FFF8ACFB000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "2.7.3-pre-", "gitVersion" : "13693a083ef299c2de0759ed34c53b9370c850f6", "uname" : { "sysname" : "Darwin", "release" : "13.2.0", "version" : "Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/kk/code/mongo/mongo/mongod", "machType" : 2, "b" : "100000000", "buildId" : "310AF0EE166E3A65B997A72AEA57DDCD" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF84E19000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libstdc++.6.dylib", "machType" : 6, "b" : "7FFF84EF8000", "buildId" : "0241E6A4136833BE950BD0A175C41F54" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8D75C000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF9156B000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF855C0000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF89FE5000", "buildId" : "5A881779D0D63029B371E3021C2DDA5E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8BFD3000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8ED07000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF90146000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF90D3E000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF88590000", "buildId" : "38D1AB2CA476385F8EA87AB604CA1F89" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF855BA000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF85F40000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF9052E000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF91A19000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF89F81000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF9014A000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF85385000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8FD54000", "buildId" : "A0B7CF19D9F233D48107A62184C9066E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF85592000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF86281000", "buildId" : "498AEBD741943CF2AA16D5D03FFBD8C0" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8AD03000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF9154F000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8F8D2000", "buildId" : "8B1E1F1DA5CC3BAE8B1EABC84337A364" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF89F83000", "buildId" : "52571EC3689437E4946E064B021ED44E" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF91548000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8ACFB000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF860F2000", "buildId" : "B46E4040A8C63EBC91F8F1CB01106614" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF87ED1000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8983D000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF904FE000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8625C000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF85A05000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8F1B5000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF85544000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8BEE8000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8DC82000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 m30001|  mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x1007dcbfa]
 m30001|  mongod(_ZN5mongo10logContextEPKc+0xCD) [0x10078865d]
 m30001|  mongod(_ZN5mongo11msgassertedEiPKc+0x180) [0x100777840]
 m30001|  mongod(_ZN5mongo11msgassertedEiRKSs+0xC) [0x1007776bc]
 m30001|  mongod(_ZN5mongo17SplitChunkCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x73A4) [0x10066b824]
 m30001|  mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3A) [0x1001ed5ea]
 m30001|  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x80A) [0x1001ee2fa]
 m30001|  mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x51D) [0x1001ef56d]
 m30001|  mongod(_ZN5mongo11newRunQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_+0x2EC) [0x100444c1c]
 m30001|  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x92E) [0x1002d89ee]
 m30001|  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xB9) [0x10000c1c9]
 m30001|  mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x390) [0x100798c30]
 m30001|  mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x100818121]
 m30001|  libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8acfc899]
 m30001|  libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8acfc72a]
 m30001|  libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ad00fc9]
 m30001| -----  END BACKTRACE  -----
 m30001| 2014-07-02T20:09:55.019-0400 [conn4] distributed lock 'too_large.foo/Kams-MacBook-Pro.local:30001:1404346195:1781788995' unlocked. 
 m30999| 2014-07-02T20:09:55.019-0400 [conn1] warning: splitChunk failed - cmd: { splitChunk: "too_large.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "shard0001", splitKeys: [ { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } ], shardId: "too_large.foo-_id_MinKey", configdb: "localhost:30000" } result: { errmsg: "exception: saving chunks failed.  cmd: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "too_large.foo-_id_MinKey", lastmod: Timestamp...", code: 13593, ok: 0.0 }
{
	"cause" : "SplitFailed splitChunk failed - cmd: { splitChunk: \"too_large.foo\", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: \"shard0001\", splitKeys: [ { _id: \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...\" } ], shardId: \"too_large.foo-_id_MinKey\", configdb: \"localhost:30000\" } result: { errmsg: \"exception: saving chunks failed.  cmd: { applyOps: [ { op: \"u\", b: true, ns: \"config.chunks\", o: { _id: \"too_large.foo-_id_MinKey\", lastmod: Timestamp...\", code: 13593, ok: 0.0 }",
	"ok" : 0,
	"errmsg" : "split failed"
}


// retry the split
> st.s0.adminCommand({split: ns, middle: {_id: new Array(1025).join('a')}});
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.518-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.529-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] ChunkManager: time to load chunks for too_large.foo: 0ms sequenceNumber: 3 version: 1|1||53b49f4afc63753e54c82028 based on: 1|0||53b49f4afc63753e54c82028
 m30999| last->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|1||000000000000000000000000, min: { _id: MinKey }, max: { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| it->second->toString(): ns: too_large.foo, shard: shard0001:localhost:30001, lastmod: 1|0||000000000000000000000000, min: { _id: MinKey }, max: { _id: MaxKey }
 m30999| it->second->getMin(): { _id: MinKey }
 m30999| last->second->getMax(): { _id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] ChunkManager::_isValid failed: it->second->getMin() == last->second->getMax()
 m30999| 2014-07-02T20:09:56.551-0400 [conn1] warning: ChunkManager loaded an invalid config for too_large.foo, trying again
 m30999| 2014-07-02T20:09:56.582-0400 [conn1] Assertion: 13282:Couldn't load a valid config for too_large.foo after 3 attempts. Please try again.
 m30999| 2014-07-02T20:09:56.584-0400 [conn1] 
 m30999|  0x10052cd1a 0x1004dd47d 0x1004cb1a0 0x1004cb01c 0x10033058d 0x1003afd13 0x10037028e 0x10041f3e0 0x1003870b6 0x100436bf9 0x10041e18a 0x100003f55 0x1004e7950 0x100568241 0x7fff8acfc899 0x7fff8acfc72a 0x7fff8ad00fc9
 m30999| ----- BEGIN BACKTRACE -----
 m30999| {"backtrace":[{"b":"100000000","o":"52CD1A"},{"b":"100000000","o":"4DD47D"},{"b":"100000000","o":"4CB1A0"},{"b":"100000000","o":"4CB01C"},{"b":"100000000","o":"33058D"},{"b":"100000000","o":"3AFD13"},{"b":"100000000","o":"37028E"},{"b":"100000000","o":"41F3E0"},{"b":"100000000","o":"3870B6"},{"b":"100000000","o":"436BF9"},{"b":"100000000","o":"41E18A"},{"b":"100000000","o":"3F55"},{"b":"100000000","o":"4E7950"},{"b":"100000000","o":"568241"},{"b":"7FFF8ACFB000","o":"1899"},{"b":"7FFF8ACFB000","o":"172A"},{"b":"7FFF8ACFB000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "2.7.3-pre-", "gitVersion" : "13693a083ef299c2de0759ed34c53b9370c850f6", "uname" : { "sysname" : "Darwin", "release" : "13.2.0", "version" : "Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/kk/code/mongo/mongo/mongos", "machType" : 2, "b" : "100000000", "buildId" : "3A1A686D504A373387A8AA2B4779A06D" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF84E19000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libstdc++.6.dylib", "machType" : 6, "b" : "7FFF84EF8000", "buildId" : "0241E6A4136833BE950BD0A175C41F54" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8D75C000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF9156B000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF855C0000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF89FE5000", "buildId" : "5A881779D0D63029B371E3021C2DDA5E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8BFD3000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8ED07000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF90146000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF90D3E000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF88590000", "buildId" : "38D1AB2CA476385F8EA87AB604CA1F89" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF855BA000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF85F40000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF9052E000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF91A19000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF89F81000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF9014A000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF85385000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8FD54000", "buildId" : "A0B7CF19D9F233D48107A62184C9066E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF85592000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF86281000", "buildId" : "498AEBD741943CF2AA16D5D03FFBD8C0" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8AD03000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF9154F000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8F8D2000", "buildId" : "8B1E1F1DA5CC3BAE8B1EABC84337A364" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF89F83000", "buildId" : "52571EC3689437E4946E064B021ED44E" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF91548000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8ACFB000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF860F2000", "buildId" : "B46E4040A8C63EBC91F8F1CB01106614" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF87ED1000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8983D000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF904FE000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8625C000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF85A05000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8F1B5000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF85544000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8BEE8000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8DC82000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 m30999|  mongos(_ZN5mongo15printStackTraceERSo+0x3A) [0x10052cd1a]
 m30999|  mongos(_ZN5mongo10logContextEPKc+0xCD) [0x1004dd47d]
 m30999|  mongos(_ZN5mongo11msgassertedEiPKc+0x180) [0x1004cb1a0]
 m30999|  mongos(_ZN5mongo11msgassertedEiRKSs+0xC) [0x1004cb01c]
 m30999|  mongos(_ZN5mongo12ChunkManager18loadExistingRangesERKSs+0xBBD) [0x10033058d]
 m30999|  mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0xD43) [0x1003afd13]
 m30999|  mongos(_ZN5mongo11dbgrid_cmds18SplitCollectionCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x65E) [0x10037028e]
 m30999|  mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x340) [0x10041f3e0]
 m30999|  mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x136) [0x1003870b6]
 m30999|  mongos(_ZN5mongo8Strategy15clientCommandOpERNS_7RequestE+0x5D9) [0x100436bf9]
 m30999|  mongos(_ZN5mongo7Request7processEi+0x3AA) [0x10041e18a]
 m30999|  mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x65) [0x100003f55]
 m30999|  mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x390) [0x1004e7950]
 m30999|  mongos(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x100568241]
 m30999|  libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8acfc899]
 m30999|  libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8acfc72a]
 m30999|  libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ad00fc9]
 m30999| -----  END BACKTRACE  -----
{
	"code" : 13282,
	"ok" : 0,
	"errmsg" : "exception: Couldn't load a valid config for too_large.foo after 3 attempts. Please try again."
}



// display the chunk data
> st.c0.getDB('config').chunks.find();
{ "_id" : "too_large.foo-_id_MinKey", "lastmod" : Timestamp(1, 1), "lastmodEpoch" : ObjectId("53b4a22af430925aa314290f"), "ns" : "too_large.foo", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, "shard" : "shard0001" }

Version: 13693a083ef299c2de0759ed34c53b9370c850f6

depends on

SERVER-14602 Update records document but not indexed values due to index error

Closed

related to

SERVER-14763 Update jstests/sharding/split_large_key.js

Closed

Details

Description

Original description

Attachments

Issue Links

Activity

People

Dates