[SERVER-26688] Chunks consisting of a single shard key value do not mark Jumbo when move/split fails Created: 18/Oct/16  Updated: 11/Jul/17  Resolved: 19/Oct/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Scott Kurowski Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File jumbo.js    
Issue Links:
Duplicate
duplicates SERVER-26531 jumbo flag in chunk can be cleared wh... Closed
Operating System: ALL
Steps To Reproduce:

use test;
db.c.createIndex( { uid: 1} )
use config;
sh.enableSharding("test")
sh.shardCollection("test.c", { uid: 1 } )
sh.stopBalancer();
db.adminCommand({split : "test.c", middle : {uid : 10} })
db.adminCommand({split : "test.c", middle : {uid : 100} })
db.adminCommand({split : "test.c", middle : {uid : 1000} })
db.settings.find();
// we want:
//     { "_id" : "chunksize", "value" : NumberLong(64) }
//     { "_id" : "balancer", "stopped" : true }
use test;
// doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid
var doc = {
 "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901"
};
for (var ix=0; ix < 251000; ix++) {
  doc.uid = 99;
  // if ( ix % 2 == 0 ) { doc.uid = 100; }  // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919)
  db.c.insert( doc );
  if ( ix % 10000 == 0 ) { print(ix); }
}
print(ix);
db.c.stats();
sh.status(true);

Note the balancer is off and the Jumbo flags are absent, as initially expected:

mongos> sh.status(true)
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("57fd594ff35daed3247bea8d")
}
  shards:
	{  "_id" : "sh1",  "host" : "sh1/localhost:27018,localhost:27019" }
	{  "_id" : "sh2",  "host" : "sh2/localhost:27020,localhost:27021" }
  active mongoses:
	{  "_id" : "Scotts-MacBook-Pro.local:27017",  "ping" : ISODate("2016-10-11T21:41:26.026Z"),  "up" : NumberLong(822),  "waiting" : true,  "mongoVersion" : "3.2.8" }
  balancer:
	Currently enabled:  no
	Currently running:  no
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		No recent migrations
  databases:
	{  "_id" : "test",  "primary" : "sh2",  "partitioned" : true }
		test.c
			shard key: { "uid" : 1 }
			unique: false
			balancing: true
			chunks:
				sh2	4
			{ "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(1, 1) 
			{ "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(1, 3) 
			{ "uid" : 100 } -->> { "uid" : 1000 } on : sh2 Timestamp(1, 5) 
			{ "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh2 Timestamp(1, 6) 

Finally, start the balancer,

mongos> sh.startBalancer();

and note the server log entries and no-Jumbo flag result:

2016-10-11T14:42:27.572-0700 I SHARDING [Balancer] moving chunk ns: test.c moving ( ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 }) sh2 -> sh1
2016-10-11T14:42:27.799-0700 I SHARDING [Balancer] moveChunk result: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" }
2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] balancer move failed: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" } from: sh2 to: sh1 chunk:  min: { uid: 10.0 } max: { uid: 100.0 }
2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] performing a split because migrate failed for size reasons
2016-10-11T14:42:28.085-0700 I SHARDING [Balancer] split results: CannotSplit: chunk not full enough to trigger auto-split
2016-10-11T14:42:28.086-0700 I SHARDING [Balancer] marking chunk as jumbo: ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 }

Inspecting the related source code for these messages ...

Suggests the state of txn is deficient to permit setting the chunk's Jumbo flag. The resulting shard status is absent the Jumbo flag that should have been set:

mongos> sh.status(true)
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("57fd594ff35daed3247bea8d")
}
  shards:
	{  "_id" : "sh1",  "host" : "sh1/localhost:27018,localhost:27019" }
	{  "_id" : "sh2",  "host" : "sh2/localhost:27020,localhost:27021" }
  active mongoses:
	{  "_id" : "Scotts-MacBook-Pro.local:27017",  "ping" : ISODate("2016-10-11T21:55:54.530Z"),  "up" : NumberLong(1690),  "waiting" : true,  "mongoVersion" : "3.2.8" }
  balancer:
	Currently enabled:  yes
	Currently running:  no
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		2 : Success
		1 : Failed with error 'chunk too big to move', from sh2 to sh1
  databases:
	{  "_id" : "test",  "primary" : "sh2",  "partitioned" : true }
		test.c
			shard key: { "uid" : 1 }
			unique: false
			balancing: true
			chunks:
				sh1	2
				sh2	2
			{ "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(7, 0) 
			{ "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(6, 1) 
			{ "uid" : 100 } -->> { "uid" : 1000 } on : sh1 Timestamp(7, 1) 
			{ "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh1 Timestamp(4, 0) 

Repeating with collection 'C' inserting TWO shard key values, uid = 99, 100 – splits normally:

use test;
db.C.createIndex( { uid: 1} )
use config;
sh.enableSharding("test")
sh.shardCollection("test.C", { uid: 1 } )
sh.stopBalancer();
db.adminCommand({split : "test.C", middle : {uid : 10} })
db.adminCommand({split : "test.C", middle : {uid : 100} })
db.adminCommand({split : "test.C", middle : {uid : 1000} })
db.settings.find();
// we want:
//     { "_id" : "chunksize", "value" : NumberLong(64) }
//     { "_id" : "balancer", "stopped" : true }
use test;
// doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid
var doc = {
 "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901"
};
for (var ix=0; ix < 251000; ix++) {
  doc.uid = 99;
  if ( ix % 2 == 0 ) { doc.uid = 100; }  // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919)
  db.C.insert( doc );
  if ( ix % 10000 == 0 ) { print(ix); }
}
print(ix);
db.C.stats();
sh.status(true);
mongos> sh.status(true);

Participants:

 Description   

Inserting 250001+ small documents (~270B) having a highly-skewed corner case of a single shard key value results in a chunk that

  • meets the size-by-count criteria for a split (has > 250K docs), and
  • meets the size-by-bytes criteria for an auto-split (has > maxChunkSize bytes), but
  • apparently because of 0 computed split points in a single shard key value, it should be marked Jumbo, but the Jumbo flag does not get set by the balancer, even though the server log says it did so.

If the shard is drained, however, using the removeShard and movePrimary commands, the balancer correctly marks the same chunk as Jumbo, as expected.



 Comments   
Comment by Arkadiusz Borek [ 11/Jul/17 ]

We have the same problem (single value shard key with more then 250000 documents) on mongo in version 3.2.13;
Chunk with single value of shard key is not marked as jumbo, but should be.
Situation causes lack of performance because shard is occupied on command splitVector. Additionally problem becames bigger and bigger becasue next updates (e.g. insert new document to collection) on collection triggers next splitVector command. SplitVector commands stack eachother and on the result of db.currentOp() almost all oprations are splitVector for the same collection.

Comment by Randolph Tan [ 19/Oct/16 ]

Was not able to repro with the exact procedure since the collection ends up being balanced without even touching the supposedly jumbo chunk. Modified the test to have the key on the lowest valued chunk and confirmed that this is a duplicate of SERVER-26531.

Generated at Thu Feb 08 04:12:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.