[SERVER-14052] With only 2 distinct key values SplitVector Returns numSplits : 1 but no split is done Created: 26/May/14  Updated: 08/Apr/20  Resolved: 02/Jan/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.1
Fix Version/s: 4.3.3

Type: Bug Priority: Minor - P4
Reporter: Adam Comerford Assignee: Tommaso Tocci
Resolution: Done Votes: 0
Labels: ShardingRoughEdges, sharding-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Start a test cluster, small chunk size:

cluster = new ShardingTest({shards: 2, chunksize: 1});

Connect, create a test database, insert data, manually split to create the "problem" chunk:

sh.enableSharding("splitTest");
sh.shardCollection("splitTest.bar", {a : 1});
for(var i = 0; i < 1000; i++){db.bar.insert({ a : i})};
// this assumes that the $min --> $max initial chunk was split at mid-point of zero
sh.splitAt("splitTest.bar", {a : 2});

Now, a chunk should exist with just two distinct shard key values (0 and 1). No matter how many documents I insert, no further splits happen on that chunk (inserted millions of docs).

The mongos logs for the splitVector look something like this (repeated multiple times):

m30999| 2014-05-26T16:55:14.857+0100 [Balancer] distributed lock 'balancer/adamc-mbp.local:30999:1401115446:16807' unlocked. 
 m30001| 2014-05-26T16:55:17.665+0100 [conn2] request split points lookup for chunk splitTest.bar { : 0.0 } -->> { : 2.0 }
 m30001| 2014-05-26T16:55:19.798+0100 [conn2] warning: chunk is larger than 1048576 bytes because of key { a: 0.0 }
 m30001| 2014-05-26T16:55:19.798+0100 [conn2] warning: Finding the split vector for splitTest.bar over { a: 1.0 } keyCount: 10922 numSplits: 1 lookedAt: 0 took 2133ms
 m30001| 2014-05-26T16:55:19.798+0100 [conn2] command admin.$cmd command: splitVector { splitVector: "splitTest.bar", keyPattern: { a: 1.0 }, min: { a: 0.0 }, max: { a: 2.0 }, maxChunkSizeBytes: 1048576, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 numYields:2 locks(micros) r:3526041 reslen:88 2133ms
 m30999| 2014-05-26T16:55:20.859+0100 [Balancer] distributed lock 'balancer/adamc-mbp.local:30999:1401115446:16807' acquired, ts : 538363e855d6538b0901712c
 m30000| 2014-05-26T16:55:20.860+0100 [conn9] CMD fsync: sync:1 lock:0

Here's sh.status():

	splitTest.foobar
			shard key: { "a" : 1 }
			chunks:
				shard0000	1
				shard0001	2
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0 } -->> { "a" : 2 } on : shard0001 Timestamp(2, 2) 
			{ "a" : 2 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 3) 

A manual split will succeed and create the final possible 2 chunks:

sh.splitAt("splitTest.foobar", {a : 1})
{ "ok" : 1 }
splitTest.foobar
			shard key: { "a" : 1 }
			chunks:
				shard0000	2
				shard0001	2
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0 } -->> { "a" : 1 } on : shard0001 Timestamp(3, 1) 
			{ "a" : 1 } -->> { "a" : 2 } on : shard0000 Timestamp(3, 0) 
			{ "a" : 2 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 3) 

Sprint: Sharding 2019-12-30, Sharding 2020-01-13
Participants:
Case:

 Description   

This is something of an edge case, and it doesn't gain very much, but figured it is technically still a bug.

While checking some related logic, I realized that once I created a chunk with just 2 distinct values (0 and 1 in the test case), no splits occurred even though one more split should be possible. A manual splitAt() is successful, so the split is still possible, but the autosplitter never seems to attempt it.



 Comments   
Comment by Githook User [ 02/Jan/20 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@10gen.com', 'username': 'toto-dev'}

Message: SERVER-14052 Remove reduntant param in splitVector func

The internal splitVector function was accepting both

  • maxChunkSize
  • maxChunkSizeBytes

The first one has been removed because is obviusly redundant.

The SplitVector command still accepts both of them and for compatibility
reason maxChunkSize has precedence over maxChunkSizeBytes over there.
Branch: master
https://github.com/mongodb/mongo/commit/45bf534466eb5e82dcb71e8bb7c7f1791b759276

Comment by Githook User [ 02/Jan/20 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@10gen.com', 'username': 'toto-dev'}

Message: SERVER-14052 Make the autosplitter work with less then three splitpoints

SplitVector is always removing the first splitpoint from the splitvector
before returning it, thus the autsplitter should try to split even when
the splitvector contains just one splitpoint.
Branch: master
https://github.com/mongodb/mongo/commit/c92e20479618b22355b9fb7efa935ff6db5883a9

Comment by Tommaso Tocci [ 18/Dec/19 ]

I figured out that currently the AutoSplitter will split a chunk only if the number of unique keys of its documents is grater or equal to three.
This explain both the scenario presented by adam@comerford.net and the simpler one presented by me. In fact in both of them documents are accumulated under only two keys, that is less then the Autosplitter trashold 3.

Comment by Tommaso Tocci [ 16/Dec/19 ]

I can confirm that the bug is still present on `r4.3.2`.
Actually I discovered that the bug is triggered even if we don't assume that there is a split point at 0
The following are the exact steps to reproduce the bug:

sh.enableSharding("splitTest");
sh.enableAutoSplit();
sh.shardCollection("splitTest.bar", {a : 1});
for(var i = 0; i < 1000; i++){db.getSiblingDB("splitTest").bar.insert({ a : i})};
sh.splitAt("splitTest.bar", {a : 2});
for(var i = 0; i < 90000; i++){db.getSiblingDB("splitTest").bar.insert({ a : 1, sub: i, b: "X".repeat(10000)})};

At this point you can see that no splits have been performed and we still have 2 chunks.

mongos> db.getSiblingDB("config").chunks.find({ns: "splitTest.bar"}).pretty()
{
        "_id" : ObjectId("5df79bf35a0d5566bd06936e"),
        "lastmod" : Timestamp(2, 1),
        "lastmodEpoch" : ObjectId("5df79bf3d28f79264fd86851"),
        "ns" : "splitTest.bar",
        "min" : {
                "a" : { "$minKey" : 1 }
        },
        "max" : {
                "a" : 2
        },
        "shard" : "__unknown_name__-rs1",
        "history" : [
                {
                        "validAfter" : Timestamp(1576508403, 6),
                        "shard" : "__unknown_name__-rs1"
                }
        ]
}
{
        "_id" : ObjectId("5df79bfe5a0d5566bd0693dc"),
        "lastmod" : Timestamp(2, 2),
        "lastmodEpoch" : ObjectId("5df79bf3d28f79264fd86851"),
        "ns" : "splitTest.bar",
        "min" : {
                "a" : 2
        },
        "max" : {
                "a" : { "$maxKey" : 1 }
        },
        "shard" : "__unknown_name__-rs1",
        "history" : [
                {
                        "validAfter" : Timestamp(1576508403, 6),
                        "shard" : "__unknown_name__-rs1"
                }
        ]
}

Even though we have 862.97MiB of total data, of which almost all of it is for the first chunk.

mongos> db.getSiblingDB("splitTest").bar.getShardDistribution()
 
Shard __unknown_name__-rs1 at __unknown_name__-rs1/ubuntu1:20001
 data : 862.97MiB docs : 91000 chunks : 2
 estimated data per chunk : 431.48MiB
 estimated docs per chunk : 45500
 
Totals
 data : 862.97MiB docs : 91000 chunks : 2
 Shard __unknown_name__-rs1 contains 100% data, 100% docs in cluster, avg obj size on shard : 9KiB

Generated at Thu Feb 08 03:33:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.