[SERVER-51248] Using _secondaryThrottle and _waitForDelete leads a high number of chunks Created: 30/Sep/20  Updated: 05/Nov/20  Resolved: 05/Nov/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Vinicius Grippa Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File template.json    
Operating System: ALL
Participants:

 Description   

Performing a synthetic workload I noticed a very different number of chunks and an error rate in the balancer when using the option:

Below are the test results:

# Default configuration
mongos> sh.status(0 )mongos> sh.status(0 )
--- Sharding Status ---
  sharding version:
"_id" : 1"minCompatibleVersion" : 5"currentVersion" : 6"clusterId" : ObjectId("5f74ba2926ac7490e83e7c65")  }
  shards:
        {  "_id" : "shard01""host" : "shard01/localhost:37018,localhost:37019,localhost:37020""state" : 1 }
        {  "_id" : "shard02""host" : "shard02/localhost:37021,localhost:37022,localhost:37023""state" : 1 }
        {  "_id" : "shard03""host" : "shard03/localhost:37024,localhost:37025,localhost:37026""state" : 1 }
  active mongoses:        "4.0.20-13" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no 
       Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                16 : Success
                129 : Failed with error 'aborted', from shard01 to shard03
  databases:
        {  "_id" : "config""primary" : "config""partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                               shard01 1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard01 Timestamp(1, 0)
        {  "_id" : "vinnie""primary" : "shard01""partitioned" : true"version" : {  "uuid" : UUID("4bd88f33-481a-47d2-86c6-6a1fad5ed677"),  "lastMod" : 1 } }
                vinnie.companies
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard01 10
                                shard02 9
                                shard03 9
                        too many chunks to print, use verbose if you want to force print
mongos> db.companies.count()
4813948

 And with the settings in place: 

# Using the config below
use config;
db.settings.save( { _id:"chunksize", value: 64 });
db.settings.save( { "_id" : "balancer", "_secondaryThrottle" : { "w" : "majority" }, "_waitForDelete" : true} )

mongos> sh.status()
--- Sharding Status ---
  sharding version: {   "_id" : 1"minCompatibleVersion" : 5"currentVersion" : 6"clusterId" : ObjectId("5f74ccd17a58008967dd6fbf")  }
  shards:
        {  "_id" : "shard01""host" : "shard01/localhost:37018,localhost:37019,localhost:37020""state" : 1 }
        {  "_id" : "shard02""host" : "shard02/localhost:37021,localhost:37022,localhost:37023""state" : 1 }
        {  "_id" : "shard03""host" : "shard03/localhost:37024,localhost:37025,localhost:37026""state" : 1 }
  active mongoses:        "4.0.20-13" : 1
  autosplit:
        Currently enabled: yes  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                64 : Success
  databases:
        {  "_id" : "config""primary" : "config""partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard01 1
                       { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard01 Timestamp(1, 0)
        {  "_id" : "vinnie""primary" : "shard01""partitioned" : true"version" : {  "uuid" : UUID("94232603-84da-43e0-bef7-783db0019696"),  "lastMod" : 1 } }
                vinnie.companies
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard01 33
                                shard02 34
                                shard03 34
                        too many chunks to print, use verbose if you want to force print
mongos> db.companies.count()
4442950

It seems like an issue since Mongo "gives up" about splitting the chunk and I'm afraid this can lead to jumbo chunks. On the other hand, with the settings enabled, I can observe an increased load on the shards (probably due to the write concern majority).



 Comments   
Comment by Eric Sedor [ 05/Nov/20 ]

Thanks as well, vgrippa@gmail.com!

Comment by Vinicius Grippa [ 05/Nov/20 ]

Hi Eric,

That makes sense. Thanks a lot for your time checking this.

Comment by Eric Sedor [ 04/Nov/20 ]

Hi vgrippa@gmail.com and thanks for your patience here. Big picture, we expect chunk counts to vary depending on a variety of factors that are not necessarily bugs. I am not able to reproduce a clear relationship between those settings and a higher chunk count. A few things that are relevant to this scenario:

  • In 4.2 we moved the auto-splitter to run on the shard primary (SERVER-9287), which improves chunk splits. If you can upgrade to 4.2 you should see more predictable chunk split behavior as a result. But that said, the splitter still must acquire a collection lock to split a chunk. If that lock is not available when the split is triggered then it could be delayed.
  • A monotonically increasing shard key of {_id:1} is something we recommend against. That choice in this scenario means that all deletion and splitting activity is likely to occur on the shard containing the maxKey chunk, so competition for locks is expected.

If it's possible for you to narrow down this scenario to a specific chunk split that should or should not occur, it would help us consider the possibility of a bug. Otherwise, we'd recommend upgrading to later versions of MongoDB to ensure the latest sharded collection behavior is in place.

Does this make sense?

Sincerely,
Eric

Comment by Vinicius Grippa [ 06/Oct/20 ]

Hi Eric,

 

1) What is the version of MongoDB
A: I tested with 4.0.20.

2) Is the first sh.status() output you provided for default settings? Above it looks like provided settings are the same for each case
A: The first run was with default settings. The 2nd with updated _waitForDelete and _secondaryThrottle.

3) How is data being added to the collection
A: The steps:
I'm using the template.json to create dummy data.
template.json

Then creating data:

mgeneratejs template.json -n1000000 > ./companies.json

Next, on MongoS:

sh.enableSharding("vinnie");
sh.shardCollection("vinnie.companies", {_id:1});

Then importing the data:

mongoimport --uri mongodb://localhost:37017/vinnie?authSource=admin --collection companies --mode insert --numInsertionWorkers=6 --file=./companies.json

4) Is the collection sharded before or after adding data
A: It is sharded, but it is empty.

Comment by Eric Sedor [ 05/Oct/20 ]

Hi vgrippa@gmail.com,

Can you please clarify some aspects of the description and provide additional details? In particular:

1) What is the version of MongoDB
2) Is the first sh.status() output you provided for default settings? Above it looks like provided settings are the same for each case
3) How is data being added to the collection
4) Is the collection sharded before or after adding data

Generated at Thu Feb 08 05:24:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.