[SERVER-27945] Primary shard not distributed Created: 08/Feb/17  Updated: 27/Oct/23  Resolved: 08/Feb/17

Status: Closed
Project: Core Server
Component/s: Sharding, WiredTiger
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Anthony Brodard Assignee: Kaloian Manassiev
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Sprint: Sharding 2017-02-13
Participants:

 Description   

Hi team,

Mongodb configuration :

  • Version: 3.4.2
  • 20 shards : 1 primary / 1 secondary / 1 arbiter
  • Engine : Wiredtiger
  • 3 configs
  • 3 mongos

New dbs created on the cluster are allocated on the same shard.
More than 100 dbs have been created (mostly contains only a collection with an index), primary shard is the same. We tried to put some data on few DBs (as allocation algorithm is based on the number of db by shard + their size), but new database still goes on the same place.
We didn't find any relevant thing in conf or logs that could explain or help us to debug.

Is there a specific setting to set ? Or is it an (known) issue ?

Thank you,
Anthony



 Comments   
Comment by Kaloian Manassiev [ 08/Feb/17 ]

Great, thank you for confirming.

Comment by Anthony Brodard [ 08/Feb/17 ]

Hi Kaloian,

Thank you for your reply.
We found our problem. When we setup the cluster, we have imported an huge database to see how it scale and distribute data.
Then we dropped this database, but not all oplogs, which is used to calculate the total shard size.
Now that we have inserted some data on shard19, new databases are created on another shards.

About the balancer alert, it is an old one we fixed last week.
This issue can be closed now.
.
Regards,
Anthony

Comment by Kaloian Manassiev [ 08/Feb/17 ]

Hi anthony@sendinblue.com,

Database creation uses the estimated total size of the shards to determine where to place new databases and tries to evenly utilize the shards. Is it possible that shard19 happens to have the least amount of data on it?

You should be able to see your shards' utilization in CloudManager, but I also sometimes use this script to sort the shards by their utilization (note that it requires ability to query the config.shards database directly and to connect directly to the shards in order to work):

use config;
db.shards.find({}).map(function(shard) {
    var mongod = new Mongo(shard.host);
    var dbInfo = assert.commandWorked(mongod.adminCommand({ listDatabases: 1 }));
    return { shardName: shard._id, size: dbInfo.totalSize };
}).sort(function(a, b) {
    return a.size - b.size; 
});

Additionally, please note the following error that that was reported by the balancer:

	Failed balancer rounds in last 5 attempts:  5
	Last reported error:  Cannot accept sharding commands if not started with --shardsvr

This indicates that one or more of your shards were not started with the --shardsvr command line option, which was made mandatory in 3.4.

-Kal.

Comment by Anthony Brodard [ 08/Feb/17 ]

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("58494e882a0c3ddf8e1ff2c3")
}
  shards:
	{  "_id" : "shard1",  "host" : "<anonymized>",  "state" : 1 }
	{  "_id" : "shard10",  "host" : "<anonymized>",  "state" : 1 }
	{  "_id" : "shard11",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard12",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard13",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard14",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard15",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard16",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard17",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard18",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard19",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard20",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard4",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard5",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard6",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard7",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard8",  "host" : "<anonymized>",  "state" : 1 }
        {  "_id" : "shard9",  "host" : "<anonymized>",  "state" : 1 }
 
  active mongoses:
	"3.4.2" : 3
 autosplit:
	Currently enabled: yes
  balancer:
	Currently enabled:  yes
	Currently running:  no
		Balancer lock taken at Wed Feb 08 2017 11:53:41 GMT+0100 (CET) by ConfigServer:Balancer
	Failed balancer rounds in last 5 attempts:  5
	Last reported error:  Cannot accept sharding commands if not started with --shardsvr
	Time of Reported error:  Tue Jan 31 2017 15:28:35 GMT+0100 (CET)
	Migration Results for the last 24 hours: 
		No recent migrations
  databases:
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
        {  "_id" : "<anonymized>",  "primary" : "shard19",  "partitioned" : false }
   <truncated>

Generated at Thu Feb 08 04:16:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.