[SERVER-8022] Balancer doesn't work Created: 23/Dec/12  Updated: 15/Feb/13  Resolved: 27/Dec/12

Status: Closed
Project: Core Server
Component/s: Admin, Storage
Affects Version/s: 2.2.1
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: larry.loi Assignee: Thomas Rueckstiess
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Production environment, MongoDB 2.2.1 64bit for linux


Attachments: Text File sh_status.txt    
Issue Links:
Duplicate
duplicates SERVER-2487 Remove empty chunks (consolidate to n... Closed
Related
Operating System: ALL
Participants:

 Description   

most of the data hold in one shard server. data doesn't distribute on all shard servers, I tried to stop and start the balancer, but didn't work.



 Comments   
Comment by Thomas Rueckstiess [ 28/Dec/12 ]

Hi Larry,

1. Deleting data is a normal operation, but since you deleted only data prior to 2012-12-19 and your shardkey contains a date part, the deletion has left behind empty chunks.

2. A chunk is nothing more than a range (min - max) for each component of the shard key. The empty chunks on your system all have date ranges prior to 2012-12-19, and therefore won't be reused anymore, because all the new documents you insert now will have a later date.

3. The "coarsely ascending + search key" recommendation in Kristina Chodorow's book used months instead of days (which is much coarser) and didn't mention your scenario where old data is being deleted.

Please also see the new support ticket SUPPORT-439 that I've opened for you for additional questions. We will continue to work on resolving this issue for you there.

Regards,
Thomas

Comment by larry.loi [ 27/Dec/12 ]

first of all, I've deleted some old data prior to 2012-12-19 from the collection counter_data, not deleted chunks. it should be normal operations.

second of all, Does new data re-use empty chunks?

third of all, our shard key combined collect_date (date) and game_session_id (UUID). this is a recommend configuration from "Scaling MongoDB"; with Coarsely ascending key + search key. I think it can fulfill ours and most of the situation.

Comment by Thomas Rueckstiess [ 26/Dec/12 ]

Deleting all data prior to 2012-12-19 would explain the uneven distribution of documents, as the chunks remain empty on their shards.

We don't recommend manually deleting empty chunks as it can cause other problems. Unfortunately, with your date-based shard key, if you prune old data the chunks will remain empty forever and thus confuse the balancer. There is an open SERVER issue that describes your problem and can be considered a feature request to facilitate such an operations: SERVER-2487

Generally, we do not recommend using a monotonically increasing field (such as ObjectIDs, dates or timestamps) as a shard key, as new data will always only get inserted into a single shard.

Now that we've determined that this is not a bug, I've opened the separate SUPPORT-439 ticket for you and would like us to continue all ongoing discussion regarding possible solutions privately there.

Regards,
Thomas

Comment by larry.loi [ 26/Dec/12 ]

some old data was removed. say before collect_date: 2012-12-19. do I need to remove old chunks manually?

Comment by larry.loi [ 26/Dec/12 ]

Yes, chunks are distributed but data doesn't in my case. and there is no big jump in terms of traffic or customers after checked. Humm it is quite strange. here is the db.counter_data.stats(). all the rows are inside "shard0001".

mongos> db.counter_data.stats()
{
        "sharded" : true,
        "ns" : "fryatt_mcms_production.counter_data",
        "count" : 41506017,
        "numExtents" : 100,
        "size" : 9961444080,
        "storageSize" : 37221679040,
        "totalIndexSize" : 11722969552,
        "indexSizes" : {
                "_id_" : 2187701376,
                "collect_date_1_game_session_id_1" : 4867810528,
                "game_session_id_1_counter_id_1" : 4667457648
        },
        "avgObjSize" : 240,
        "nindexes" : 3,
        "nchunks" : 825,
        "shards" : {
                "shard0000" : {
                        "ns" : "fryatt_mcms_production.counter_data",
                        "count" : 0,
                        "size" : 0,
                        "storageSize" : 9305419760,
                        "numExtents" : 25,
                        "nindexes" : 3,
                        "lastExtentSize" : 2146426864,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 24528,
                        "indexSizes" : {
                                "_id_" : 8176,
                                "collect_date_1_game_session_id_1" : 8176,
                                "game_session_id_1_counter_id_1" : 8176
                        },
                        "ok" : 1
                },
                "shard0001" : {
                        "ns" : "fryatt_mcms_production.counter_data",
                        "count" : 41506017,
                        "size" : 9961444080,
                        "avgObjSize" : 240,
                        "storageSize" : 13598273488,
                        "numExtents" : 27,
                        "nindexes" : 3,
                        "lastExtentSize" : 2146426864,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 11722895968,
                        "indexSizes" : {
                                "_id_" : 2187676848,
                                "collect_date_1_game_session_id_1" : 4867786000,
                                "game_session_id_1_counter_id_1" : 4667433120
                        },
                        "ok" : 1
                },
                "shard0002" : {
                        "ns" : "fryatt_mcms_production.counter_data",
                        "count" : 0,
                        "size" : 0,
                        "storageSize" : 7158992896,
                        "numExtents" : 24,
                        "nindexes" : 3,
                        "lastExtentSize" : 1861541888,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 24528,
                        "indexSizes" : {
                                "_id_" : 8176,
                                "collect_date_1_game_session_id_1" : 8176,
                                "game_session_id_1_counter_id_1" : 8176
                        },
                        "ok" : 1
                },
                "shard0003" : {
                        "ns" : "fryatt_mcms_production.counter_data",
                        "count" : 0,
                        "size" : 0,
                        "storageSize" : 7158992896,
                        "numExtents" : 24,
                        "nindexes" : 3,
                        "lastExtentSize" : 1861541888,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 24528,
                        "indexSizes" : {
                                "_id_" : 8176,
                                "collect_date_1_game_session_id_1" : 8176,
                                "game_session_id_1_counter_id_1" : 8176
                        },
                        "ok" : 1
                }
        },
        "ok" : 1
}

Comment by Thomas Rueckstiess [ 26/Dec/12 ]

It appears that the chunks are evenly distributed:

{  "_id" : "fryatt_mcms_production",  "partitioned" : true,  "primary" : "shard0000" }
	fryatt_mcms_production.counter_data chunks:
		shard0002	212
		shard0003	212
		shard0000	212
		shard0001	213

However, chunks are only balanced by numbers, not by their actual size. Therefore it's possible that some of the chunks carry a lot more documents than others. Usually, when that happens, large chunks get split into smaller ones and distributed over the shards. In some cases, it is possible that a chunk becomes unsplittable, for example if all the documents contain the same shard key. In your case, this would be the same date and game_session_id number. Are you aware of any irregularities where a substantially higher number of documents with the same date and game_session_id have been inserted?

I've also noticed that shard0001 contains the most recent chunks that have a date range from Dec 19 to 24. I'm wondering if the unequal distribution could just be a natural artifact of the fact that the days leading up christmas are busier than others? Or did you perhaps experience a large increase in traffic or customers over the last days?

Could you please attach a db.counter_data.stats() from the fryatt_mcms_production database? This would show us the distribution of actual documents per shard for that particular collection. If we see the same discrepancy in the document numbers, we can then continue by finding out in which chunk(s) all these documents reside and why they aren't split and balanced.

Regards,
Thomas

Comment by larry.loi [ 25/Dec/12 ]

please see the attached sh_status.txt

Comment by larry.loi [ 25/Dec/12 ]

please see the output of db.locks.find().pretty()

mongos> db.locks.find().pretty()
{
"_id" : "balancer",
"process" : "prd-hgsms-app01.prod.laxigames.com:27007:1356277517:1804289383",
"state" : 0,
"ts" : ObjectId("50d96b3e01a519e8287ffec8"),
"when" : ISODate("2012-12-25T09:00:46.782Z"),
"who" : "prd-hgsms-app01.prod.laxigames.com:27007:1356277517:1804289383:Balancer:846930886",
"why" : "doing balance round"
}
{
"_id" : "fryatt_mcms_production.counter_data",
"process" : "prd-mcms-db01:27021:1356280569:953896823",
"state" : 0,
"ts" : ObjectId("50d7bc1260217eebb3696106"),
"when" : ISODate("2012-12-24T02:21:06.410Z"),
"who" : "prd-mcms-db01:27021:1356280569:953896823:conn64:292317335",
"why" : "split-

{ collect_date: \"2012-12-23\", game_session_id: \"ff351049-abc5-4867-90c5-d1d8cbb4c4ef\" }

"
}

Comment by Thomas Rueckstiess [ 24/Dec/12 ]

Hi Larry,

Can we have a look at the output of a sh.status(true) from the mongos? If the list is very long, you can also pipe it into a file by running this from your command line prompt

mongos --host <host>:<port> --eval "sh.status(true)" > sh_status.txt 

and attach the file to the ticket.

Additionally, I'd like to look at the locks collection. Can you attach the output of the following commands (also run in the mongos shell) as well?

use config
db.locks.find().pretty()

Thanks,
Thomas

Comment by larry.loi [ 23/Dec/12 ]

most of data holding on shard "prd-mcms-vdb01.prod.laxigames.com:27021"

mongos> db.stats()
{
"raw" : {
"prd-mcms-vdb01.prod.laxigames.com:27020" :

{ "db" : "fryatt_mcms_production", "collections" : 6, "objects" : 144, "avgObjSize" : 110.97222222222223, "dataSize" : 15980, "storageSize" : 9339326448, "numExtents" : 39, "indexes" : 10, "indexSize" : 81760, "fileSize" : 25691160576, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb01.prod.laxigames.com:27021" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 37220978, "avgObjSize" : 239.99996894224543, "dataSize" : 8933033564, "storageSize" : 13636075472, "numExtents" : 36, "indexes" : 3, "indexSize" : 10649779616, "fileSize" : 34276900864, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27022" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 8, "avgObjSize" : 95.5, "dataSize" : 764, "storageSize" : 7196794880, "numExtents" : 33, "indexes" : 3, "indexSize" : 24528, "fileSize" : 19251855360, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27023" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 8, "avgObjSize" : 95.5, "dataSize" : 764, "storageSize" : 7196794880, "numExtents" : 33, "indexes" : 3, "indexSize" : 24528, "fileSize" : 19251855360, "nsSizeMB" : 16, "ok" : 1 }

},
"objects" : 37221138,
"avgObjSize" : 239.99940764841742,
"dataSize" : 8933051072,
"storageSize" : 37368991680,
"numExtents" : 141,
"indexes" : 19,
"indexSize" : 10649910432,
"fileSize" : 98471772160,
"ok" : 1
}

Generated at Thu Feb 08 03:16:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.