[SERVER-3186] Setp5 of sharding process is very slow Created: 03/Jun/11  Updated: 29/May/12  Resolved: 22/Nov/11

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Senmiao Liu Assignee: Greg Studer
Resolution: Incomplete Votes: 0
Labels: sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

archlinux, soft raid 1, lvm, 7200rpm harddisk, xeon 3xxx, 4G ECC


Participants:

 Description   

The collection being sharded has 25M records.
First 4 steps only takes few seconds. The step5 (last one?) usually takes 20 minutes.
It would take about 2 days to shard the entire collection.
What does the `step5` mean? And how can I make the process faster?

the output of db.currentOp() :

{
"opid" : 6903,
"active" : true,
"lockType" : "write",
"waitingForLock" : false,
"secs_running" : 940,
"op" : "query",
"ns" : "my_db.my_collection",
"query" : {
"moveChunk" : "my_db.my_collection",
"from" : "rs1/mgrs1a:27017",
"to" : "rs3/mgrs3a:27017",
"min" :

{ "date" : "20100511", "place" : "02102010101" }

,
"max" :

{ "date" : "20100520", "place" : "02102010101" }

,
"maxChunkSizeBytes" : NumberLong(67108864),
"shardId" : "my_db.my_collection-date_\"20100511\"place_\"02102010101\"",
"configdb" : "mgconf1:27017,mgconf2:27017,mgconf3:27017"
},
"client" : "10.0.0.31:45831",
"desc" : "conn",
"msg" : "step5"
}



 Comments   
Comment by Eliot Horowitz (Inactive) [ 02/Sep/11 ]

Is this still an issue?

Comment by Senmiao Liu [ 04/Jun/11 ]

Yes, here's part of the output.
It seems step6 of `moveChunk.from` takes most time.

{
"_id" : "mgrs1a.example.com-2011-06-04T13:13:03-66",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T13:13:03.109Z"),
"what" : "moveChunk.start",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101018", "place" : "02107010704" }

,
"max" :

{ "date" : "20101018", "place" : "02108010702" }

,
"from" : "rs1",
"to" : "rs3"
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T13:12:57-65",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T13:12:57.870Z"),
"what" : "moveChunk.from",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101017", "place" : "02108010101" }

,
"max" :

{ "date" : "20101018", "place" : "02107010704" }

,
"step1" : 6,
"step2" : 100,
"step3" : 288,
"step4" : 51771,
"step5" : 530,
"step6" : 1099580
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:54:38-64",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:54:38.075Z"),
"what" : "moveChunk.commit",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101017", "place" : "02108010101" }

,
"max" :

{ "date" : "20101018", "place" : "02107010704" }

,
"from" : "rs1",
"to" : "rs2"
}
}
{
"_id" : "mgrs2a.example.com-2011-06-04T12:54:37-9",
"server" : "mgrs2a.example.com",
"clientAddr" : "(NONE)",
"time" : ISODate("2011-06-04T12:54:37.402Z"),
"what" : "moveChunk.to",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101017", "place" : "02108010101" }

,
"max" :

{ "date" : "20101018", "place" : "02107010704" }

,
"step1" : 71,
"step2" : 0,
"step3" : 50733,
"step4" : 0,
"step5" : 972
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:53:45-63",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:53:45.563Z"),
"what" : "moveChunk.start",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101017", "place" : "02108010101" }

,
"max" :

{ "date" : "20101018", "place" : "02107010704" }

,
"from" : "rs1",
"to" : "rs2"
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:53:40-62",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:53:40.155Z"),
"what" : "moveChunk.from",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "37101010402" }

,
"max" :

{ "date" : "20101017", "place" : "02108010101" }

,
"step1" : 7,
"step2" : 72,
"step3" : 167,
"step4" : 48159,
"step5" : 540,
"step6" : 1108040
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:35:11-61",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:35:11.947Z"),
"what" : "moveChunk.commit",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "37101010402" }

,
"max" :

{ "date" : "20101017", "place" : "02108010101" }

,
"from" : "rs1",
"to" : "rs3"
}
}
{
"_id" : "mgrs3a.example.com-2011-06-04T12:35:11-8",
"server" : "mgrs3a.example.com",
"clientAddr" : "(NONE)",
"time" : ISODate("2011-06-04T12:35:11.267Z"),
"what" : "moveChunk.to",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "37101010402" }

,
"max" :

{ "date" : "20101017", "place" : "02108010101" }

,
"step1" : 346,
"step2" : 0,
"step3" : 46895,
"step4" : 0,
"step5" : 923
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:34:23-60",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:34:23.084Z"),
"what" : "moveChunk.start",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "37101010402" }

,
"max" :

{ "date" : "20101017", "place" : "02108010101" }

,
"from" : "rs1",
"to" : "rs3"
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:34:16-59",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:34:16.360Z"),
"what" : "moveChunk.from",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "02107040302" }

,
"max" :

{ "date" : "20101016", "place" : "37101010402" }

,
"step1" : 6,
"step2" : 110,
"step3" : 174,
"step4" : 80444,
"step5" : 522,
"step6" : 1009546
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:17:26-58",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:17:26.424Z"),
"what" : "moveChunk.commit",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "02107040302" }

,
"max" :

{ "date" : "20101016", "place" : "37101010402" }

,
"from" : "rs1",
"to" : "rs2"
}
}
{
"_id" : "mgrs2a.example.com-2011-06-04T12:17:25-8",
"server" : "mgrs2a.example.com",
"clientAddr" : "(NONE)",
"time" : ISODate("2011-06-04T12:17:25.888Z"),
"what" : "moveChunk.to",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "02107040302" }

,
"max" :

{ "date" : "20101016", "place" : "37101010402" }

,
"step1" : 153,
"step2" : 0,
"step3" : 80125,
"step4" : 0,
"step5" : 166
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:16:05-57",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:16:05.486Z"),
"what" : "moveChunk.start",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101016", "place" : "02107040302" }

,
"max" :

{ "date" : "20101016", "place" : "37101010402" }

,
"from" : "rs1",
"to" : "rs2"
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T12:15:59-56",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T12:15:59.644Z"),
"what" : "moveChunk.from",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101015", "place" : "37101010302" }

,
"max" :

{ "date" : "20101016", "place" : "02107040302" }

,
"step1" : 14,
"step2" : 58,
"step3" : 428,
"step4" : 100782,
"step5" : 372,
"step6" : 1226892
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T11:55:32-55",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T11:55:32.450Z"),
"what" : "moveChunk.commit",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101015", "place" : "37101010302" }

,
"max" :

{ "date" : "20101016", "place" : "02107040302" }

,
"from" : "rs1",
"to" : "rs3"
}
}
{
"_id" : "mgrs3a.example.com-2011-06-04T11:55:32-7",
"server" : "mgrs3a.example.com",
"clientAddr" : "(NONE)",
"time" : ISODate("2011-06-04T11:55:32.034Z"),
"what" : "moveChunk.to",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101015", "place" : "37101010302" }

,
"max" :

{ "date" : "20101016", "place" : "02107040302" }

,
"step1" : 110,
"step2" : 0,
"step3" : 100527,
"step4" : 0,
"step5" : 151
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T11:53:50-54",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T11:53:50.977Z"),
"what" : "moveChunk.start",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101015", "place" : "37101010302" }

,
"max" :

{ "date" : "20101016", "place" : "02107040302" }

,
"from" : "rs1",
"to" : "rs3"
}
}
{
"_id" : "mgrs1a.example.com-2011-06-04T11:53:45-53",
"server" : "mgrs1a.example.com",
"clientAddr" : "10.0.0.31:38941",
"time" : ISODate("2011-06-04T11:53:45.682Z"),
"what" : "moveChunk.from",
"ns" : "mydb.cars",
"details" : {
"min" :

{ "date" : "20101015", "place" : "02107040302" }

,
"max" :

{ "date" : "20101015", "place" : "37101010302" }

,
"step1" : 10,
"step2" : 111,
"step3" : 176,
"step4" : 126143,
"step5" : 284,
"step6" : 992469
}
}

Comment by Eliot Horowitz (Inactive) [ 04/Jun/11 ]

Can you send the output of
use config
db.changelog.find().sort(

{ time : -1 }

).forEach( printjson )

Generated at Thu Feb 08 03:02:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.