[SERVER-9320] Several connections request to split a same chunk at the same time. Created: 11/Apr/13  Updated: 12/Apr/13  Resolved: 12/Apr/13

Status: Closed
Project: Core Server
Component/s: Performance, Sharding
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: stronglee Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 2.6.38-2-amd64
90 mongos linked to one cluster with 4 shards.


Issue Links:
Related
related to SERVER-9287 Decision to split chunk should happen... Closed
Operating System: Linux
Participants:

 Description   

Hi,
By looking into the log, I found that several connections will request to split a same chunk a the same time, which may affect the performance.

log:

 
Tue Apr  9 09:48:45 [conn57348] request split points lookup for chunk tw2_log_10191.gm_quest { : 387362153 } -->> { : 403798381 }
Tue Apr  9 09:48:45 [conn57347] request split points lookup for chunk tw2_log_10191.gm_quest { : 387362153 } -->> { : 403798381 }
Tue Apr  9 09:48:45 [conn57646] request split points lookup for chunk tw2_log_10191.gm_quest { : 387362153 } -->> { : 403798381 }
Tue Apr  9 09:48:45 [conn57648] request split points lookup for chunk tw2_log_10191.gm_quest { : 387362153 } -->> { : 403798381 }
Tue Apr  9 09:48:45 [conn57647] request split points lookup for chunk tw2_log_10191.gm_quest { : 387362153 } -->> { : 403798381 }
Tue Apr  9 09:48:46 [conn57348] warning: Finding the split vector for tw2_log_10191.gm_quest over { _sk: 1 } keyCount: 203360 numSplits: 1 lookedAt: 61189 took 1246ms
Tue Apr  9 09:48:46 [conn57348] command admin.$cmd command: { splitVector: "tw2_log_10191.gm_quest", keyPattern: { _sk: 1 }, min: { _sk: 387362153 }, max: { _sk: 403798381 }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000, $auth: { local: { __system: 2 } } } ntoreturn:1 keyUpdates:0 numYields: 301 locks(micros) r:2263610 reslen:70 1247ms
Tue Apr  9 09:48:46 [conn57647] warning: Finding the split vector for tw2_log_10191.gm_quest over { _sk: 1 } keyCount: 203360 numSplits: 1 lookedAt: 61191 took 1085ms
Tue Apr  9 09:48:46 [conn57647] command admin.$cmd command: { splitVector: "tw2_log_10191.gm_quest", keyPattern: { _sk: 1 }, min: { _sk: 387362153 }, max: { _sk: 403798381 }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000, $auth: { local: { __system: 2 } } } ntoreturn:1 keyUpdates:0 numYields: 32 locks(micros) r:2150012 reslen:70 1085ms
Tue Apr  9 09:48:46 [conn57646] warning: Finding the split vector for tw2_log_10191.gm_quest over { _sk: 1 } keyCount: 203360 numSplits: 1 lookedAt: 61193 took 1109ms
Tue Apr  9 09:48:46 [conn57646] command admin.$cmd command: { splitVector: "tw2_log_10191.gm_quest", keyPattern: { _sk: 1 }, min: { _sk: 387362153 }, max: { _sk: 403798381 }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000, $auth: { local: { __system: 2 } } } ntoreturn:1 keyUpdates:0 numYields: 33 locks(micros) r:2188370 reslen:70 1109ms
Tue Apr  9 09:48:46 [conn57648] warning: Finding the split vector for tw2_log_10191.gm_quest over { _sk: 1 } keyCount: 203360 numSplits: 1 lookedAt: 61196 took 1118ms
Tue Apr  9 09:48:46 [conn57648] command admin.$cmd command: { splitVector: "tw2_log_10191.gm_quest", keyPattern: { _sk: 1 }, min: { _sk: 387362153 }, max: { _sk: 403798381 }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000, $auth: { local: { __system: 2 } } } ntoreturn:1 keyUpdates:0 numYields: 22 locks(micros) r:2220804 reslen:70 1118ms
Tue Apr  9 09:48:46 [conn57347] warning: Finding the split vector for tw2_log_10191.gm_quest over { _sk: 1 } keyCount: 203360 numSplits: 1 lookedAt: 61202 took 1138ms
Tue Apr  9 09:48:46 [conn57347] command admin.$cmd command: { splitVector: "tw2_log_10191.gm_quest", keyPattern: { _sk: 1 }, min: { _sk: 387362153 }, max: { _sk: 403798381 }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000, $auth: { local: { __system: 2 } } } ntoreturn:1 keyUpdates:0 numYields: 102 locks(micros) r:2260997 reslen:70 1138ms



 Comments   
Comment by stronglee [ 11/Apr/13 ]

Hi Scott Hernandez,

Thank you for your introductions.

Comment by Scott Hernandez (Inactive) [ 11/Apr/13 ]

This is normal since every mongos may request a split point independently. These message are more a symptom of other activity on the system causing it to be slow, not because of the split requests themselves. Note how many yields the splitsVectors had to do to complete... this is an indication of a highly active (and probably over capacity) system.

Generated at Thu Feb 08 03:20:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.