[SERVER-2118] Indexing during phase 2 of a Map Reduce execution blocks all db operations Created: 18/Nov/10  Updated: 15/Aug/12  Resolved: 29/May/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.6.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Giri Iyengar Assignee: Antoine Girbal
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 64 bit, Amazon EC2


Operating System: Linux
Participants:

 Description   

When we are running large Map Reduce jobs, we have noticed that an index building operation seems to happen as phase 2 of 3 phases. When this happens, it seems to prevent the database from being used. That is, the indexing is not happening in the background and also not during insertions into the temporary MR collections.

See output snippet from the currentOp command. This index building seemed to happen during the execution of a Map/Reduce job after the Map phase ended and before the Reduce phase began.

{
"opid" : 853796039,
"active" : true,
"lockType" : "write",
"waitingForLock" : false,
"secs_running" : 229,
"op" : "insert",
"ns" : "sociocast.system.indexes",
"client" : "0.0.0.0:0",
"desc" : "conn",
"msg" : "index: (2/3) btree bottom up 4200477/18418261 22%"
},



 Comments   
Comment by Antoine Girbal [ 29/May/12 ]

please reopen if needed

Comment by Antoine Girbal [ 29/May/12 ]

this ticket is old..
Looking at code the only indices created by MR are:

  • on "0" for the inc collection
  • on "_id" and potentially other ones for the temp collection, copied over from the pre-existing target collection
    In any case those are all done when the collections are empty, so there is no risk of blocking the database.
Generated at Thu Feb 08 02:59:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.