[SERVER-13449] Possible to prevent global lock during map reduce? Created: 01/Apr/14  Updated: 10/Dec/14  Resolved: 24/Jul/14

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Garrett Kolpin Assignee: Mathias Stearn
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-13552 remove unnecessary global lock during... Backlog
Participants:

 Description   

During the reduce phase of a map-reduce job I'm running, it seems the entire mongod process becomes locked. I believe the following output shows the global write lock being held (from db.currentOp()):

 {
                        "opid" : 128630,
                        "active" : true,
                        "secs_running" : 626,
                        "op" : "query",
                        "ns" : "mr_universal_split_2.mr_collection0",
                        "query" : {
                                "$msg" : "query not recording (too large)"
                        },
                        "client" : "10.11.60.172:48370",
                        "desc" : "conn929",
                        "threadId" : "0x7e73fee9c700",
                        "connectionId" : 929,
                        "locks" : {
                                "^" : "W"
                        },
                        "waitingForLock" : false,
                        "msg" : "m/r: reduce post processing M/R Reduce Post Processing Progress: 3222/5517 58%",
                        "progress" : {
                                "done" : 3222,
                                "total" : 5517
                        },
                        "numYields" : 9040,
                        "lockStats" : {
                                "timeLockedMicros" : {
                                        "R" : NumberLong(0),
                                        "W" : NumberLong(79265742),
                                        "r" : NumberLong(56685244),
                                        "w" : NumberLong(728735)
                                },
                                "timeAcquiringMicros" : {
                                        "R" : NumberLong(0),
                                        "W" : NumberLong(44727900),
                                        "r" : NumberLong(449781385),
                                        "w" : NumberLong(164798959)
                                }
                        }
                }

Is there a way to prevent map-reduce from holding the global write lock during the reduce phase? I recently modified the MR job by passing nonAtomic: true to the output specification. This improves the situation, but it still causes other clients to hang for many seconds before returning results from queries on unrelated databases.



 Comments   
Comment by Ramon Fernandez Marina [ 24/Jul/14 ]

gkolpin, the behavior you describe is expected. That being said, SERVER-13552 was opened to improve the situation, so please tune in to that ticket for more information.

Regards,
Ramón.

Generated at Thu Feb 08 03:31:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.