[SERVER-13552] remove unnecessary global lock during "replace" out action Created: 11/Apr/14  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Jianbin Wei Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 8
Labels: pull-request, query-44-grooming
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-30111 Mapreduce uses global lock when outpu... Closed
is duplicated by SERVER-6892 Convert GlobalWrite to db lock in mr.cpp Closed
is duplicated by SERVER-13449 Possible to prevent global lock durin... Closed
is duplicated by SERVER-13442 mapReduce nonAtomic output option Closed
Related
related to SERVER-7831 m/r: reduce post processing: reduce l... Backlog
is related to DOCS-3286 Map/reduce documentation is wrong abo... Closed
is related to SERVER-13748 Global Lock in Reduce Phase of Map/Re... Closed
Assigned Teams:
Query Execution
Operating System: ALL
Participants:
Case:

 Description   

During map-reduce operation there are unnecessary global lock used and should be removed.



 Comments   
Comment by Andre Spiegel [ 27/Apr/14 ]

Just so it doesn't get lost in the duplicate ticket (SERVER-13748): This global write lock occurs with both atomic and non-atomic output mode. While for non-atomic mode, the lock is released and re-acquired for every single document that is written, the global lock is pre-emptive in that case and thus has pretty much the same effect as in atomic mode (per kaloian.manassiev).

Comment by Mathias Stearn [ 25/Apr/14 ]

There is a potential solution in the pull request attached to SERVER-7831

Comment by Jianbin Wei [ 11/Apr/14 ]

Also in the same mr.cpp file, there are three TODO list about global lock

line 511
            Lock::GlobalWrite lock; // TODO(erh): this is how it was, but seems it doesn't need to be global
            return postProcessCollectionNonAtomic(op, pm);
 
line 547
            if (_config.outputOptions.outType == Config::REPLACE ||
                    _safeCount(_db, _config.outputOptions.finalNamespace) == 0) {
                Lock::GlobalWrite lock; // TODO(erh): why global???
                // replace: just rename from temp to final collection name, dropping previous collection
                _db.dropCollection( _config.outputOptions.finalNamespace );
                BSONObj info;
 
line 586
                while ( cursor->more() ) {
                    Lock::GlobalWrite lock; // TODO(erh) why global?
                    BSONObj temp = cursor->nextSafe();
                    BSONObj old;

In theory I don't see why global lock is needed in these cases. We use map reduce extensively and these issues make the whole system slow. I think mongodb should seriously minimize the usage of global lock.

Comment by Jianbin Wei [ 11/Apr/14 ]

Sorry it is during "replace" action.

mr.cpp:545

            if (_config.outputOptions.outType == Config::REPLACE ||
                    _safeCount(_db, _config.outputOptions.finalNamespace) == 0) {
                Lock::GlobalWrite lock; // TODO(erh): why global???

I am not sure about the checking for safeCount. In our case we are replacing a new temporary collection and it generates a global lock. We have to insert a dummy document into the temp collection and filter it out manually to workaround the global lock.

Generated at Thu Feb 08 03:32:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.