[SERVER-15778] ClientCursor "totalNoTimeout" counter incorrectly includes cursors that have been cleaned up Created: 22/Oct/14  Updated: 11/Jul/16  Resolved: 05/Dec/14

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 2.6.1, 2.6.5
Fix Version/s: 2.8.0-rc3

Type: Bug Priority: Major - P3
Reporter: Mateusz Jaworski Assignee: J Rassi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Every time I run a simple MapReduce command like this

db.runCommand({
    "mapReduce" : "test", 
    "map" : "function() { emit(this._id, 1)}",
    "reduce" : "function(key, values) { return Array.sum(values)}",
    "out" : {inline  :1}
    })

a new open cursor is added:

{
        "timedOut" : NumberLong(0),
        "open" : {
                "noTimeout" : NumberLong(<cursor++>),
                "pinned" : NumberLong(0),
                "total" : NumberLong(0)
        }
}

Participants:

 Description   

Every time I run mapReduce command a new cursor with noTimeout is added. Those cursors are never closed. On the production server I reached thousands of them. I tried with setting the timeout option somewhere, but I did not succeeded. The same issue is when I run

db.getCollection("$cmd").findOne

, but as far as I understood this is only a kind of alias for runCommand.

Initially I spotted it using node.js driver, but it in command line the behaviour is the same.

I tried to find a way to close the cursor in the documentation, but there is no mentioning that.



 Comments   
Comment by Githook User [ 17/Dec/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-15778 Fix ClientCursorPin::deleteUnderlying() race condition
Branch: master
https://github.com/mongodb/mongo/commit/31b4966485670b8351aed45227edf2ffacfa8444

Comment by Githook User [ 05/Dec/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-15778 Clean up ClientCursor

  • Removes ClientCursor::_pinValue and corresponding public methods.
  • Adds ClientCursor::_isPinned and ClientCursor::_isNoTimeout and
    corresponding public methods.
  • "isAggCursor" is now an immutable property of a ClientCursor.
  • It is now an error to destroy a pinned ClientCursor before unpinning
    (and, visibility of ~ClientCursor() changed to private to help
    enforce this).
    Branch: master
    https://github.com/mongodb/mongo/commit/5a0d148ffd2db70901eb0cdbeaf159b77bc1fc91
Comment by Githook User [ 05/Dec/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-15778 Fix ClientCursor "noTimeout" counter reporting error

The "noTimeout" counter was never being decremented for no-timeout
cursors that were cleaned up when pinned.
Branch: master
https://github.com/mongodb/mongo/commit/169b372f87441b6b0757c3c24bbc18f4f2b30f81

Comment by J Rassi [ 03/Dec/14 ]

I've diagnosed this as a reporting error, not a cursor leak. Those cursors are indeed being created by the mapReduce command and reflected in the "noTimeout" counter, but they are actually properly being cleaned up when the command finishes executing. The issue here is that the "noTimeout" counter is not being properly decremented in this case.

Please continue to watch this ticket for updates on when a fix may be available.

Comment by Mateusz Jaworski [ 03/Nov/14 ]

Thanks Ramon. Those cursors consume lots of RAM, this is why it is so problematic.

Comment by Ramon Fernandez Marina [ 29/Oct/14 ]

Thanks for your report mjaworsk. Looks like in SERVER-5845 the noTimeout option was added to mapReduce. However, mapReduce should exhaust the cursor and close it.

There's currently no way to kill cursors by id from the shell, but SERVER-5813 is opened to provide such a feature. If too many open cursors are an issue you may need to restart mongod periodically – I'll let you know if there's a better workaround.

Generated at Thu Feb 08 03:38:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.