[SERVER-25141] $near cursor stores buffered data in memory Created: 19/Jul/16 Updated: 15/Nov/21 Resolved: 22/Jul/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | 3.2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Zhang Youdong | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||
| Operating System: | ALL | ||
| Steps To Reproduce: | 1. import data
2. start query
use mongostat to watch the process, the resident meomry will increase continully until OOM. |
||
| Participants: |
| Description |
Original summary2d query makes mongod memory usage increase continually, maybe memory leak Original Descriptionresident memory of mongod continually increase when query |
| Comments |
| Comment by Zhang Youdong [ 23/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Thomas Schubert Thank you for detailed explaining this problem. I think the $near query's behaivor should be documented, otherwise mongod may easily OOM when there are a lot of $near query concurrently. If users know this behavior, they can control the cursor by set a lower timeout or set {singleBatch: true}option to close the cursor when find completed if no getMore needed. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 22/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi zyd_com, The $near query is essentially a set of sorts as it examines the cells of data covering circles around the centroid. As a result, it must buffer data in memory. The number of buffered documents is recorded in the explain output:
After examining this output, we do not see anything to indicate a bug in the MongoDB server. The data buffered explains the increased memory use associated with the cursor. Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 22/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi zyd_com: yes, this issue is assigned and is being investigated, but there's nothing to report yet. Please continue to watch the ticket for updates. Thanks for your patience, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zhang Youdong [ 22/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Is anybody tracking this issue? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zhang Youdong [ 20/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
after cursor timeout
After cursor timeout, the memory usage did decrease, and tcmalloc hold the memory in freelist. (Maybe we should add an interface to call ReleaseFreeMemory()) I am confused that why the cursor take so much memory when doing 2d query? normal query didn't take so much memory. I thought curosr is just a position pointer clue. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zhang Youdong [ 20/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I attached query2.js which do simple find(), the opened cursor also increse, but the memory usage doesn't increase, is the cursor with 2d query special ? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 19/Jul/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi zyd_com, Thanks for the report. The increase in memory that you are observing appears to be the result of keeping a large number of open cursors. Would you please confirm that decreasing the cursor timeout ( Thank you, |