[SERVER-25141] $near cursor stores buffered data in memory Created: 19/Jul/16  Updated: 15/Nov/21  Resolved: 22/Jul/16

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zhang Youdong Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File mongostat     File query.js     File query2.js     HTML File serverStatus     File userData.ar    
Operating System: ALL
Steps To Reproduce:

1. import data

mongorestore --host ip:port -d testdb -c userData --gzip --archive=userData.ar

2. start query

mongo --host ip:port query.js

use mongostat to watch the process, the resident meomry will increase continully until OOM.

Participants:

 Description   
Original summary

2d query makes mongod memory usage increase continually, maybe memory leak

Original Description

resident memory of mongod continually increase when query



 Comments   
Comment by Zhang Youdong [ 23/Jul/16 ]

Hi Thomas Schubert Thank you for detailed explaining this problem.

I think the $near query's behaivor should be documented, otherwise mongod may easily OOM when there are a lot of $near query concurrently. If users know this behavior, they can control the cursor by set a lower timeout or set

{singleBatch: true}

option to close the cursor when find completed if no getMore needed.

Comment by Kelsey Schubert [ 22/Jul/16 ]

Hi zyd_com,

The $near query is essentially a set of sorts as it examines the cells of data covering circles around the centroid. As a result, it must buffer data in memory.

The number of buffered documents is recorded in the explain output:

db.runCommand({ "explain": {"find" : "userData", "filter" : { "latlng" : { "$near" : [ 116.34642045073839, 39.87082232130999  ], "$maxDistance" : 0.9009009009009009  }  }, "ntoreturn" : 10000  }, "verbosity":"executionStats"})

After examining this output, we do not see anything to indicate a bug in the MongoDB server. The data buffered explains the increased memory use associated with the cursor.

Kind regards,
Thomas

Comment by Ramon Fernandez Marina [ 22/Jul/16 ]

Hi zyd_com: yes, this issue is assigned and is being investigated, but there's nothing to report yet. Please continue to watch the ticket for updates.

Thanks for your patience,
Ramón.

Comment by Zhang Youdong [ 22/Jul/16 ]

Is anybody tracking this issue?

Comment by Zhang Youdong [ 20/Jul/16 ]

mgset-1149113:PRIMARY> db.serverStatus().metrics.cursor
{
	"timedOut" : NumberLong(22139),
	"open" : {
		"noTimeout" : NumberLong(0),
		"pinned" : NumberLong(2),
		"total" : NumberLong(743)
	}
}
mgset-1149113:PRIMARY> db.serverStatus().tcmalloc
{
	"generic" : {
		"current_allocated_bytes" : NumberLong("3182535168"),
		"heap_size" : NumberLong("3329359872")
	},
	"tcmalloc" : {
		"pageheap_free_bytes" : 17801216,
		"pageheap_unmapped_bytes" : 10297344,
		"max_total_thread_cache_bytes" : NumberLong(1073741824),
		"current_total_thread_cache_bytes" : 27521496,
		"central_cache_free_bytes" : 76974120,
		"transfer_cache_free_bytes" : 14230528,
		"thread_cache_free_bytes" : 27521496,
		"aggressive_memory_decommit" : 0
	},
	"formattedString" : "------------------------------------------------\nMALLOC:     3182535168 ( 3035.1 MiB) Bytes in use by application\nMALLOC: +     17801216 (   17.0 MiB) Bytes in page heap freelist\nMALLOC: +     76974120 (   73.4 MiB) Bytes in central cache freelist\nMALLOC: +     14230528 (   13.6 MiB) Bytes in transfer cache freelist\nMALLOC: +     27521496 (   26.2 MiB) Bytes in thread cache freelists\nMALLOC: +     13185184 (   12.6 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =   3332247712 ( 3177.9 MiB) Actual memory used (physical + swap)\nMALLOC: +     10297344 (    9.8 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =   3342545056 ( 3187.7 MiB) Virtual address space used\nMALLOC:\nMALLOC:         175392              Spans in use\nMALLOC:             77              Thread heaps in use\nMALLOC:           8192              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}
mgset-1149113:PRIMARY> db.serverStatus().mem
{
	"bits" : 64,
	"resident" : 3178,
	"virtual" : 4054,
	"supported" : true,
	"mapped" : 0,
	"mappedWithJournal" : 0
}

after cursor timeout

mgset-1149113:PRIMARY> db.serverStatus().tcmalloc
{
	"generic" : {
		"current_allocated_bytes" : 815258776,
		"heap_size" : NumberLong("3329359872")
	},
	"tcmalloc" : {
		"pageheap_free_bytes" : NumberLong("2329747456"),
		"pageheap_unmapped_bytes" : 54976512,
		"max_total_thread_cache_bytes" : NumberLong(1073741824),
		"current_total_thread_cache_bytes" : 27592944,
		"central_cache_free_bytes" : 76202744,
		"transfer_cache_free_bytes" : 25581440,
		"thread_cache_free_bytes" : 27592944,
		"aggressive_memory_decommit" : 0
	},
	"formattedString" : "------------------------------------------------\nMALLOC:      815258776 (  777.5 MiB) Bytes in use by application\nMALLOC: +   2329747456 ( 2221.8 MiB) Bytes in page heap freelist\nMALLOC: +     76202744 (   72.7 MiB) Bytes in central cache freelist\nMALLOC: +     25581440 (   24.4 MiB) Bytes in transfer cache freelist\nMALLOC: +     27592944 (   26.3 MiB) Bytes in thread cache freelists\nMALLOC: +     13185184 (   12.6 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =   3287568544 ( 3135.3 MiB) Actual memory used (physical + swap)\nMALLOC: +     54976512 (   52.4 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =   3342545056 ( 3187.7 MiB) Virtual address space used\nMALLOC:\nMALLOC:          46157              Spans in use\nMALLOC:             77              Thread heaps in use\nMALLOC:           8192              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}
mgset-1149113:PRIMARY> db.serverStatus().mem
{
	"bits" : 64,
	"resident" : 3122,
	"virtual" : 4056,
	"supported" : true,
	"mapped" : 0,
	"mappedWithJournal" : 0
}

After cursor timeout, the memory usage did decrease, and tcmalloc hold the memory in freelist. (Maybe we should add an interface to call ReleaseFreeMemory())

I am confused that why the cursor take so much memory when doing 2d query? normal query didn't take so much memory. I thought curosr is just a position pointer clue.

Comment by Zhang Youdong [ 20/Jul/16 ]

I attached query2.js which do simple find(), the opened cursor also increse, but the memory usage doesn't increase, is the cursor with 2d query special ?

Comment by Kelsey Schubert [ 19/Jul/16 ]

Hi zyd_com,

Thanks for the report. The increase in memory that you are observing appears to be the result of keeping a large number of open cursors. Would you please confirm that decreasing the cursor timeout (SERVER-8188) resolves this issue?

Thank you,
Thomas

Generated at Thu Feb 08 04:08:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.