[SERVER-9829] Add ability to separate data from indexes per extent in collStats verbose Created: 31/May/13  Updated: 31/May/13  Resolved: 31/May/13

Status: Closed
Project: Core Server
Component/s: Logging, Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Hows Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-3433 Command to report %-in memory per nam... Closed
duplicates SERVER-3109 Expose index stats Closed
Related
related to SERVER-9833 Add coll.indexPagesInRAM shell helper Open
Operating System: ALL
Participants:

 Description   

Currently when collstats in verbose mode we display a list of that collections extents and their offset in a given disk file.

Can we add stats that display what portion of the information in each extent is occupied by records and what portion is occupied by which index.



 Comments   
Comment by Scott Hernandez (Inactive) [ 31/May/13 ]

We have not had any bug reports from it, but we are still evaluating it from a summer research project. The feature is still being worked on (at some future point) so it is both experimental in terms of functionality and the level of testing. It is hard to know how many people are using it production but the code which does the in memory calculations are very simple since it just loops over the namespace extents to calculate what is in memory using mincore (on linux). I would consider this checking of what is ram much less experimental than other parts of the commands.

I would suggest looking over the code yourself (if you feel up to it) to judge the level of risk: https://github.com/mongodb/mongo/blob/master/src/mongo/db/commands/storage_details.cpp#L600

Comment by Adam Flynn [ 31/May/13 ]

Ah, cool. Didn't know storageDetails gave that low-level information.

What're the implications of enabling it? Being guarded by an experimental switch makes me a bit concerned about turning it on in prod.

Comment by Scott Hernandez (Inactive) [ 31/May/13 ]

Extents hold either index or collection data but not both.

Please not the the current storageDetails reports in memory status of collections/indexes (really it just deals with a namespace, and its extents).

Here is an example from the suggested helper, which you can add to your shell or emulate in your own program.

db.testcollection.indexPagesInRAM("a_1_b_1")
{
	"extents" : [
		{
			"pageBytes" : 4096,
			"onDiskBytes" : 36688,
			"inMem" : 0.8888888888888888
		},
		{
			"pageBytes" : 4096,
			"onDiskBytes" : 147280,
			"inMem" : 0.16666666666666666
		}
	],
	"ok" : 1
}

Comment by Adam Flynn [ 31/May/13 ]

Use case for this a tool we have that joins extent map against filesystem pages in cache to get resident memory breakdown by collection. Collection-level is nice, but it'd be really helpful if we could get collections broken down by data & each index to better understand what's hot.

Generated at Thu Feb 08 03:21:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.