[SERVER-67051] Create per action memory and duration metric for linux Created: 06/Jun/22  Updated: 29/Oct/23  Resolved: 08/Jul/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.1.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Daniel Moody Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt Dependency
has to be done before SERVER-67052 Create per action CPU metrics for linux Closed
has to be done after SERVER-67044 Create build metrics CLI interface an... Closed
has to be done after SERVER-67109 create build metrics json validator Closed
Backwards Compatibility: Fully Compatible
Sprint: Dev Platform 2022-06-27, Dev Platform 2022-07-11
Participants:

 Description   

For Command line actions, we can take over the SPAWN construction variable, which will call our own spawn function and then we can tailor the command and process output before returning back to scons.

For function actions we can duck-type the FunctionAction calls to execute from our own function.

Memory

For Linux we can use procfs and the process spawn pid.

For function actions we can use https://pypi.org/project/memory-profiler/

CPU

for now we will leave the cpu field empty.

Duration

From python, we can measure the wall clock time of the subprocess. I plan on using https://docs.python.org/3/library/timeit.html#timeit.default_timer

Json Format:
 
Note: for the action field, either the function name or command line will be used.
{
 build_tasks: [{
	 array_index: <int>(2),
	 outputs: [<str>](2),
	 inputs: [<str>](2),
	 action: <str>(2),
 cpu_time: <float>(2),
 mem_usage: <long>(2),
	 start_time: <datetime timestamp>(2),
	 end_time: <datetime timestamp>(2)
}]
}



 Comments   
Comment by Githook User [ 08/Jul/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-67051 SERVER-67052 SERVER-67053 SERVER-67054 add per action metrics
Branch: master
https://github.com/mongodb/mongo/commit/847e884a5360a295921edd841e1de261ffdfeea4

Comment by Daniel Moody [ 27/Jun/22 ]

Wanted to note that when using a sampling method, like psutil, it is hard to make sure that the process is sampled as close as possible near the end of its life, which is also where it's most likely to have the true Peak Memory usage. We can use valgrind to get very accurate results, but valgrind will significantly affect CPU usage and and process duration, so if valgrind were in use it must be done in a separate measurement from CPU usage and duration. This implies valgrind will require building things twice.

We decided to continue with the sampling method with psutil for now, as it will still be able to identify problem areas even without the accuracy of valgrind.

Comment by Daniel Moody [ 08/Jun/22 ]

I'll keep in mind investigating the psutil common solution approach. I did mention in a comment in the tech design doc I would at least prototype a bit.

Comment by Alex Neben [ 07/Jun/22 ]

SERVER-67051, SERVER-67052, SERVER-67053, SERVER-67054  - these should be done by the same person since there is a lot of overlap

Generated at Thu Feb 08 06:07:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.