[SERVER-67051] Create per action memory and duration metric for linux Created: 06/Jun/22 Updated: 29/Oct/23 Resolved: 08/Jul/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.1.0-rc0 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Daniel Moody | Assignee: | Daniel Moody |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | Dev Platform 2022-06-27, Dev Platform 2022-07-11 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
For Command line actions, we can take over the SPAWN construction variable, which will call our own spawn function and then we can tailor the command and process output before returning back to scons. For function actions we can duck-type the FunctionAction calls to execute from our own function. Memory For Linux we can use procfs and the process spawn pid. For function actions we can use https://pypi.org/project/memory-profiler/ CPU for now we will leave the cpu field empty. Duration From python, we can measure the wall clock time of the subprocess. I plan on using https://docs.python.org/3/library/timeit.html#timeit.default_timer
|
| Comments |
| Comment by Githook User [ 08/Jul/22 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: |
| Comment by Daniel Moody [ 27/Jun/22 ] |
|
Wanted to note that when using a sampling method, like psutil, it is hard to make sure that the process is sampled as close as possible near the end of its life, which is also where it's most likely to have the true Peak Memory usage. We can use valgrind to get very accurate results, but valgrind will significantly affect CPU usage and and process duration, so if valgrind were in use it must be done in a separate measurement from CPU usage and duration. This implies valgrind will require building things twice. We decided to continue with the sampling method with psutil for now, as it will still be able to identify problem areas even without the accuracy of valgrind. |
| Comment by Daniel Moody [ 08/Jun/22 ] |
|
I'll keep in mind investigating the psutil common solution approach. I did mention in a comment in the tech design doc I would at least prototype a bit. |
| Comment by Alex Neben [ 07/Jun/22 ] |
|
|