[SERVER-29367] Add buildscripts/collect_resource_info.py script to collect system resource information in Evergreen Created: 24/May/17  Updated: 30/Oct/23  Resolved: 22/Jun/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.10

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Eddie Louie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-30285 The buildscripts/collect_resource_inf... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-06-19, TIG 2017-07-10
Participants:

 Description   

The gopsutil package has been integrated into the Evergreen agent (via grip) and system resource information about processes running on the machine are available by sending an HTTP GET request to localhost:2285/status. Since the data is not yet available through the Evergreen API, we should instead add a new background command/script to the etc/evergreen.yml project configuration file that periodically sends (e.g. every 5 seconds) an HTTP GET request to localhost:2285/status. The "ps_info" array in the Evergreen agent's response should be examined to determine the CPU, memory, I/O utilization of each individual process at/until the particular time instant the HTTP GET request was issued. The values extracted from the Evergreen agent's response should be written to a file as one line per process—including a timestamp—and uploaded to S3.

Note: To avoid having the size of the file balloon with information about idle system processes, it may be useful to only include processes within resmoke.py's process subtree.



 Comments   
Comment by Eddie Louie [ 23/Jun/17 ]

According to sam.kleinman currently there is no port of the gopsutil to Solaris platform, hence no process resource information is available on Solaris.

On Windows, there is a limitation that only info for a single process is returned, but not for the process tree. A ticket MAKE-220 has been opened to track this issue.

Comment by Githook User [ 22/Jun/17 ]

Author:

{u'username': u'elouie99', u'name': u'Eddie Louie', u'email': u'eddie.louie@mongodb.com'}

Message: SERVER-29367 Add buildscripts/collect_resource_info.py to collect system resource information in Evergreen
Branch: master
https://github.com/mongodb/mongo/commit/46f57a1e5213b0c06b54dcc65286f199155c126a

Comment by Eddie Louie [ 19/Jun/17 ]

Code review: https://mongodbcr.appspot.com/140250002/
Evergreen patch: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_concurrency_fd26a441a06498183f1217fa06f32f86e4a5a372_17_06_17_18_22_01

Comment by Max Hirschhorn [ 24/May/17 ]

$ curl localhost:2285/status
{
  "agent_revision": "cbb4eb409e60df316e65fc8bbd4b59b5dcad9bc8",
  "pid": 15794,
  ...
  "ps_info": [
   {
    "pid": 15794,
    "parentPid": 5872,
    "numThreads": 6,
    "command": "./executables/linux_amd64/main",
    "cpu": {
     "cpu": "cpu",
     "user": 0,
     "system": 0.01,
     "idle": 0,
     "nice": 0,
     "iowait": 0,
     "irq": 0,
     "softirq": 0,
     "steal": 0,
     "guest": 0,
     "guestNice": 0,
     "stolen": 0
    },
    "io": {
     "readCount": 243,
     "writeCount": 47,
     "readBytes": 339968,
     "writeBytes": 24576
    },
    "net": [
     {
      "name": "all",
      "bytesSent": 4922033151702,
      "bytesRecv": 5769721400235,
      "packetsSent": 2730485829,
      "packetsRecv": 4199734939,
      "errin": 0,
      "errout": 0,
      "dropin": 0,
      "dropout": 0,
      "fifoin": 0,
      "fifoout": 0
     }
    ],
    "mem": {
     "rss": 6352896,
     "vms": 196161536,
     "swap": 0
    },
    "memExtra": {
     "rss": 6352896,
     "vms": 196161536,
     "shared": 4349952,
     "text": 4935680,
     "lib": 0,
     "data": 0,
     "dirty": 180842496
    },
    "errors": [
     "process does not have children"
    ],
    "metadata": {
     "time": "0001-01-01T00:00:00Z"
    }
   }
  ],
  "task_id": ""
 }

Generated at Thu Feb 08 04:20:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.