[SERVER-46501] Add /proc/self/mountinfo to hostInfo response Created: 28/Feb/20  Updated: 29/Oct/23  Resolved: 20/Mar/20

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: 4.2.6, 4.4.0-rc0, 4.7.0

Type: Improvement Priority: Major - P3
Reporter: Billy Donahue Assignee: Billy Donahue
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
is related to SERVER-40507 Track I/O stats in diagnostic data wh... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4, v4.2
Sprint: Dev Tools 2020-03-09, Dev Tools 2020-03-23
Participants:

 Description   

Ticket SERVER-40507 requested adding Kubernetes Persistent Volume Claims to the FTDC stats. After a long investigation, they were found to already be there.

The problem was that there was nothing in FTDC binding the ephemeral block device name for the underlying Elastic Block Store device to the filesystem seen by mongod. So there was no way for an operator to interpret the FTDC stats and see this.

So we should add the mount table to the hostInfo. This way we can gather some clue as to why a disk would be seeing the I/O stats it's reporting.



 Comments   
Comment by Githook User [ 07/Apr/20 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-46501 add mount table to hostInfo

(cherry picked from commit 198a9d8ae3619362e00bd3a14e15e633ab731d18)
Branch: v4.2
https://github.com/mongodb/mongo/commit/4a806ff563725af9f36d115eeb5a99c7750f17c7

Comment by Githook User [ 06/Apr/20 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-46501 add mount table to hostInfo

(cherry picked from commit 198a9d8ae3619362e00bd3a14e15e633ab731d18)
Branch: v4.4
https://github.com/mongodb/mongo/commit/ec89b5d683680993f629d143992281b6c8ae521d

Comment by Billy Donahue [ 06/Apr/20 ]

4.2 backport requested in the comments. Should be easy.

Comment by James Broadhead (Inactive) [ 23/Mar/20 ]

billy.donahue same for MDB 4.2 (if feasible)

Comment by Billy Donahue [ 20/Mar/20 ]

 

Would this table be useful on pre-4.4 branches?

Comment by Githook User [ 20/Mar/20 ]

Author:

{'email': 'billy.donahue@mongodb.com', 'name': 'Billy Donahue', 'username': 'BillyDonahue'}

Message: SERVER-46501 add mount table to hostInfo
Branch: master
https://github.com/mongodb/mongo/commit/198a9d8ae3619362e00bd3a14e15e633ab731d18

Comment by Billy Donahue [ 20/Mar/20 ]

Shortened the names a bit. The records look like this now:

{
 "mountId": 87,
 "parentId": 27,
 "major": 8,
 "minor": 1,
 "root": "/",
 "mountPoint": "/boot",
 "options": "rw,relatime",
 "fields": "shared:32",
 "type": "ext2",
 "source": "/dev/sda1",
 "superOpt": "rw"
 },

I'm showing the record for a "real" block device here, so we can see that the "source" field has a useful name. This will be the key that maps these records to the FTDC disks stats.

I believe we can associate a k8s PV spec with a "mountPoint" to find a mountInfo record, get that record's corresponding "source" field, and use that to find the iostats in the ftdc disk stats array.

The mechanics of doing all that in operator code are going to be easier if hostInfo() returns a brokendown object of interpreted mountinfo fields.

Comment by Louis Plissonneau (Inactive) [ 20/Mar/20 ]

FYI james.broadhead

Comment by Sara Williamson [ 20/Mar/20 ]

We decided to go with JSON output.

CC louis.plissonneau andrey.belik crystal.horn

Comment by Billy Donahue [ 19/Mar/20 ]

http://mongodbcr.appspot.com/566860003

Comment by Billy Donahue [ 19/Mar/20 ]

Could report the lines of /proc/mountinfo as-is in the hostInfo output, or report a breakdown in JSON Objects, which takes a lot more "screen space" but is obviously easier to automatically consume.

 "mountInfo" : [
 "22 27 0:21 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw",
 "23 27 0:4 / /proc rw,nosuid,nodev,noexec,relatime shared:13 - proc proc rw",
 "24 27 0:6 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=32891944k,nr_inodes=8222986,mode=755",
 "25 24 0:22 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000",
 "26 27 0:23 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=6583068k,mode=755",
 "27 0 253:0 / / rw,relatime shared:1 - ext4 /dev/mapper/ubuntu--vg-root rw,errors=remount-ro,data=ordered",
 "28 22 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw",
 "29 24 0:24 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw",
 "30 26 0:25 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k",
 "31 22 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755",
 "32 31 0:27 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup rw",
 "33 31 0:28 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,xattr,name=systemd",
 "34 22 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:12 - pstore pstore rw",
 "35 31 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,hugetlb",
 "36 31 0:31 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,freezer",
 "37 31 0:32 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,net_cls,net_prio",
 "38 31 0:33 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,cpuset",
 "39 31 0:34 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,cpu,cpuacct",
 "40 31 0:35 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,pids",
 "41 31 0:36 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,rdma",
 "42 31 0:37 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,memory",
 "43 31 0:38 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,blkio",
 "44 31 0:39 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,devices",
 "45 31 0:40 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,perf_event",
 "46 23 0:41 / /proc/sys/fs/binfmt_misc rw,relatime shared:25 - autofs systemd-1 rw,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=25422",
 "48 24 0:19 / /dev/mqueue rw,relatime shared:26 - mqueue mqueue rw",
 "47 22 0:8 / /sys/kernel/debug rw,relatime shared:27 - debugfs debugfs rw",
 "49 24 0:42 / /dev/hugepages rw,relatime shared:28 - hugetlbfs hugetlbfs rw,pagesize=2M",
 "50 47 0:11 / /sys/kernel/debug/tracing rw,relatime shared:29 - tracefs tracefs rw",
 "51 22 0:20 / /sys/kernel/config rw,relatime shared:30 - configfs configfs rw",
 "52 22 0:43 / /sys/fs/fuse/connections rw,relatime shared:31 - fusectl fusectl rw",
 "87 27 8:1 / /boot rw,relatime shared:32 - ext2 /dev/sda1 rw",
 "86 27 253:1 / /mnt/bulk rw,relatime shared:33 - ext4 /dev/mapper/bulk--vg-bulk rw,data=ordered",
 "90 46 0:45 / /proc/sys/fs/binfmt_misc rw,relatime shared:34 - binfmt_misc binfmt_misc rw",
 "314 26 0:50 / /run/user/1000 rw,nosuid,nodev,relatime shared:233 - tmpfs tmpfs rw,size=6583064k,mode=700,uid=1000,gid=1000",
 "372 26 0:51 / /run/user/125 rw,nosuid,nodev,relatime shared:239 - tmpfs tmpfs rw,size=6583064k,mode=700,uid=125,gid=132",
 "420 314 0:52 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:283 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000",
 "349 27 7:0 / /snap/core/8592 ro,nodev,relatime shared:203 - squashfs /dev/loop0 ro",
 "478 27 7:2 / /snap/core/8689 ro,nodev,relatime shared:329 - squashfs /dev/loop2 ro",
 "490 27 7:3 / /snap/microk8s/1214 ro,nodev,relatime shared:337 - squashfs /dev/loop3 ro",
 "502 27 7:4 / /snap/microk8s/1247 ro,nodev,relatime shared:345 - squashfs /dev/loop4 ro"
 ]

Or....

 {
 "mountId": 22,
 "parentId": 27,
 "major": 0,
 "minor": 21,
 "root": "/",
 "mountPoint": "/sys",
 "mountOptions": "rw,nosuid,nodev,noexec,relatime",
 "optionalFields": "shared:7",
 "type": "sysfs",
 "mountSource": "sysfs",
 "superblockOptions": "rw"
 },
 {
 "mountId": 23,
 "parentId": 27,
 "major": 0,
 "minor": 4,
 "root": "/",
 "mountPoint": "/proc",
 "mountOptions": "rw,nosuid,nodev,noexec,relatime",
 "optionalFields": "shared:13",
 "type": "proc",
 "mountSource": "proc",
 "superblockOptions": "rw"
 },
 ...  and 39 more of those blocks on my machine.
 

Comment by Billy Donahue [ 28/Feb/20 ]

Essentially replaces SERVER-40507.

Generated at Thu Feb 08 05:11:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.