[SERVER-34488] hang_analyzer.py fails because ptrace protection is not disabled Created: 15/Apr/18  Updated: 23/Apr/18  Resolved: 23/Apr/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
is related to SERVER-29797 Capability to invoke Hang_analyzer on... Closed
Operating System: ALL
Sprint: TIG 2018-05-07
Participants:
Linked BF Score: 19
Story Points: 3

 Description   

[2018/04/11 23:57:55.068] Return code: 0 for command ['buildscripts/remote_operations.py', '--verbose', '--userHost', 'ubuntu@10.122.4.143', '--sshConnectionOptions', '-i /data/mci/70130a2c6fcea1306c5dce5ebe2ec512/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=20 -o ConnectionAttempts=20', '--retries', '10', '--commands', 'PATH="/opt/mongodbtoolchain/gdb/bin:$PATH"  /opt/mongodbtoolchain/v2/bin/python2 buildscripts/hang_analyzer.py -c -o file -o stdout -p dbtest,java,mongo,mongod,mongos,python,_test -g bsondump,mongodump,mongoexport,mongofiles,mongoimport,mongoreplay,mongorestore,mongostat,mongotop', '--commandDir', '/log/powercycle']
...
[2018/04/11 23:57:55.068] Found 1 interesting processes [(2858, 'mongod')]
[2018/04/11 23:57:55.068] Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing mongod process with PID 2858
[2018/04/11 23:57:55.068] Dumping core to dump_mongod.2858.core
...
[2018/04/11 23:57:55.068] ['/opt/mongodbtoolchain/gdb/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file mongod', '-ex', 'attach 2858', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'set python print-stack full', '-ex', 'echo \\nWriting raw stacks to debugger_mongod_2858_raw_stacks.log.\\n', '-ex', 'set logging redirect on', '-ex', 'set logging file debugger_mongod_2858_raw_stacks.log', '-ex', 'set logging on', '-ex', 'thread apply all bt', '-ex', 'set logging off', '-ex', 'source /log/powercycle/buildscripts/gdb/mongo.py', '-ex', 'source /log/powercycle/buildscripts/gdb/mongo_printers.py', '-ex', 'source /log/powercycle/buildscripts/gdb/mongo_lock.py', '-ex', 'mongodb-uniqstack mongodb-bt-if-active', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_mongod.2858.core', '-ex', 'mongodb-dump-locks', '-ex', 'mongodb-show-locks', '-ex', 'mongodb-waitsfor-graph debugger_waitsfor_mongod_2858.gv', '-ex', 'mongodb-javascript-stack', '-ex', 'set confirm off', '-ex', 'quit']
[2018/04/11 23:57:55.068] Reading symbols from mongod...Reading symbols from /log/powercycle/mongod.debug...done.
[2018/04/11 23:57:55.068] done.
[2018/04/11 23:57:55.068] Attaching to program: /log/powercycle/mongod, process 2858
[2018/04/11 23:57:55.068] ptrace: Operation not permitted.



 Comments   
Comment by Jonathan Abrahams [ 23/Apr/18 ]

The build images were updated, so this work is no longer necessary.

Comment by Jonathan Abrahams [ 23/Apr/18 ]

We can disable the ptrace protection on the 2 distros, ubuntu1404-test & ubuntu1604-test. Once the build images have been updated we can remove this extra code.

Comment by Max Hirschhorn [ 20/Apr/18 ]

The ubuntu1404_test distro AMI does not disable ptrace protection. A new image will fix this issue.

jonathan.abrahams, could we change the configuration of the machine within the powercycle task ourselves? SERVER-28809 mentions in its description either (a) writing kernel.yama.ptrace_scope = 0 in the /etc/sysctl.d/10-ptrace.conf file or (b) doing echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope.

Comment by Jonathan Abrahams [ 20/Apr/18 ]

The ubuntu1404_test distro AMI does not disable ptrace protection. A new image will fix this issue.

Generated at Thu Feb 08 04:36:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.