[SERVER-36842] Core dump not generated after invariant failure in powercycle hosts Created: 24/Aug/18  Updated: 29/Oct/23  Resolved: 17/Sep/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.9, 4.0.3, 4.1.4

Type: Bug Priority: Major - P3
Reporter: Yves Duhem Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6
Sprint: TIG 2018-09-24
Participants:
Linked BF Score: 45

 Description   

After an invariant failure, the powercycle_replication_smalloplog task fails to find the core dump on the remote host.

From the task log:

[2018/07/18 10:42:23.228] Return code: 1 for command ['buildscripts/remote_operations.py', '--verbose', '--userHost', 'ubuntu@10.122.3.1', '--operation', 'copy_from', '--sshConnectionOptions', '-i /data/mci/df66a34b85dac21a0a2b25a4431a40f9/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20', '--retries', '10', '--file', '/log/powercycle/*.core', '--file', '/log/powercycle/*.mdmp']
[2018/07/18 10:42:23.228] scp: /log/powercycle/*.core: No such file or directory
[2018/07/18 10:42:23.228] scp: /log/powercycle/*.mdmp: No such file or directory
[2018/07/18 10:42:23.231] Command failed: command [pid=27775] encountered problem: exit status 1 



 Comments   
Comment by Githook User [ 21/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Trap error when printing informational message

(cherry picked from commit 790cdfebf736064d3ab1368ce3a5c9bde71ef7c0)
Branch: v3.6
https://github.com/mongodb/mongo/commit/ba151950b1faff07afb1302bf770569f4eb9c7d0

Comment by Githook User [ 21/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Core dump not generated after invariant failure in powercycle hosts

(cherry picked from commit 9f634fbb3b49e1a591f054b097caf862f192f0c2)
Branch: v3.6
https://github.com/mongodb/mongo/commit/9947327e88bc43f187d235c44f79492dfb00bf0f

Comment by Githook User [ 20/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Trap error when printing informational message

(cherry picked from commit 790cdfebf736064d3ab1368ce3a5c9bde71ef7c0)
Branch: v4.0
https://github.com/mongodb/mongo/commit/693520e52214c72a12efd132d2274d4d32bc2f21

Comment by Githook User [ 20/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Core dump not generated after invariant failure in powercycle hosts

(cherry picked from commit 9f634fbb3b49e1a591f054b097caf862f192f0c2)
Branch: v4.0
https://github.com/mongodb/mongo/commit/30e55781d8e4e2860b778b4b57c31323f5a774a5

Comment by Githook User [ 18/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Trap error when printing informational message
Branch: master
https://github.com/mongodb/mongo/commit/790cdfebf736064d3ab1368ce3a5c9bde71ef7c0

Comment by Githook User [ 17/Sep/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-36842 Core dump not generated after invariant failure in powercycle hosts
Branch: master
https://github.com/mongodb/mongo/commit/9f634fbb3b49e1a591f054b097caf862f192f0c2

Comment by Jonathan Abrahams [ 14/Sep/18 ]

The ulimit has to be set to unlimited on the spawned EC2 hosts.

Generated at Thu Feb 08 04:44:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.