-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Fully Compatible
-
ALL
-
v4.0, v3.6
-
TIG 2018-09-24
-
0
-
5
It's possible that due to an ssh connection error, the remote command to internally crash a server will never run. The powertest.py script expects that the crash command will fail, as the ssh connection will be terminated. However, it should examine the output of the crash command to determine it it was actually run on the remote host.
Here's a case where the remote command failed to execute:
[2018/07/15 16:11:38.976] 2018-07-15 20:10:47,078 INFO Crashing server in 46 seconds
[2018/07/15 16:11:38.976] 2018-07-15 20:11:37,188 INFO Inserting canary document {'x': 1531685447.025} to DB power Collection cycle
[2018/07/15 16:11:38.976] ssh -o ServerAliveCountMax=10 -o ServerAliveInterval=6 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o ConnectionAttempts=20 -i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20 10.122.5.210 /bin/bash -c "$'source venv_powercycle/Scripts/activate; python -u powertest.py --remoteOperation --sshUserHost 10.122.5.210 --sshConnection \'-i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20\' --rsync --rsyncExcludeFiles diagnostic.data/metrics.interim* --backupPathBefore /log/powercycle/beforerecovery --backupPathAfter /log/powercycle/afterrecovery --validate local --canary local --docForCanary None --seedDocNum 10000 --crashOption \'notmyfault/notmyfaultc64.exe -accepteula crash 1\' --instanceId i-093c2bc45b5317756 --crashWaitTime 45 --jitterForCrashWaitTime 5 --numCrudClients 10 --numFsmClients 10 --rootDir /log/powercycle-mongodb_mongo_v3.6_windows_64_2k8_ssl_powercycle_syncdelay_WT_f1bcba35cefd0c5c0402e32575327a77507ac03e_18_07_14_22_41_33 --mongodbBinDir /log/powercycle --dbPath /data/db --logPath /log/powercycle/mongod.log --mongodUsablePorts 20000 20001 --mongodOptions \'--setParameter enableTestCommands=1 --syncdelay 10 --storageEngine wiredTiger\' --remotePython \'source venv_powercycle/Scripts/activate; python -u\' crash_server'"
[2018/07/15 16:12:29.518] 2018-07-15 20:12:16,477 INFO Connection timed out during banner exchange