Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36162

Powercycle - ensure internal crash command has been executed on the remote host

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.0, v3.6
    • Sprint:
      TIG 2018-09-24
    • Linked BF Score:
      0
    • Story Points:
      5

      Description

      It's possible that due to an ssh connection error, the remote command to internally crash a server will never run. The powertest.py script expects that the crash command will fail, as the ssh connection will be terminated. However, it should examine the output of the crash command to determine it it was actually run on the remote host.

      Here's a case where the remote command failed to execute:

      [2018/07/15 16:11:38.976] 2018-07-15 20:10:47,078 INFO Crashing server in 46 seconds
      [2018/07/15 16:11:38.976] 2018-07-15 20:11:37,188 INFO Inserting canary document {'x': 1531685447.025} to DB power Collection cycle
      [2018/07/15 16:11:38.976] ssh -o ServerAliveCountMax=10 -o ServerAliveInterval=6 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o ConnectionAttempts=20 -i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20  10.122.5.210 /bin/bash -c "$'source venv_powercycle/Scripts/activate; python -u powertest.py --remoteOperation  --sshUserHost 10.122.5.210 --sshConnection \'-i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20\' --rsync  --rsyncExcludeFiles diagnostic.data/metrics.interim* --backupPathBefore /log/powercycle/beforerecovery --backupPathAfter /log/powercycle/afterrecovery --validate local --canary local --docForCanary None --seedDocNum 10000 --crashOption \'notmyfault/notmyfaultc64.exe -accepteula crash 1\' --instanceId i-093c2bc45b5317756 --crashWaitTime 45 --jitterForCrashWaitTime 5 --numCrudClients 10 --numFsmClients 10 --rootDir /log/powercycle-mongodb_mongo_v3.6_windows_64_2k8_ssl_powercycle_syncdelay_WT_f1bcba35cefd0c5c0402e32575327a77507ac03e_18_07_14_22_41_33 --mongodbBinDir /log/powercycle --dbPath /data/db --logPath /log/powercycle/mongod.log --mongodUsablePorts 20000 20001 --mongodOptions \'--setParameter enableTestCommands=1 --syncdelay 10 --storageEngine wiredTiger\' --remotePython \'source venv_powercycle/Scripts/activate; python -u\'   crash_server'"
      [2018/07/15 16:12:29.518] 2018-07-15 20:12:16,477 INFO Connection timed out during banner exchange
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: