Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36162

Powercycle - ensure internal crash command has been executed on the remote host

    XMLWordPrintable

Details

    • Fully Compatible
    • ALL
    • v4.0, v3.6
    • TIG 2018-09-24
    • 0
    • 5

    Description

      It's possible that due to an ssh connection error, the remote command to internally crash a server will never run. The powertest.py script expects that the crash command will fail, as the ssh connection will be terminated. However, it should examine the output of the crash command to determine it it was actually run on the remote host.

      Here's a case where the remote command failed to execute:

      [2018/07/15 16:11:38.976] 2018-07-15 20:10:47,078 INFO Crashing server in 46 seconds
      [2018/07/15 16:11:38.976] 2018-07-15 20:11:37,188 INFO Inserting canary document {'x': 1531685447.025} to DB power Collection cycle
      [2018/07/15 16:11:38.976] ssh -o ServerAliveCountMax=10 -o ServerAliveInterval=6 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o ConnectionAttempts=20 -i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20  10.122.5.210 /bin/bash -c "$'source venv_powercycle/Scripts/activate; python -u powertest.py --remoteOperation  --sshUserHost 10.122.5.210 --sshConnection \'-i /cygdrive/c/data/mci/3ab7f95ff9a32d5ea1ad8ffe3e1a09fd/powercycle.pem -o GSSAPIAuthentication=no -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o ConnectionAttempts=20\' --rsync  --rsyncExcludeFiles diagnostic.data/metrics.interim* --backupPathBefore /log/powercycle/beforerecovery --backupPathAfter /log/powercycle/afterrecovery --validate local --canary local --docForCanary None --seedDocNum 10000 --crashOption \'notmyfault/notmyfaultc64.exe -accepteula crash 1\' --instanceId i-093c2bc45b5317756 --crashWaitTime 45 --jitterForCrashWaitTime 5 --numCrudClients 10 --numFsmClients 10 --rootDir /log/powercycle-mongodb_mongo_v3.6_windows_64_2k8_ssl_powercycle_syncdelay_WT_f1bcba35cefd0c5c0402e32575327a77507ac03e_18_07_14_22_41_33 --mongodbBinDir /log/powercycle --dbPath /data/db --logPath /log/powercycle/mongod.log --mongodUsablePorts 20000 20001 --mongodOptions \'--setParameter enableTestCommands=1 --syncdelay 10 --storageEngine wiredTiger\' --remotePython \'source venv_powercycle/Scripts/activate; python -u\'   crash_server'"
      [2018/07/15 16:12:29.518] 2018-07-15 20:12:16,477 INFO Connection timed out during banner exchange
      

      Attachments

        Activity

          People

            jonathan.abrahams Jonathan Abrahams
            jonathan.abrahams Jonathan Abrahams
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: