[SERVER-18727] Remove pkill lines from etc/evergreen.yml Created: 29/May/15  Updated: 06/Dec/22  Resolved: 19/Nov/21

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Minor - P4
Reporter: Max Hirschhorn Assignee: Backlog - Server Tooling and Methods (STM) (Inactive)
Resolution: Won't Fix Votes: 0
Labels: stm, tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-17526 option to allow disabling job object ... Closed
depends on SERVER-23959 option to allow disabling job object ... Closed
Problem/Incident
Related
Assigned Teams:
Server Tooling & Methods
Sprint: TIG 2018-1-1, TIG 2017-12-18
Participants:
Linked BF Score: 0

 Description   

The shell.cleanup command should reliably kill processes that were started during the execution of the task.



 Comments   
Comment by Brooke Miller [ 19/Nov/21 ]

We don't have any plans to work on this in the near-future, unfortunately. However, in the longer-term we expect Evergreen's container initiative to resolve this issue. 

Comment by Jonathan Abrahams [ 07/Dec/17 ]

max.hirschhorn There are a lot of running processes that are being caught in the inclusion list, which should not necessarily cause a "system failure". It might be better to just flag this instead of failing

Original inclusion/exclusion list:

process_list="(^cl\.exe$|bsondump|java|lein|lldb|mongo|python|_test$|_test\.exe$)"
process_exclude_list="(main|tuned|evergreen)"

Latest exclusion list.

process_exclude_list="(main|tuned|evergreen|python.* /usr/bin/|python.* /usr/sbin/|python.* /usr/lib|easy_install |pip |yum-dump|rm |git ls-remote)"

Note that latest exclusion list was able to ignore all of the following processes that have been observed in the patch builds, except the grep -i ^pymongo=:

  • /usr/bin/python3 /usr/bin/unattended-upgrade
  • /usr/bin/python /usr/bin/landscape-sysinfo
  • /usr/bin/python /usr/local/bin/pip freeze
  • grep -i ^pymongo=
  • /usr/bin/python -Es /usr/libexec/rhsmcertd-worker --autoheal
  • /usr/bin/python /usr/sbin/registercloudguest
  • sh -c git ls-remote "https://github.com/10gen/mongo-orchestration.git" "v0.6.9*"
  • /usr/bin/python /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/provider/package/yum/yum-dump.py --options --installed-provides --yum-lock-timeout 30
  • rm -f /var/cache/apt/archives/libnss3-nssdb_2%3a3.28.4-0ubuntu0.16.04.3_all.deb /var/cache/apt/archives/rubygems-integration_1.10_all.deb /var/cache/apt/archives/libtimedate-perl_2.3000-2_all.deb /var/cache/apt/archives/libpython2.7-minimal_2.7.12-1ubuntu0~16.04.1_amd64.deb /var/cache/apt/archives/libpython2.7-dev_2.7.12-1ubuntu0~16.04.1_amd64.deb /var/cache/apt/archives/python2.7_2.7.12-1ubuntu0~16.04.1_amd64.deb /var/cache/apt/archives/python3.5-minimal_3.5.2-2ubuntu0~16.04.3_amd64.deb /var/cache/apt/archives/libcups2_2.1.3-4ubuntu0.3_amd64.deb /var/cache/apt/archives/gcc_4%3a5.3.1-1ubuntu1_amd64.deb /var/cache/apt/archives/glib-networking-common_2.48.2-1~ubuntu16.04.1_all.deb /var/cache/apt/archives/lib32stdc+6_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/libcurl4-openssl-dev_7.47.0-1ubuntu2.4_amd64.deb /var/cache/apt/archives/gcc-5_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/libxcb-render0_1.11.1-1ubuntu1_amd64.deb /var/cache/apt/archives/libflac8_1.3.1-4_amd64.deb /var/cache/apt/archives/swig_3.0.8-0ubuntu3_amd64.deb /var/cache/apt/archives/libxcb-dri2-0_1.11.1-1ubuntu1_amd64.deb /var/cache/apt/archives/libpython2.7_2.7.12-1ubuntu0~16.04.1_amd64.deb /var/cache/apt/archives/python-all_2.7.11-1_amd64.deb /var/cache/apt/archives/libev4_1%3a4.22-1_amd64.deb /var/cache/apt/archives/libssl-dev_1.0.2g-1ubuntu4.8_amd64.deb /var/cache/apt/archives/libllvm3.8_1%3a3.8-2ubuntu4_amd64.deb /var/cache/apt/archives/libstdc-5-dev_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/colord_1.2.12-1ubuntu1_amd64.deb /var/cache/apt/archives/libpthread-stubs0-dev_0.3-4_amd64.deb /var/cache/apt/archives/ca-certificates-java_20160321_all.deb /var/cache/apt/archives/cpp-5_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/binutils-mingw-w64-i686_2.26-3ubuntu1+6.6_amd64.deb /var/cache/apt/archives/libx32gomp1_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/cpp_4%3a5.3.1-1ubuntu1_amd64.deb /var/cache/apt/archives/gsettings-desktop-schemas_3.18.1-1ubuntu1_all.deb /var/cache/apt/archives/llvm-3.8-runtime_1%3a3.8-2ubuntu4_amd64.deb /var/cache/apt/archives/libxml-namespacesupport-perl_1.11-1_all.deb /var/cache/apt/archives/libpcre3-dev_2%3a8.38-3.1_amd64.deb /var/cache/apt/archives/libhtml-tree-perl_5.03-2_all.deb /var/cache/apt/archives/libboost-serialization1.58.0_1.58.0+dfsg-5ubuntu3.1_amd64.deb /var/cache/apt/archives/libx11-dev_2%3a1.6.3-1ubuntu2_amd64.deb /var/cache/apt/archives/fontconfig-config_2.11.94-0ubuntu1.1_all.deb /var/cache/apt/archives/libboost-filesystem-dev_1.58.0.1ubuntu1_amd64.deb /var/cache/apt/archives/libatk-bridge2.0-0_2.18.1-2ubuntu1_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.3_amd64.deb /var/cache/apt/archives/libxrender1_1%3a0.9.9-0ubuntu1_amd64.deb /var/cache/apt/archives/libdb5.3+-dev_5.3.28-11_amd64.deb /var/cache/apt/archives/libgd3_2.1.1-4ubuntu0.16.04.8_amd64.deb /var/cache/apt/archives/x11proto-render-dev_2%3a0.11.1-2_all.deb /var/cache/apt/archives/lib32mpx0_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/libsasl2-dev_2.1.26.dfsg1-14build1_amd64.deb /var/cache/apt/archives/libev-dev_1%3a4.22-1_amd64.deb /var/cache/apt/archives/x11-utils_7.7+3_amd64.deb /var/cache/apt/archives/libruby2.3_2.3.1-2~16.04.2_amd64.deb /var/cache/apt/archives/python-all-dev_2.7.11-1_amd64.deb /var/cache/apt/archives/libpython3.5-minimal_3.5.2-2ubuntu0~16.04.3_amd64.deb /var/cache/apt/archives/libcdt5_2.38.0-12ubuntu2.1_amd64.deb /var/cache/apt/archives/libxtst6_2%3a1.2.2-1_amd64.deb /var/cache/apt/archives/python3-alabaster_0.7.7-1_all.deb /var/cache/apt/archives/libglib2.0-dev_2.48.2-0ubuntu1_amd64.deb /var/cache/apt/archives/bzip2-doc_1.0.6-8_all.deb /var/cache/apt/archives/libxkbfile1_1%3a1.0.9-0ubuntu1_amd64.deb /var/cache/apt/archives/libtinfo-dev_6.0+20160213-1ubuntu1_amd64.deb /var/cache/apt/archives/lib32gcc-5-dev_5.4.0-6ubuntu1~16.04.5_amd64.deb /var/cache/apt/archives/libcroco3_0.6.11-1_amd64.deb /var/cache/apt/archives/libwebp5_0.4.4-1_amd64.deb /var/cache/apt/archives/python3-sphinx-rtd-theme_0.1.9-1_all.deb /var/cache/apt/archives/libc-dev-bin
  • /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/libxml2-dev_2.9.3+dfsg1-1ubuntu0.4_arm64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.4_arm64.deb /var/cache/apt/archives/libxml2-utils_2.9.3+dfsg1-1ubuntu0.4_arm64.deb /var/cache/apt/archives/linux-firmware_1.157.14_all.deb /var/cache/apt/archives/python-libxml2_2.9.3+dfsg1-1ubuntu0.4_arm64.deb
Comment by Githook User [ 05/Dec/17 ]

Author:

{'username': 'hptabster', 'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams'}

Message: Revert "SERVER-18727 Remove kill logic and fail if there are unexpected running processes in evergreen.yml"

This reverts commit 265b20e7fd053d0cd73506d78da0875bfdc7593d.
Branch: master
https://github.com/mongodb/mongo/commit/ca0d9be22ab42de52868a3060445ea5649b3c064

Comment by Githook User [ 04/Dec/17 ]

Author:

{'username': 'hptabster', 'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams'}

Message: SERVER-18727 Remove kill logic and fail if there are unexpected running processes in evergreen.yml
Branch: master
https://github.com/mongodb/mongo/commit/265b20e7fd053d0cd73506d78da0875bfdc7593d

Comment by Jonathan Abrahams [ 04/Dec/17 ]

max.hirschhorn I think the check for running processes should be invoked in "do setup", as not all tasks use "run tests".

Comment by Max Hirschhorn [ 01/Dec/17 ]

We cannot fail the Evergreen task during the phases which check for running processes.

jonathan.abrahams, could we check for the existence of a file at the beginning of the "run tests" function during the main part of the task (after the "pre" section) to cause the Evergreen to system fail before we'd run resmoke.py?

Comment by Jonathan Abrahams [ 01/Dec/17 ]

kevin.duong We cannot fail the Evergreen task during the phases which check for running processes.

Comment by Jonathan Abrahams [ 30/Nov/17 ]

Based on the previous comments in this ticket, these are the planned steps:

  • Rename references to kill processes function to check running processes
  • Remove the kill_process function in kill processes and report if any processes have been detected
Comment by Kevin Duong [ 03/Oct/17 ]

Work on this ticket should involve checking to see if any processes from earlier tasks are still running and causing the Evergreen task to system fail if any are found.

Comment by Max Hirschhorn [ 28/Sep/17 ]

I'm moving this back into "Needs triage" since Sam implemented EVG-1913 and EVG-1914 as what I had identified as the last remaining design flaws in Evergreen's process management.

Comment by Max Hirschhorn [ 07/Dec/16 ]

I'm moving this into "features we're not sure of" until there's consensus on EVG-1362.

Generated at Thu Feb 08 03:48:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.