[SERVER-23959] option to allow disabling job object creation in smoke.py for win32 Created: 27/Apr/16  Updated: 05/Apr/17  Resolved: 02/Feb/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 2.6.12, 3.0.11
Fix Version/s: 2.6.13, 3.0.15, 3.2.13, 3.4.3, 3.5.3

Type: Improvement Priority: Major - P3
Reporter: Ernie Hershey Assignee: Max Hirschhorn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-18727 Remove pkill lines from etc/evergreen... Closed
Related
related to SERVER-17526 option to allow disabling job object ... Closed
related to SERVER-23946 Disable Evergreen Job Cleanup on 3.0 ... Closed
related to TOOLS-1605 Conditionally create a Windows job ob... Closed
Backwards Compatibility: Fully Compatible
Backport Completed:
Backport Requested:
v3.4, v3.2, v3.0
Sprint: TIG 2017-01-02, TIG 2017-02-13
Participants:

 Description   

We should do the same thing in smoke.py (in v3.0 and v2.6 branches) that resmoke does and not create new job objects if it's already running in one. That will allow us to use Evergreen's sophisticated process cleanup logic and remove pkill lines from evergreen.yml that clutter it up.

cc max.hirschhorn, mpobrien



 Comments   
Comment by Githook User [ 14/Feb/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-23959 Conditionally create a Windows job object in smoke.py.

Changes smoke.py to (like resmoke.py) only create a job object if the
Python process isn't already inside of one. This allows the Evergreen
agent to create a job object for managing processes spawned by a task to
ensure they are cleaned up reliably.

(cherry picked from commit dd6acd3551dc9e409a863e517609f2509a0f87f7)
Branch: v2.6
https://github.com/mongodb/mongo/commit/6f6546be9b9554753d5cbf765b315817cb2eb45b

Comment by Githook User [ 13/Feb/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-23959 Conditionally create a Windows job object in smoke.py.

Changes smoke.py to (like resmoke.py) only create a job object if the
Python process isn't already inside of one. This allows the Evergreen
agent to create a job object for managing processes spawned by a task to
ensure they are cleaned up reliably.

(cherry picked from commit dd6acd3551dc9e409a863e517609f2509a0f87f7)
Branch: v3.0
https://github.com/mongodb/mongo/commit/ca3083852896e024a7da6a48eb4dc80de850ae2b

Comment by Githook User [ 13/Feb/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-23959 Conditionally create a Windows job object in smoke.py.

Changes smoke.py to (like resmoke.py) only create a job object if the
Python process isn't already inside of one. This allows the Evergreen
agent to create a job object for managing processes spawned by a task to
ensure they are cleaned up reliably.

(cherry picked from commit dd6acd3551dc9e409a863e517609f2509a0f87f7)
Branch: v3.2
https://github.com/mongodb/mongo/commit/c8dcbf11dc4b3824edce8cbd5d591e98e4067a4c

Comment by Githook User [ 13/Feb/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-23959 Conditionally create a Windows job object in smoke.py.

Changes smoke.py to (like resmoke.py) only create a job object if the
Python process isn't already inside of one. This allows the Evergreen
agent to create a job object for managing processes spawned by a task to
ensure they are cleaned up reliably.

(cherry picked from commit dd6acd3551dc9e409a863e517609f2509a0f87f7)
Branch: v3.4
https://github.com/mongodb/mongo/commit/da15419653527b74c76e87326d0fffec5dc2adfc

Comment by Githook User [ 02/Feb/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-23959 Conditionally create a Windows job object in smoke.py.

Changes smoke.py to (like resmoke.py) only create a job object if the
Python process isn't already inside of one. This allows the Evergreen
agent to create a job object for managing processes spawned by a task to
ensure they are cleaned up reliably.
Branch: master
https://github.com/mongodb/mongo/commit/dd6acd3551dc9e409a863e517609f2509a0f87f7

Comment by Mark Benvenuto [ 19/Jan/17 ]

As discussed, since we only need this fix it in a few cases to support tools, and 2.6/3.0 mongo, option 3 is a good choice. If the process is not killed by smoke.py, the process will be killed by the job object in the end.

Comment by Max Hirschhorn [ 19/Jan/17 ]

smoke.py currently relies on creating its own job object to ensure the Python process spawned to log output to logkeeper and the mongod process buildlogger.py ends up spawning are both terminated. I've been informed by sam.kleinman the Windows builders in Evergreen are all running Windows Server 2008, despite any display names (e.g. Windows Vista) suggesting otherwise. However, Windows Server 2008 is still too old of a Windows version to support nested job objects, which foils the obvious solution to this issue of assigning the aforementioned processes to multiple job objects.

Windows 7, Windows Server 2008 R2, Windows XP with SP3, Windows Server 2008, Windows Vista and Windows Server 2003: A process can be associated with only a single job. Nested jobs were introduced in Windows 8 and Windows Server 2012.

https://msdn.microsoft.com/en-us/library/windows/desktop/hh448388(v=vs.85).aspx

I've had the following ideas as to how to proceed on this ticket and would appreciate feedback from mark.benvenuto on his thoughts of the least bad option.

  1. Upgrade all Windows builders for the mongodb-mongo-xx and mongo-tools-xx Evergreen projects to Windows Server 2012 and use nested jobs objects.
  2. Backport the contents of the buildscripts/resmokelib/logging/ directory to avoid spawning a buildlogger.py process to log output to logkeeper by using resmoke.py's client instead.
  3. Have the Python process running buildlogger.py write proc.pid to its stdout so that smoke.py can call win32process.TerminateProcess(int(pid), -1).
  4. Create a pair of Windows event objects - one that gets set by smoke.py to signal the Python process running buildlogger.py to terminate the mongod process it spawned, and another that gets set in buildlogger.py after the mongod process was terminated to signal to smoke.py that the Python process running buildloggger.py can be terminated without "leaking" a child process.
Comment by Max Hirschhorn [ 07/Dec/16 ]

I'm moving this into "features we're not sure of" until there's consensus on EVG-1362.

Generated at Thu Feb 08 04:04:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.