[SERVER-5140] SConstruct new smoke_python_name() hangs compilation Created: 29/Feb/12  Updated: 11/Jul/16  Resolved: 22/Mar/12

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 2.0.4
Fix Version/s: 2.0.5, 2.1.1

Type: Bug Priority: Minor - P4
Reporter: Ultrabug Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Gentoo Linux

dev-lang/python: 2.7.2-r3, 3.2.2


Attachments: Text File build.log     Text File debug.txt     Text File mongodb-2.0.3-fix-scons.patch    
Issue Links:
Duplicate
is duplicated by SERVER-5259 scons is stuck in futex system call Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

When running scons to compile mongoDB from source, the smoke tests hang when calling smoke_python_name() at the second binary loop in the function. The compilation is then stale and won't go any further.

This patch fixes the problem by checking for the python binary in PATH instead of only relying on the exception handler.

NB : this patch also fixes user respect of FLAGS but you can ignore it if you want, this bug is fixed with the @@ -827,14 +830,16 @@ part only



 Comments   
Comment by auto [ 22/Mar/12 ]

Author:

{u'login': u'andy10gen', u'email': u'schwerin@10gen.com', u'name': u'Andy Schwerin'}

Message: SERVER-5140: Short-circuit smoke_python_name() if the running python interpreter is new enough.

This version is for the master branch.

There's no reason to search for another version of the python interpreter if the
running version is at least the minimum supported version, and this stops a build
hang.
Branch: master
https://github.com/mongodb/mongo/commit/245468533a48c850d7d894950b047d9252707629

Comment by auto [ 20/Mar/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: SERVER-5140: Short-circuit smoke_python_name() if the running python interpreter is new enough.

There's no reason to search for another version of the python interpreter if the running version
is at least the minimum supported version, and this stops a build hang.
Branch: v2.0
https://github.com/mongodb/mongo/commit/bb76d2d736570a975ff4f609cae899e11d04eb9e

Comment by Ultrabug [ 20/Mar/12 ]

I will package 2.0.4 for Gentoo Linux using your patch until the next release, so yes I'm unstuck for now. Thanks mate.

Comment by Andy Schwerin [ 20/Mar/12 ]

Great. We'll get that into an upcoming 2.0 drop, and an equivalent into the next 2.1 drop. I'd been meaning to do it, anyways. Are you unstuck, for the time being?

Comment by Ultrabug [ 20/Mar/12 ]

@Andy : yes it works with your patch.

Comment by Andy Schwerin [ 20/Mar/12 ]

Can you try applying this patch, which should short-circuit the whole process in the event that you're running python2.5 or later:

diff --git a/buildscripts/utils.py b/buildscripts/utils.py
index 91409c7..bde2b08 100644
--- a/buildscripts/utils.py
+++ b/buildscripts/utils.py
@@ -3,6 +3,8 @@ import re
 import socket
 import time
 import os
+import sys
+
 # various utilities that are handy
 
 def getAllSourceFiles( arr=None , prefix="." ):
@@ -139,6 +141,14 @@ def smoke_python_name():
     # then we assume that "python" points to a 2.5 or
     # greater python VM. otherwise, explicitly use 2.5
     # which we assume to be installed.
+    min_version_tuple = (2, 5)
+    try:
+        if sys.version_info >= min_version_tuple:
+            return sys.executable
+    except AttributeError:
+        # In case the version of Python is somehow missing sys.version_info or sys.executable.
+        pass
+
     import subprocess
     version = re.compile(r'[Pp]ython ([\d\.]+)', re.MULTILINE)
     binaries = ['python2.5', 'python2.6', 'python2.7', 'python25', 'python26', 'python27', 'python']
@@ -150,7 +160,7 @@ def smoke_python_name():
                 match = version.search(stream)
                 if match:
                     versiontuple = tuple(map(int, match.group(1).split('.')))
-                    if versiontuple >= (2, 5):
+                    if versiontuple >= min_version_tuple:
                         return binary
         except:
             pass

Comment by Ultrabug [ 20/Mar/12 ]

Hi, I'm sorry to say that the problem is still there in 2.0.4 even with your previous commit. It hangs on the second iteration still.

Checking for C library pcap... yes
python2.5
exception [Errno 2] No such file or directory
python2.6

(hangs)

Comment by auto [ 01/Mar/12 ]

Author:

{u'login': u'dcrosta', u'name': u'Dan Crosta', u'email': u'dcrosta@10gen.com'}

Message: SERVER-5140: avoid a bug in SCons with subprocess.Popen

SCons overrides the subprocess module with its own custom implementation
(forked from an earlier version of Python, but with some patches).
SCons' implementation acquires a lock but does not release it during
Popen() if the command being executed cannot be found. The next
invocation of Popen() will hang indefinitely attempting to acquire the
lock.

Moving smoke_python_name() to a python module (i.e. out of the
SCons-managed files) causes "import subprocess" to find the actual,
system-installed subprocess module, which does not contain the buggy
locking.
Branch: v2.0
https://github.com/mongodb/mongo/commit/c5463305226dfb8f42eb5235ac381442006b29dd

Comment by Ultrabug [ 01/Mar/12 ]

This is exactly how I debugged this to make the patch Dan. Here is the modified function and its output from the unpatched SConstruct.

FYI, I also tried to swap the order of python2.5 and python2.6 in the binaries list with the same result.

Comment by Daniel Crosta [ 29/Feb/12 ]

Can you add some debugging into the SConstruct (the original, un-patched) version to see where it's getting stopped? I'd be curious, especially, to see information about what's doing inside the smoke_python_name loop (e.g. "print 'about to try %s' % binary", "print 'got exception %s when trying %s' % (e, binary)", stuff like that). If we can narrow down the problem, then we can be more confident in any patch we apply.

Comment by Ultrabug [ 29/Feb/12 ]

As per the attached build log, the last "print" I get is the detection of pcap of the checkLib, I add to go through debugging the SConstruct to narrow the problem in the smoke_python_name() function. That's where it hangs forever :

out, err = subprocess.Popen([binary, '-V'], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()

EDIT: yes, I expected the except to catch it too but it seems that it only does at the first iteration, once going through the second binary it just hangs there.

My python versions :
ls -l /usr/bin/python*
lrwxrwxrwx 1 root root 14 7 nov. 12:00 /usr/bin/python -> python-wrapper
lrwxrwxrwx 1 root root 9 7 nov. 12:00 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 6104 7 nov. 14:18 /usr/bin/python2.7
lrwxrwxrwx 1 root root 9 20 févr. 10:48 /usr/bin/python3 -> python3.2
-rwxr-xr-x 1 root root 10256 7 nov. 14:17 /usr/bin/python3.2
-rwxr-xr-x 1 root root 217 3 nov. 05:45 /usr/bin/python-config
-rwxr-xr-x 1 root root 1400 7 nov. 14:18 /usr/bin/python-config-2.7
-rwxr-xr-x 1 root root 1601 7 nov. 14:17 /usr/bin/python-config-3.2
-rwxr-xr-x 1 root root 10320 10 nov. 09:40 /usr/bin/python-wrapper

Comment by Ultrabug [ 29/Feb/12 ]

build log

Comment by Daniel Crosta [ 29/Feb/12 ]

Can you characterize what is hanging when it tries to execute the binary? If no such binary exists anywhere on the PATH, then the call to subprocess.Popen() should return essentially instantly. If this is not the case, I'd be curious to find out why (is there another executable shadowing one of the python executable names we look for?)

Generated at Thu Feb 08 03:08:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.