[SERVER-37640] no member named 'SSLCopyRequestedPeerNameLength' in the global namespace Created: 15/Oct/18  Updated: 29/Oct/23  Resolved: 26/Oct/18

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 4.0.3
Fix Version/s: 4.0.4, 4.1.5

Type: Bug Priority: Major - P3
Reporter: Ryan Schmidt Assignee: Andrew Morrow (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File main.log.bz2    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:

Build mongodb 4.0.3 on OS X 10.11.

Sprint: Dev Tools 2018-10-22, Dev Tools 2018-11-05
Participants:

 Description   

Hello, we're able to build mongodb 4.0.3 on the 3 most recent versions of macOS (10.12, 10.13, 10.14), but on 10.11, the build fails:

/opt/local/bin/clang++-mp-5.0 -o build/MP/mongo/util/net/ssl_parameters.o -c -Woverloaded-virtual -Werror=unused-result -Wpessimizing-move -Wredundant-move -Wno-undefined-var-template -Wno-instantiation-after-specialization -stdlib=libc++ -std=c++14 -arch x86_64 -fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -Wno-inconsistent-missing-override -Wno-potentially-evaluated-expression -Wno-unused-lambda-capture -Wno-exceptions -fstack-protector-strong -fno-builtin-memcmp -DPCRE_STATIC -DNDEBUG -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -Isrc/third_party/asio-master/asio/include -Isrc/third_party/pcre-8.41 -Isrc/third_party/boost-1.60.0 -I/opt/local/include -Ibuild/MP -Isrc src/mongo/util/net/ssl_parameters.cpp
src/mongo/util/net/ssl_manager_apple.cpp:969:25: error: no member named 'SSLCopyRequestedPeerNameLength' in the global namespace
        auto status = ::SSLCopyRequestedPeerNameLength(_ssl.get(), &len);
                      ~~^
src/mongo/util/net/ssl_manager_apple.cpp:975:20: error: no member named 'SSLCopyRequestedPeerName' in the global namespace
        status = ::SSLCopyRequestedPeerName(_ssl.get(), &ret[0], &len);
                 ~~^
src/mongo/util/net/ssl_manager_apple.cpp:1360:57: error: cannot initialize a parameter of type 'SecTrustResultType * _Nullable' (aka 'unsigned int *') with an rvalue of type '(anonymous enum at /System/Library/Frameworks/Security.framework/Headers/SecTrust.h:87:1) *'
    uassertOSStatusOK(::SecTrustEvaluate(cftrust.get(), &result), ErrorCodes::SSLHandshakeFailed);
                                                        ^~~~~~~
/System/Library/Frameworks/Security.framework/Headers/SecTrust.h:345:78: note: passing argument to parameter 'result' here
OSStatus SecTrustEvaluate(SecTrustRef trust, SecTrustResultType * __nullable result)
                                                                             ^
/opt/local/bin/clang++-mp-5.0 -o build/MP/mongo/util/net/ssl_manager.o -c -Woverloaded-virtual -Werror=unused-result -Wpessimizing-move -Wredundant-move -Wno-undefined-var-template -Wno-instantiation-after-specialization -stdlib=libc++ -std=c++14 -arch x86_64 -fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -Wno-inconsistent-missing-override -Wno-potentially-evaluated-expression -Wno-unused-lambda-capture -Wno-exceptions -fstack-protector-strong -fno-builtin-memcmp -DPCRE_STATIC -DNDEBUG -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -Isrc/third_party/asio-master/asio/include -Isrc/third_party/pcre-8.41 -Isrc/third_party/boost-1.60.0 -I/opt/local/include -Ibuild/MP -Isrc src/mongo/util/net/ssl_manager.cpp
3 errors generated.
scons: *** [build/MP/mongo/util/net/ssl_manager_apple.o] Error 1
scons: building terminated because of errors.
build/MP/mongo/util/net/ssl_manager_apple.o failed: Error 1

This surprises me, because Apple documentation shows that the SSLCopyRequestedPeerName and SSLCopyRequestedPeerNameLength functions should be available on 10.11, and they do seem to be defined in the 10.11 SDK, so I don't know why it says they're not found.

I'm not familiar with C++ namespaces, and I was unfamiliar with the :: notation used in the source for accessing these functions, so I'm not sure whether the error message qualification in the global namespace is significant here, or whether these symbols would always be in the global namespace anyway.



 Comments   
Comment by Ryan Schmidt [ 26/Oct/18 ]

Sure, go ahead and close it. Thanks for working with me to find the answer. I've committed the change to MacPorts to use the 10.12 SDK for mongodb when building on 10.11.

For anyone else looking to build mongodb on 10.11 outside of MacPorts: that means using Xcode 8 and adding -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk to CCFLAGS and -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk to LINKFLAGS.

Also, mongodb requires a newer version of clang than Xcode 8.2.1 (the last version that works on 10.11) provides, so we use a newer open-source clang provided by MacPorts. clang 3.8 and later probably works; at the moment we happen to use clang 5.0.

The 10.12 SDK uses version 2 of Apple's text-based stub files, which is a feature that Apple's private versions of clang/llvm/ld64 understand, but which are not currently compatible with the open-source version of clang/llvm. For this reason, if you are building with open-source clang/llvm and have an open-source version of Apple's ld64 installed, you will get the error message ld: unexpected token: !tapi-tbd-v2 file if you try to use the 10.12 SDK. The solution is to use Xcode's ld64 instead. In MacPorts, that means installing the ld64 port with the +ld64_xcode variant. (If you're installing mongodb with MacPorts, it'll remind you to do that if you haven't already.)

Comment by Andrew Morrow (Inactive) [ 25/Oct/18 ]

ryandesign - I think I'd like to close out this ticket: I think we understand the issue, and I don't think it is a problem in practice if MacPorts is able to specify the minimum SDK. As mentioned before, it did cause us to learn something new and important about how these flags work, such that future versions of MongoDB are less likely to unintentionally end up in this state. Please let me know if you have any concerns about resolving this ticket.

Comment by Githook User [ 25/Oct/18 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-37640 Ensure we don't use OSX symbols above our target version

(cherry picked from commit 61cdb80fdc42e57c570066b17f0712c62b68db61)
Branch: v4.0
https://github.com/mongodb/mongo/commit/fc8f1106f494f24cb2559aa22c11aa825523541f

Comment by Githook User [ 25/Oct/18 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-37640 Ensure we don't use OSX symbols above our target version
Branch: master
https://github.com/mongodb/mongo/commit/61cdb80fdc42e57c570066b17f0712c62b68db61

Comment by Andrew Morrow (Inactive) [ 25/Oct/18 ]

All of our build systems and developer systems are almost certainly 10.12 or newer.

Comment by Andrew Morrow (Inactive) [ 25/Oct/18 ]

ryandesign - Looks like our messages crossed. It seems like we are getting close to an explanation. IIUC, the API call of relevance was made available as a public API in 10.11, but that this is not reflected in the 10.11 SDK, only the 10.12 SDK and later (an error on Apple's side?). In that case, I think we have all of the pieces of the puzzle, and your idea to require people building on 10.11 to have the 10.12 SDK would solve the issue.

Note that we will be committing a change (and backporting to 4.0) that adds the -Wunguarded-unavailable flag to our build. That was a very useful thing to learn as a side effect of this investigation.

Comment by Ryan Schmidt [ 25/Oct/18 ]

Andrew, on what version of macOS are you doing that build? If you're on 10.12 or later, you won't see the problem.

Comment by Sara Golemon [ 25/Oct/18 ]

Or, if this functionality being built here is optional, or if there is already an alternate implementation that can be used on earlier systems, then the build system could detect whether the right header is available and do the right thing in either case.

The previous implementation which doesn't depend on this OSX API uses OpenSSL for the crypto layer (by default, we use OSX's SecureTransport as of MongoDB 4.0). You can force using OpenSSL instead by passing --ssl-provider=openssl to scons, but bear in mind a few things.

  • Certificate selectors (via OSX Keychain) won't be available (documentation says this is available on MongoDB 4.0+ for OSX, but that assumes the SecureTransport implementation).
  • The OpenSSL as delivered by OSX is quite old (doesn't cover TLS 1.1 or later, for example).  If the ports builds provide a newer OpenSSL then that's a non-issue, just something to be aware of.
Comment by Andrew Morrow (Inactive) [ 25/Oct/18 ]

ryandesign - Then I think something else is going wrong. We did discover, BTW, that we were building with -mmacosx-version-min=10.10 in our official builds, which was an error: it should have said -mmacosx-version-min=10.11. Puzzled, we discovered that were missing a necessary flag (-Wunguarded-unavailable) to request enforcement: https://stackoverflow.com/questions/52977581/why-isnt-mmacosx-version-min-10-10-preventing-use-of-a-function-tagged-as-star/52993300#52993300

Once we added that flag, we did reproduce your error when building with -mmacosx-version-min=10.10, but did not reproduce with -mmacosx-version-min=10.11. So it is extremely odd to me that you are still getting this error when building with {-mmacosx-version-min=10.11}} on the compile line.

Could you write a small test program along the lines of the one I posted to SO and validate that the toolchain you are using is actually honoring the flags as expected?

Comment by Ryan Schmidt [ 25/Oct/18 ]

I have now misplaced the link, but I did find a reference to these functions being private API originally. On OS X 10.11, I can see that the Security framework provides the symbol in the library, but there's no mention of it in the headers:

$ grep SSLCopyRequestedPeerNameLength -r /System/Library/Frameworks/Security.framework/Versions/Current
Binary file /System/Library/Frameworks/Security.framework/Versions/Current/Security matches

It's not mentioned in the headers in the 10.11 SDK either (this is on OS X 10.10 with Xcode 7.2.1, which contains only the 10.11 SDK):

$ grep SSLCopyRequestedPeerNameLength -r /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Security.framework/Versions/Current
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Security.framework/Versions/Current/Security.tbd:                       _SSLCopyPeerTrust, _SSLCopyRequestedPeerName, _SSLCopyRequestedPeerNameLength,

It is mentioned in the 10.12 SDK (this is on OS X 10.11 with Xcode 8.2.1, which contains only the 10.12 SDK):

$ grep SSLCopyRequestedPeerNameLength -r /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Security.framework/Versions/Current
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Security.framework/Versions/Current/Headers/SecureTransport.h: * Determine the buffer size needed for SSLCopyRequestedPeerNameLength().
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Security.framework/Versions/Current/Headers/SecureTransport.h:SSLCopyRequestedPeerNameLength  (SSLContextRef  ctx,
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Security.framework/Versions/Current/Security.tbd:                       _SSLCopyRequestedPeerName, _SSLCopyRequestedPeerNameLength,

So I think probably the answer for users building on 10.11 will be to require them to use the 10.12 SDK. I can test that in MacPorts later.

Or, if this functionality being built here is optional, or if there is already an alternate implementation that can be used on earlier systems, then the build system could detect whether the right header is available and do the right thing in either case.

Comment by Ryan Schmidt [ 25/Oct/18 ]

Here is a log showing that using -mmacosx-version-min=10.11 doesn't make any difference:

main.log.bz2

This is not surprising, since the default deployment target in Mac OS X 10.5 and later is the same as the OS version (so on 10.11 the default deployment target is 10.11). Further, MacPorts sets the MACOSX_DEPLOYMENT_TARGET environment variable to the OS version (unless the port overrides that value, which our mongodb port does not), which is equivalent to using the -mmacosx-version-min flag.

Comment by Andrew Morrow (Inactive) [ 25/Oct/18 ]

ryandesign - I strongly suspect that this issue will go away for you if you add the following to your SCons invocation (or, add to the existing flags if you are already setting some):

scons CCFLAGS="-mmacosx-version-min=10.11" LDFLAGS="-mmacosx-version-min=10.11" ...

Note that flags should be space delimited, so if you were already passing, say, -flto to CCFLAGS, then you would want to make this CCFLAGS="-flto -mmacosx-version-min=10.11"

Could you please see if that resolves the issue for you?

Comment by Andrew Morrow (Inactive) [ 24/Oct/18 ]

Apparently, as of XCode 9, we need to pass the new magic flag -Wunguarded-availability to get enforcement of the availability macros, which is not enabled by -Wall, which is why this slipped by.

Comment by Andrew Morrow (Inactive) [ 22/Oct/18 ]

Thanks ryandesign, we are investigating

Comment by Ryan Schmidt [ 19/Oct/18 ]

The user definitely has a 10.11 machine, and so does our 10.11 build machine which experienced the same problem; here's the log: https://build.macports.org/builders/ports-10.11_x86_64-builder/builds/67885/steps/install-port/logs/stdio

Comment by Andrew Morrow (Inactive) [ 19/Oct/18 ]

OK, so that API should be available on 10.11:

/*
 * Server Only: obtain the hostname specified by the client in the ServerName extension (SNI)
 */
OSStatus
SSLCopyRequestedPeerNameLength  (SSLContextRef  ctx,
                                 size_t         *peerNameLen)
    __OSX_AVAILABLE_STARTING(__MAC_10_11, __IPHONE_9_0);

Note the __MAC_10_11 above.

I wonder if for whatever reason the system where the 10.11 build is happening isn't setting -mmacosx-version-min=10.11 on the compile and link lines via CCFLAGS=-mmacosx-version-min=10.11 LINKFLAGS=-mmacosx-version-min=10.11. ryandesign, is it possible that the machine where the 10.11 build is happening is actually an older macOS release, like 10.10, so that the default version minima are less than 10.11?

Generated at Thu Feb 08 04:46:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.