[SERVER-23208] Results of TryLinks with cached scons configs can be erroneous if command line options change the order of config tests Created: 17/Mar/16  Updated: 21/Aug/20  Resolved: 05/Feb/20

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 4.3.4

Type: Bug Priority: Minor - P4
Reporter: Eric Milkie Assignee: Andrew Morrow (Inactive)
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File config.log.1     File config.log.2     File config.log.3     File failedBuildConfig.log    
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

In a directory set up with the Enterprise module, type:

scons all

Eventually you get:

SSL not enabled. Enterprise MongoDB must be built with --ssl specified

Then type:

scons all --ssl

You get, eventually:

Checking that linking to OpenSSL works...no
SSL is enabled, but is unavailable
See /media/DATA2/data/m/ent/mongo/build/scons/config.log for details

Looking in config.log, this is apparently because the linker cannot find the "main" method in the conftest file that is linked. To fix the problem, you can do this:

scons all --ssl --config=force

Sprint: Dev Platform 2020-01-27, Dev Platform 2020-02-10
Participants:

 Description   

If I mistakenly run "scons" to build with the Enterprise module and forget --ssl, I end up breaking my cached config such that I need to use "--config=force". I'm not sure if there is a bug in scons TryLink, or in the way we're melding things from the Enterprise module into the configuration stage.



 Comments   
Comment by Githook User [ 21/Aug/20 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-50403 SERVER-23208 Pick SCons upstream fix for SCons issue #3469

https://github.com/SCons/scons/commit/7c32091fccad5e70755dc46174fe516fff4549c3
(cherry picked from commit 18966553fc7f9d350c3428ea1a23162aba2526ef)
Branch: v4.2
https://github.com/mongodb/mongo/commit/b0b9ed24ad0511a1f8d91d6731ed36459daef2dd

Comment by Andrew Morrow (Inactive) [ 05/Feb/20 ]

Let the celebrations (and backports requests) begin!

Comment by Githook User [ 05/Feb/20 ]

Author:

{'name': 'Andrew Morrow', 'username': 'acmorrow', 'email': 'acm@mongodb.com'}

Message: SERVER-23208 Pick SCons upstream fix for SCons issue #3469

https://github.com/SCons/scons/commit/7c32091fccad5e70755dc46174fe516fff4549c3
Branch: master
https://github.com/mongodb/mongo/commit/18966553fc7f9d350c3428ea1a23162aba2526ef

Comment by Andrew Morrow (Inactive) [ 05/Feb/20 ]

This has been merged upstream in https://github.com/SCons/scons/commit/7c32091fccad5e70755dc46174fe516fff4549c3

I will work to get it cherry-picked into our vendored SCons ASAP.

Comment by Andrew Morrow (Inactive) [ 16/Jan/20 ]

bdbaddog - OK, I've attached three config.log files here. The first one, config.log.1 matches up to the first --ssl=on build, and then config.log.2 matches up to the --ssl=off build, and then the last one, config.log.3, is from the failing third run where we switched back to --ssl=on again. Note that I added VERBOSE=1 to all of the command lines to generate these, as they are more complete when verbose mode is on.

Comment by Andrew Morrow (Inactive) [ 15/Jan/20 ]

The above repro was on community 4119889d9cf5d3e1b9e23ecd112c7d4bf9773390 and enterprise 914271f2efcc444af84d737a1120ecac74b00dfb.

Comment by Andrew Morrow (Inactive) [ 15/Jan/20 ]

bdbaddog - Here are updated reproduction instructions on master.

  • Clone the community repo, and enable the enterprise module. It is important to start with a clean tree, so remove any existing build directory or other local state.
  • Do a build with --ssl=on of the file build/opt/mongo/config.h as follows: /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --variables-files= --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars build/opt/mongo/config.h --ssl=on. The build will work.
  • Now do a build with --ssl=off, and the build will fail with a message saying that SSL is required for the enterprise build. This is expected and correct behavior: /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --variables-files= --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars build/opt/mongo/config.h --ssl=off.
  • Last, re-run the --ssl=on build from step two above: /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --variables-files= --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars build/opt/mongo/config.h --ssl=on. This build, unlike the first build, will fail, stating that SSL is enabled, but is unavailable, because a configure check that should have passed failed: Checking that linking to OpenSSL works...no.

Looking at the build/scons/config.log we see this:

cons: Configure: Checking that linking to OpenSSL works...
scons: Configure: "build/scons/opt/sconf_temp/conftest_63.c" is up to date.
scons: Configure: The original builder output was:
  |build/scons/opt/sconf_temp/conftest_63.c <-
  |  |
  |  |#include <openssl/err.h>
  |  |#include <openssl/ssl.h>
  |  |#include <stdlib.h>
  |  |
  |  |int main() {
  |  |    SSL_library_init();
  |  |    SSL_load_error_strings();
  |  |    ERR_load_crypto_strings();
  |  |
  |  |    OpenSSL_add_all_algorithms();
  |  |    ERR_free_strings();
  |  |
  |  |    return EXIT_SUCCESS;
  |  |}
  |  |
  |
scons: Configure: "build/scons/opt/sconf_temp/conftest_63.o" is up to date.
scons: Configure: The original builder output was:
  |Compiling build/scons/opt/sconf_temp/conftest_63.o
  |
Linking build/scons/opt/sconf_temp/conftest_63
/usr/lib/x86_64-linux-gnu/Scrt1.o:function _start: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
scons: Configure: no

This suggests that the numbering has gotten confused and we have tried to link an object file that didn't define main as the last step of this TryLink.

Comment by Daniel Gottlieb (Inactive) [ 30/Aug/19 ]

I ran into this today with our buildscript/scons.py on the 4.0 branch (logs attached). The --config=force workaround continues to fix the issue.

Comment by Eric Milkie [ 04/Apr/16 ]

Also I think the problem is restricted to configure checks that have multiple output items (that can be cached). TryLink outputs both an object file and an executable, so scons can get confused if the object file is already seemingly up-to-date but the executable binary is not.

Comment by Eric Milkie [ 17/Mar/16 ]

Ah, I think I see the problem. scons assumes that the conftests are always in the same order, no matter what command line flags are used to affect scons' behavior. When I run scons without "--ssl", my 40th configure test is a cpp file, which explains why the linker gets confused when it tries to link it as a C program later on.

Comment by Eric Milkie [ 17/Mar/16 ]

Here's the config.log output:

scons: Configure: The original builder output was:
  |gcc -o build/scons/sconf_temp/conftest_40.o -c -std=c99 -fno-omit-frame-pointer -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fno-builtin-memcmp -DNDEBUG build/scons/sconf_temp/conftest_40.c
  |
gcc -o build/scons/sconf_temp/conftest_40 -fPIC -pthread -Wl,-z,now -rdynamic -fuse-ld=gold build/scons/sconf_temp/conftest_40.o -Wl,--whole-archive -Wl,--start-group -Wl,--end-group -Wl,--no-whole-archive -lm -lssl -lcrypto
/usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../lib64/crt1.o:function _start: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
scons: Configure: no

Curiously, if I just run the link line by itself, I get the same error. However, if I run the compilation line (verbatim), and then run the link line, the link is successful. I think that means that there is a leftover conftest_40.o from the previous invocation of scons? I'm surprised that scons's configure up-to-date detection mechanism would be fooled by this.

Generated at Thu Feb 08 04:02:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.