-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
None
-
Correctness
-
Fully Compatible
-
ALL
-
2025-01-07
https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_8_64_bit_dynamic_all_feature_flags_core_analysis_config_fuzzer_replica_sets_jscore_passthrough_2_linux_enterprise1_c47KE_patch_798b10f3d2c9078a8c7f6cfa00cb1891588f1f7d_6765fc61c7537500072ccc95_24_12_20_23_23_34/tests?execution=0&sortBy=STATUS&sortDir=ASC is an example where processing a single core dump took 59 minutes. We observe file permission errors from the Python logic upon attempting to add the GDB index to the debug symbol files. While set index-cache enabled means GDB will build a GDB index itself, its implementation for doing so is likely single-threaded and will take a long time for the server binaries.
[2024/12/22 16:34:06.288] Failed to add gdb index to core-analyzer/install/dist-test/lib/libabsl_exponential_biased.so.debug [2024/12/22 16:34:06.288] Traceback (most recent call last): [2024/12/22 16:34:06.288] File "/data/mci/97f1dd7815af3f1270734bb649e1110a/src/buildscripts/resmokelib/hang_analyzer/extractor.py", line 273, in add_index [2024/12/22 16:34:06.288] subprocess.run( [2024/12/22 16:34:06.288] File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.ecN/lib/python3.10/subprocess.py", line 524, in run [2024/12/22 16:34:06.288] raise CalledProcessError(retcode, process.args, [2024/12/22 16:34:06.288] subprocess.CalledProcessError: Command '['/opt/mongodbtoolchain/v4/bin/objcopy', '--add-section', '.debug_names=core-analyzer/install/dist-test/lib/libabsl_exponential_biased.so.debug.debug_names', '--set-section-flags', '.debug_names=readonly', '--update-section', '.debug_str=core-analyzer/install/dist-test/lib/libabsl_exponential_biased.so.debug.debug_str.new', 'core-analyzer/install/dist-test/lib/libabsl_exponential_biased.so.debug', 'core-analyzer/install/dist-test/lib/libabsl_exponential_biased.so.debug']' returned non-zero exit status 1. [2024/12/22 16:34:06.289] /opt/mongodbtoolchain/v4/bin/objcopy: unable to copy file 'core-analyzer/install/dist-test/lib/libabsl_base.so.debug'; reason: Permission denied
The debug symbol files for the shared objects being non-writable appears to be a recent change. Perhaps it is part of the Bazel build system migration?
$ curl -s -o - 'https://mciuploads.s3.amazonaws.com/mongodb-mongo-master/linux-x86-dynamic-compile/6765fc61c7537500072ccc95/debugsymbols/debugsymbols-amirsaman.memaripour_2246.tgz' | tar tvf - ... -rwxrwxr-x 0 ec2-user ec2-user 20232 Dec 20 18:39 dist-test/bin/mongotmock.debug -rwxrwxr-x 0 ec2-user ec2-user 2349672 Dec 20 18:39 dist-test/bin/optimizer_gdb_test_program.debug -rwxrwxr-x 0 ec2-user ec2-user 1775552 Dec 20 18:39 dist-test/bin/pretty_printer_test_program.debug -rwxr-xr-x 0 ec2-user ec2-user 1056136 Dec 20 18:35 dist-test/bin/wt.debug -r-xr-xr-x 0 ec2-user ec2-user 25696 Dec 20 18:34 dist-test/lib/libabsl_bad_optional_access.so.debug -r-xr-xr-x 0 ec2-user ec2-user 25976 Dec 20 18:34 dist-test/lib/libabsl_bad_variant_access.so.debug -r-xr-xr-x 0 ec2-user ec2-user 345672 Dec 20 18:34 dist-test/lib/libabsl_base.so.debug -r-xr-xr-x 0 ec2-user ec2-user 83296 Dec 20 18:34 dist-test/lib/libabsl_city.so.debug
- is related to
-
SERVER-95782 Investigate why gdb takes a really long time to analyze cores sometimes
-
- Closed
-