Details
-
Build Failure
-
Status: Open
-
Major - P3
-
Resolution: Unresolved
-
None
-
None
-
None
-
0
-
8
-
StorEng - Refinement Pipeline, Storage Engines - 2022-11-14, 2023-06-27 Lord of the Sprints
Description
mick.graham@mongodb.com has identified what he suspects is a deadlock over write locking between eviction and checkpoint.
Next steps:
- Investigate and validate the finding.
- Note there have not been any build failures for over 30 days, so this might not be worth investigating.
- Determine how to reproduce.
- Review current code to see if code on develop prevents this deadlock.
- Can the fix be back ported?
- If the current code does obviously prevent this deadlock the reproducer should be run against develop.
Host: i-048311c0d67e76841
Project: wiredtiger-mongo-v4.4
Commit: b28742ae
Task Logs:
spinlock-gcc-test task_log
Logs:
6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
|
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
|
Making check in test/utility
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
|
make[1]: Nothing to be done for 'check'.
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
|
Making check in examples/c
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
make check-TESTS
|
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
PASS: ex_access
|
PASS: ex_call_center
|
PASS: ex_col_store
|
PASS: ex_config_parse
|
PASS: ex_all
|
PASS: ex_cursor
|
PASS: ex_data_source
|
PASS: ex_event_handler
|
PASS: ex_encrypt
|
PASS: ex_extending
|
PASS: ex_extractor
|
PASS: ex_file_system
|
PASS: ex_hello
|
PASS: ex_pack
|
PASS: ex_log
|
PASS: ex_process
|
PASS: ex_schema
|
PASS: ex_smoke
|
PASS: ex_stat
|
PASS: ex_thread
|
PASS: ex_backup
|
PASS: ex_backup_block
|
============================================================================
|
Testsuite summary for WiredTiger 10.0.2
|
============================================================================
|
# TOTAL: 22
|
# PASS: 22
|
# SKIP: 0
|
# XFAIL: 0
|
# FAIL: 0
|
# XPASS: 0
|
# ERROR: 0
|
============================================================================
|
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
|
Making check in test/bloom
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
make check-TESTS
|
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
PASS: t
|
============================================================================
|
Testsuite summary for WiredTiger 10.0.2
|
============================================================================
|
# TOTAL: 1
|
# PASS: 1
|
# SKIP: 0
|
# XFAIL: 0
|
# FAIL: 0
|
# XPASS: 0
|
# ERROR: 0
|
============================================================================
|
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
|
Making check in test/checkpoint
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
make check-TESTS
|
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
PASS: smoke.sh
|
============================================================================
|
Testsuite summary for WiredTiger 10.0.2
|
============================================================================
|
# TOTAL: 1
|
# PASS: 1
|
# SKIP: 0
|
# XFAIL: 0
|
# FAIL: 0
|
# XPASS: 0
|
# ERROR: 0
|
============================================================================
|
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
|
Making check in test/cppsuite
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
make check-TESTS
|
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
PASS: csuite_style_example_test
|
PASS: run
|
============================================================================
|
Testsuite summary for WiredTiger 10.0.2
|
============================================================================
|
# TOTAL: 2
|
# PASS: 2
|
# SKIP: 0
|
# XFAIL: 0
|
# FAIL: 0
|
# XPASS: 0
|
# ERROR: 0
|
============================================================================
|
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
|
Making check in test/csuite
|
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
|
make check-TESTS
|
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
|
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
|
PASS: incr_backup/smoke.sh
|
PASS: random_directio/smoke.sh
|
PASS: test_scope
|
PASS: tiered_abort/smoke.sh
|
PASS: test_rwlock
|
PASS: truncated_log/smoke.sh
|
PASS: test_wt1965_col_efficiency
|
PASS: test_wt2403_lsm_workload
|
PASS: random_abort/smoke.sh
|
PASS: test_wt2246_col_append
|
PASS: wt2535_insert_race/smoke.sh
|
PASS: wt2447_join_main_table/smoke.sh
|
PASS: test_wt2695_checksum
|
PASS: test_wt2592_join_schema
|
PASS: schema_abort/smoke.sh
|
PASS: test_wt2719_reconfig
|
PASS: test_wt2834_join_bloom_fix
|
PASS: test_wt2323_join_visibility
|
PASS: test_wt2999_join_extractor
|
PASS: test_wt3120_filesys
|
PASS: test_wt3135_search_near_collator
|
PASS: test_wt3184_dup_index_collator
|
PASS: wt2853_perf/smoke.sh
|
PASS: test_wt3363_checkpoint_op_races
|
PASS: test_wt3874_pad_byte_collator
|
PASS: wt4105_large_doc_small_upd/smoke.sh
|
PASS: test_wt4117_checksum
|
PASS: test_wt4156_metadata_salvage
|
PASS: test_wt3338_partial_update
|
PASS: test_wt4699_json
|
PASS: test_wt4803_history_store_abort
|
PASS: test_wt4891_meta_ckptlist_get_alloc
|
PASS: wt6185_modify_ts/smoke.sh
|
PASS: wt6616_checkpoint_oldest_ts/smoke.sh
|
PASS: test_wt4333_handle_locks
|
PASS: test_wt8057_compact_stress
|
PASS: test_wt7989_compact_checkpoint
|
PASS: wt2909_checkpoint_integrity/smoke.sh
|
Command stopped early: context canceled
|
fi
|
Running task-timeout commands.
|
Running command 'shell.exec' in "run wt hang analyzer" (step 1 of 1)
|
# Dump core (-c) and debugger outputs (-o)
|
wt_hang_analyzer_option="-c -o file -o stdout"
|
echo "Calling the wt hang analyzer ..."
|
PATH="/opt/mongodbtoolchain/gdb/bin:$PATH" /opt/mongodbtoolchain/v3/bin/python3 ../test/wt_hang_analyzer/wt_hang_analyzer.py $wt_hang_analyzer_option
|
Calling the wt hang analyzer ...
|
Python Version: 3.9.2 (default, Dec 17 2021, 20:59:40)
|
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
|
OS: Linux-4.18.0-80.1.2.el8_0.x86_64-x86_64-with-glibc2.28
|
Cannot determine Linux distro since Python is too old
|
Current User: 1000
|
Cannot determine Unix Current Login
|
Getting list of processes using /usr/bin/ps
|
['/usr/bin/ps', '--version']
|
ps from procps-ng 3.3.15
|
['/usr/bin/ps', '-eo', 'pid,args']
|
Found 4 interesting processes [(705, 'platform-python'), (1154, 'platform-python'), (6353, 'test_timestamp_abort'), (6356, 'test_timestamp_abort')]
|
Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 705
|
Dumping core to dump_platform-python.705.core
|
['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
|
/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:48: SyntaxWarning: "is not" with a literal. Did you mean "!="?
|
if self.value is not '':
|
/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:60: SyntaxWarning: "is not" with a literal. Did you mean "!="?
|
if self.value is not '':
|
GNU gdb (GDB) 8.3.1
|
Copyright (C) 2019 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law.
|
['/opt/mongodbtoolchain/gdb/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 705', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.705.core', '-ex', 'set confirm off', '-ex', 'quit']
|
platform-python: No such file or directory.
|
Attaching to process 705
|
ptrace: Operation not permitted.
|
No shared libraries loaded at this time.
|
No threads.
|
Target 'None' cannot support this command.
|
You can't do that without a process to debug.
|
Done analyzing platform-python process with PID 705
|
Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 1154
|
Dumping core to dump_platform-python.1154.core
|
['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
|
GNU gdb (GDB) 8.3.1
|
Copyright (C) 2019 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the ex
|
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v4.4_rhel80_spinlock_gcc_test_b28742aed07482b945164b755ebc3967d5e03851_22_08_23_23_12_04/0?type=T#L2213
|