[WT-9820] task-timed-out: spinlock-gcc-test on rhel80 [wiredtiger-mongo-v4.4 @ b28742ae] - MongoDB Jira

XML

Word

Printable

JSON

Details

Type: Build Failure
Status: Open
Priority: Major - P3
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
- BB-Tools
- stability

Evergreen Project:
- wiredtiger-mongo-v4.4
Failure Type:
- task-timed-out
Failing Buildvariants:
- rhel80
Failing Tasks:
- spinlock-gcc-test
First Failing Revision:
- b28742aed07482b945164b755ebc3967d5e03851
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-9820
Count of Linked BFGs (Last 30 days):
0

Story Points:
8
Sprint:
StorEng - Refinement Pipeline, Storage Engines - 2022-11-14, 2023-06-27 Lord of the Sprints

Description

mick.graham@mongodb.com has identified what he suspects is a deadlock over write locking between eviction and checkpoint.
Next steps:

Investigate and validate the finding.
- Note there have not been any build failures for over 30 days, so this might not be worth investigating.
Determine how to reproduce.
Review current code to see if code on develop prevents this deadlock.
- Can the fix be back ported?
- If the current code does obviously prevent this deadlock the reproducer should be run against develop.

spinlock-gcc-test on rhel80

Host: i-048311c0d67e76841
Project: wiredtiger-mongo-v4.4
Commit: b28742ae

Task Logs:

task spinlock-gcc-test

spinlock-gcc-test task_log

Logs:

6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'

make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'

Making check in test/utility

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'

make[1]: Nothing to be done for 'check'.

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'

Making check in examples/c

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

make  check-TESTS

make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

PASS: ex_access

PASS: ex_call_center

PASS: ex_col_store

PASS: ex_config_parse

PASS: ex_all

PASS: ex_cursor

PASS: ex_data_source

PASS: ex_event_handler

PASS: ex_encrypt

PASS: ex_extending

PASS: ex_extractor

PASS: ex_file_system

PASS: ex_hello

PASS: ex_pack

PASS: ex_log

PASS: ex_process

PASS: ex_schema

PASS: ex_smoke

PASS: ex_stat

PASS: ex_thread

PASS: ex_backup

PASS: ex_backup_block

============================================================================

Testsuite summary for WiredTiger 10.0.2

============================================================================

# TOTAL: 22

# PASS:  22

# SKIP:  0

# XFAIL: 0

# FAIL:  0

# XPASS: 0

# ERROR: 0

============================================================================

make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'

Making check in test/bloom

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

make  check-TESTS

make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

PASS: t

============================================================================

Testsuite summary for WiredTiger 10.0.2

============================================================================

# TOTAL: 1

# PASS:  1

# SKIP:  0

# XFAIL: 0

# FAIL:  0

# XPASS: 0

# ERROR: 0

============================================================================

make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'

Making check in test/checkpoint

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

make  check-TESTS

make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

PASS: smoke.sh

============================================================================

Testsuite summary for WiredTiger 10.0.2

============================================================================

# TOTAL: 1

# PASS:  1

# SKIP:  0

# XFAIL: 0

# FAIL:  0

# XPASS: 0

# ERROR: 0

============================================================================

make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'

Making check in test/cppsuite

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

make  check-TESTS

make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

PASS: csuite_style_example_test

PASS: run

============================================================================

Testsuite summary for WiredTiger 10.0.2

============================================================================

# TOTAL: 2

# PASS:  2

# SKIP:  0

# XFAIL: 0

# FAIL:  0

# XPASS: 0

# ERROR: 0

============================================================================

make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'

Making check in test/csuite

make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'

make  check-TESTS

make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'

make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'

PASS: incr_backup/smoke.sh

PASS: random_directio/smoke.sh

PASS: test_scope

PASS: tiered_abort/smoke.sh

PASS: test_rwlock

PASS: truncated_log/smoke.sh

PASS: test_wt1965_col_efficiency

PASS: test_wt2403_lsm_workload

PASS: random_abort/smoke.sh

PASS: test_wt2246_col_append

PASS: wt2535_insert_race/smoke.sh

PASS: wt2447_join_main_table/smoke.sh

PASS: test_wt2695_checksum

PASS: test_wt2592_join_schema

PASS: schema_abort/smoke.sh

PASS: test_wt2719_reconfig

PASS: test_wt2834_join_bloom_fix

PASS: test_wt2323_join_visibility

PASS: test_wt2999_join_extractor

PASS: test_wt3120_filesys

PASS: test_wt3135_search_near_collator

PASS: test_wt3184_dup_index_collator

PASS: wt2853_perf/smoke.sh

PASS: test_wt3363_checkpoint_op_races

PASS: test_wt3874_pad_byte_collator

PASS: wt4105_large_doc_small_upd/smoke.sh

PASS: test_wt4117_checksum

PASS: test_wt4156_metadata_salvage

PASS: test_wt3338_partial_update

PASS: test_wt4699_json

PASS: test_wt4803_history_store_abort

PASS: test_wt4891_meta_ckptlist_get_alloc

PASS: wt6185_modify_ts/smoke.sh

PASS: wt6616_checkpoint_oldest_ts/smoke.sh

PASS: test_wt4333_handle_locks

PASS: test_wt8057_compact_stress

PASS: test_wt7989_compact_checkpoint

PASS: wt2909_checkpoint_integrity/smoke.sh

Command stopped early: context canceled

fi

Running task-timeout commands.

Running command 'shell.exec' in "run wt hang analyzer" (step 1 of 1)

# Dump core (-c) and debugger outputs (-o)

wt_hang_analyzer_option="-c -o file -o stdout"

echo "Calling the wt hang analyzer ..."

PATH="/opt/mongodbtoolchain/gdb/bin:$PATH" /opt/mongodbtoolchain/v3/bin/python3 ../test/wt_hang_analyzer/wt_hang_analyzer.py $wt_hang_analyzer_option

Calling the wt hang analyzer ...

Python Version: 3.9.2 (default, Dec 17 2021, 20:59:40)

[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]

OS: Linux-4.18.0-80.1.2.el8_0.x86_64-x86_64-with-glibc2.28

Cannot determine Linux distro since Python is too old

Current User: 1000

Cannot determine Unix Current Login

Getting list of processes using /usr/bin/ps

['/usr/bin/ps', '--version']

ps from procps-ng 3.3.15

['/usr/bin/ps', '-eo', 'pid,args']

Found 4 interesting processes [(705, 'platform-python'), (1154, 'platform-python'), (6353, 'test_timestamp_abort'), (6356, 'test_timestamp_abort')]

Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 705

Dumping core to dump_platform-python.705.core

['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']

/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:48: SyntaxWarning: "is not" with a literal. Did you mean "!="?

  if self.value is not '':

/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:60: SyntaxWarning: "is not" with a literal. Did you mean "!="?

  if self.value is not '':

GNU gdb (GDB) 8.3.1

Copyright (C) 2019 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

['/opt/mongodbtoolchain/gdb/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 705', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.705.core', '-ex', 'set confirm off', '-ex', 'quit']

platform-python: No such file or directory.

Attaching to process 705

ptrace: Operation not permitted.

No shared libraries loaded at this time.

No threads.

Target 'None' cannot support this command.

You can't do that without a process to debug.

Done analyzing platform-python process with PID 705

Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 1154

Dumping core to dump_platform-python.1154.core

['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']

GNU gdb (GDB) 8.3.1

Copyright (C) 2019 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the ex

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v4.4_rhel80_spinlock_gcc_test_b28742aed07482b945164b755ebc3967d5e03851_22_08_23_23_12_04/0?type=T#L2213

logs

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

wt9820-stacktrace.txt
23 kB
Nov 15 2022 04:23:41 AM UTC

Issue Links

is related to

WT-11585 test_timestamp_abort timed out: ftruncate-test on rhel80 (wiredtiger-mongo-v5.0)

Closed

related to

WT-9796 Add debugging for logging getting stuck

Closed

Activity

People

Assignee:: Unassigned

Reporter:: xgen-buildbaron-user

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: Aug 31 2022 06:14:16 PM UTC

Updated:: Sep 03 2023 10:16:10 PM UTC