Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9820

Waited xx seconds for checkpoint file creation in test_timestamp_abort

    • 8
    • Storage Engines - 2022-11-14, 2023-06-27 Lord of the Sprints, 2024-02-20_A_near-death_puffin
    • v7.3

      mick.graham@mongodb.com has identified what he suspects is a deadlock over write locking between eviction and checkpoint.
      Next steps:

      1. Investigate and validate the finding.
        • Note there have not been any build failures for over 30 days, so this might not be worth investigating.
      2. Determine how to reproduce.
      3. Review current code to see if code on develop prevents this deadlock.
        • Can the fix be back ported?
        • If the current code does obviously prevent this deadlock the reproducer should be run against develop.

      spinlock-gcc-test on rhel80

      Host: i-048311c0d67e76841
      Project: wiredtiger-mongo-v4.4
      Commit: b28742ae

      Task Logs:

      spinlock-gcc-test task_log

      Logs:

      6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
      make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
      Making check in test/utility
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
      make[1]: Nothing to be done for 'check'.
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
      Making check in examples/c
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      PASS: ex_access
      PASS: ex_call_center
      PASS: ex_col_store
      PASS: ex_config_parse
      PASS: ex_all
      PASS: ex_cursor
      PASS: ex_data_source
      PASS: ex_event_handler
      PASS: ex_encrypt
      PASS: ex_extending
      PASS: ex_extractor
      PASS: ex_file_system
      PASS: ex_hello
      PASS: ex_pack
      PASS: ex_log
      PASS: ex_process
      PASS: ex_schema
      PASS: ex_smoke
      PASS: ex_stat
      PASS: ex_thread
      PASS: ex_backup
      PASS: ex_backup_block
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 22
      # PASS:  22
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
      Making check in test/bloom
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      PASS: t
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 1
      # PASS:  1
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
      Making check in test/checkpoint
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      PASS: smoke.sh
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 1
      # PASS:  1
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
      Making check in test/cppsuite
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      PASS: csuite_style_example_test
      PASS: run
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 2
      # PASS:  2
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
      Making check in test/csuite
      make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
      make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
      PASS: incr_backup/smoke.sh
      PASS: random_directio/smoke.sh
      PASS: test_scope
      PASS: tiered_abort/smoke.sh
      PASS: test_rwlock
      PASS: truncated_log/smoke.sh
      PASS: test_wt1965_col_efficiency
      PASS: test_wt2403_lsm_workload
      PASS: random_abort/smoke.sh
      PASS: test_wt2246_col_append
      PASS: wt2535_insert_race/smoke.sh
      PASS: wt2447_join_main_table/smoke.sh
      PASS: test_wt2695_checksum
      PASS: test_wt2592_join_schema
      PASS: schema_abort/smoke.sh
      PASS: test_wt2719_reconfig
      PASS: test_wt2834_join_bloom_fix
      PASS: test_wt2323_join_visibility
      PASS: test_wt2999_join_extractor
      PASS: test_wt3120_filesys
      PASS: test_wt3135_search_near_collator
      PASS: test_wt3184_dup_index_collator
      PASS: wt2853_perf/smoke.sh
      PASS: test_wt3363_checkpoint_op_races
      PASS: test_wt3874_pad_byte_collator
      PASS: wt4105_large_doc_small_upd/smoke.sh
      PASS: test_wt4117_checksum
      PASS: test_wt4156_metadata_salvage
      PASS: test_wt3338_partial_update
      PASS: test_wt4699_json
      PASS: test_wt4803_history_store_abort
      PASS: test_wt4891_meta_ckptlist_get_alloc
      PASS: wt6185_modify_ts/smoke.sh
      PASS: wt6616_checkpoint_oldest_ts/smoke.sh
      PASS: test_wt4333_handle_locks
      PASS: test_wt8057_compact_stress
      PASS: test_wt7989_compact_checkpoint
      PASS: wt2909_checkpoint_integrity/smoke.sh
      Command stopped early: context canceled
      fi
      Running task-timeout commands.
      Running command 'shell.exec' in "run wt hang analyzer" (step 1 of 1)
      # Dump core (-c) and debugger outputs (-o)
      wt_hang_analyzer_option="-c -o file -o stdout"
      echo "Calling the wt hang analyzer ..."
      PATH="/opt/mongodbtoolchain/gdb/bin:$PATH" /opt/mongodbtoolchain/v3/bin/python3 ../test/wt_hang_analyzer/wt_hang_analyzer.py $wt_hang_analyzer_option
      Calling the wt hang analyzer ...
      Python Version: 3.9.2 (default, Dec 17 2021, 20:59:40)
      [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
      OS: Linux-4.18.0-80.1.2.el8_0.x86_64-x86_64-with-glibc2.28
      Cannot determine Linux distro since Python is too old
      Current User: 1000
      Cannot determine Unix Current Login
      Getting list of processes using /usr/bin/ps
      ['/usr/bin/ps', '--version']
      ps from procps-ng 3.3.15
      ['/usr/bin/ps', '-eo', 'pid,args']
      Found 4 interesting processes [(705, 'platform-python'), (1154, 'platform-python'), (6353, 'test_timestamp_abort'), (6356, 'test_timestamp_abort')]
      Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 705
      Dumping core to dump_platform-python.705.core
      ['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
      /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:48: SyntaxWarning: "is not" with a literal. Did you mean "!="?
        if self.value is not '':
      /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:60: SyntaxWarning: "is not" with a literal. Did you mean "!="?
        if self.value is not '':
      GNU gdb (GDB) 8.3.1
      Copyright (C) 2019 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      ['/opt/mongodbtoolchain/gdb/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 705', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.705.core', '-ex', 'set confirm off', '-ex', 'quit']
      platform-python: No such file or directory.
      Attaching to process 705
      ptrace: Operation not permitted.
      No shared libraries loaded at this time.
      No threads.
      Target 'None' cannot support this command.
      You can't do that without a process to debug.
      Done analyzing platform-python process with PID 705
      Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 1154
      Dumping core to dump_platform-python.1154.core
      ['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
      GNU gdb (GDB) 8.3.1
      Copyright (C) 2019 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the ex
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v4.4_rhel80_spinlock_gcc_test_b28742aed07482b945164b755ebc3967d5e03851_22_08_23_23_12_04/0?type=T#L2213
      

      logs

        1. wt9820-stacktrace.txt
          23 kB
          Mick Graham

            Assignee:
            will.korteland@mongodb.com Will Korteland
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: