Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-11410

Test Checkpoint Timeout Failure: Application Thread Timeout Due to Blocked Pages by Hazard Pointer

    • BermudaTriangle- 2023-09-05

      spinlock-gcc-test on rhel80

      Host: i-0676cbd6c88ec9d20
      Project: wiredtiger-mongo-v5.0
      Commit: b4a7c2ef

      Task Logs:

      spinlock-gcc-test task_log

      Logs:

      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/lang/python'
      make[2]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/lang/python'
      make[1]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/lang/python'
      Making check in test/utility
      make[1]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/utility'
      make[1]: Nothing to be done for 'check'.
      make[1]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/utility'
      Making check in examples/c
      make[1]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      make[3]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      PASS: ex_access
      PASS: ex_call_center
      PASS: ex_col_store
      PASS: ex_config_parse
      PASS: ex_all
      PASS: ex_cursor
      PASS: ex_data_source
      PASS: ex_event_handler
      PASS: ex_encrypt
      PASS: ex_extending
      PASS: ex_extractor
      PASS: ex_file_system
      PASS: ex_hello
      PASS: ex_pack
      PASS: ex_log
      PASS: ex_process
      PASS: ex_schema
      PASS: ex_smoke
      PASS: ex_stat
      PASS: ex_thread
      PASS: ex_backup
      PASS: ex_backup_block
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 22
      # PASS:  22
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      make[2]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      make[1]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/examples/c'
      Making check in test/bloom
      make[1]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      make[3]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      PASS: t
      ============================================================================
      Testsuite summary for WiredTiger 10.0.2
      ============================================================================
      # TOTAL: 1
      # PASS:  1
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  0
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      make[3]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      make[2]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      make[1]: Leaving directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/bloom'
      Making check in test/checkpoint
      make[1]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/checkpoint'
      make  check-TESTS
      make[2]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/checkpoint'
      make[3]: Entering directory '/data/mci/131138117ccd81a83b4f570a95f883f5/wiredtiger/build_posix/test/checkpoint'
      Command 'shell.exec' in function 'make check all' (step 3 of 5) stopped early: context canceled.
      fi
      Running task-timeout commands.
      Running command 'shell.exec' in function 'run wt hang analyzer' (step 1 of 1) in block 'timeout'.
      # Dump core (-c) and debugger outputs (-o)
      wt_hang_analyzer_option="-c -o file -o stdout"
      echo "Calling the wt hang analyzer ..."
      PATH="/opt/mongodbtoolchain/gdb/bin:$PATH" /opt/mongodbtoolchain/v3/bin/python3 ../test/wt_hang_analyzer/wt_hang_analyzer.py $wt_hang_analyzer_option
      Calling the wt hang analyzer ...
      Python Version: 3.9.2 (default, Jun 16 2023, 16:40:29)
      [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
      OS: Linux-4.18.0-80.1.2.el8_0.x86_64-x86_64-with-glibc2.28
      Cannot determine Linux distro since Python is too old
      Current User: 1000
      Cannot determine Unix Current Login
      Getting list of processes using /usr/bin/ps
      ['/usr/bin/ps', '--version']
      ps from procps-ng 3.3.15
      ['/usr/bin/ps', '-eo', 'pid,args']
      Found 3 interesting processes [(691, 'platform-python'), (1142, 'platform-python'), (1904, 't')]
      Debugger /usr/bin/gdb, analyzing platform-python process with PID 691
      Dumping core to dump_platform-python.691.core
      ['/usr/bin/gdb', '--version']
      GNU gdb (GDB) Red Hat Enterprise Linux 8.2-6.el8_0
      Copyright (C) 2018 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      ['/usr/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 691', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.691.core', '-ex', 'set confirm off', '-ex', 'quit']
      platform-python: No such file or directory.
      Attaching to process 691
      ptrace: Operation not permitted.
      No shared libraries loaded at this time.
      No threads.
      Target 'None' cannot support this command.
      You can't do that without a process to debug.
      Done analyzing platform-python process with PID 691
      Debugger /usr/bin/gdb, analyzing platform-python process with PID 1142
      Dumping core to dump_platform-python.1142.core
      ['/usr/bin/gdb', '--version']
      GNU gdb (GDB) Red Hat Enterprise Linux 8.2-6.el8_0
      Copyright (C) 2018 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      ['/usr/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 1142', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.1142.core', '-ex', 'set confirm off', '-ex', 'quit']
      platform-python: No such file or directory.
      Attaching to process 1142
      ptrace: Operation not permitted.
      No shared libraries loaded at this time.
      No threads.
      Target 'None' cannot support this command.
      You can't do that without a process to debug.
      Done analyzing platform-python process with PID 1142
      Debugger /usr/bin/gdb, analyzing t process with PID 1904
      Dumping core to dump_t.1904.core
      ['/usr/bin/gdb', '--version']
      GNU gdb (GDB) Red Hat Enterprise Linux 8.2-6.el8_0
      Copyright (C) 2018 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      ['/usr/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file t', '-ex', 'attach 1904', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_t.1904.core', '-ex', 'set confirm off', '-ex', 'quit']
      t: No such file or directory.
      Attaching to process 1904
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      0x00007f5f5b30f81d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
      From                To                  Syms Read   Shared Object Library
      0x00007f5f5b9338c0  0x00007f5f5b933d6f  Yes         /usr/local/lib/libeatmydata.so
      0x00007f5f5b72c5e0  0x00007f5f5b72f950  Yes (*)     /lib64/librt.so.1
      0x00007f5f5b527090  0x00007f5f5b527e59  Yes (*)     /lib64/libdl.so.2
      0x00007f5f5b30c8a0  0x00007f5f5b31ad75  Yes (*)     /lib64/libpthread.so.0
      0x00007f5f5af63a70  0x00007f5f5b0aefff  Yes (*)     /lib64/libc.so.6
      0x00007f5f5bb37030  0x00007f5f5bb561f4  Yes (*)     /lib64/ld-linux-x86-64.so.2
      (*): Shared library is missing debugging information.
        Id   Target Id                            Frame
      * 1    Thread 0x7f5f5bd4a100 (LWP 1904) "t" 0x00007f5f5b30f81d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
        2    Thread 0x7f5f5af41700 (LWP 1907) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        3    Thread 0x7f5f5a740700 (LWP 1908) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        4    Thread 0x7f5f53fff700 (LWP 1909) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        5    Thread 0x7f5f537fe700 (LWP 1910) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        6    Thread 0x7f5f52ffd700 (LWP 1911) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        7    Thread 0x7f5f527fc700 (LWP 1912) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        8    Thread 0x7f5f51ffb700 (LWP 1913) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        9    Thread 0x7f5f517fa700 (LWP 1914) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        10   Thread 0x7f5f50ff9700 (LWP 1915) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        11   Thread 0x7f5f4bfff700 (LWP 1916) "t" 0x00007f5f5b314878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/li
      

      logs

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: