-
Type:
Build Failure
-
Resolution: Gone away
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker
-
Storage Engines
-
None
-
None
-
- format-stress-data-validation-test-disagg-leader-palite-3
- spinlock-gcc-test
- spinlock-pthread-adaptive-test
- unit-test
- unit-test-bucket00
- unit-test-bucket01
- unit-test-bucket02
- unit-test-bucket04
- unit-test-bucket05
- unit-test-bucket06
- unit-test-bucket07
- unit-test-bucket08
- unit-test-bucket11
- unit-test-extra-long
- unit-test-hook-tiered
- unit-test-hook-tiered-s3
- unit-test-hook-tiered-with-delays
- unit-test-hook-timing-stress-log
- unit-test-macos
- unit-test-zstd
-
97
spinlock-pthread-adaptive-test on rhel80
Host: i-02e5a8b5d21ed527e
Project: wiredtiger
Commit: e143bc93
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
spinlock-pthread-adaptive-test task_log
Logs:
[pid:10010]: test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palite): 0.15 seconds [pid:10010]: ERROR in test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palite) [pid:10009/10010]: test_layered47.test_layered47.test_checkpoint_order_mismatch(palite) ... [pid:10010]: test_layered48.test_layered48.test_layered48 -s 1 (palite.shared): starting
spinlock-pthread-adaptive-test task_log
Logs:
[pid:10009]: ERROR: test_layered27.test_layered27.test_drain_insert_update(palite.large)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
self.verifyLayered()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
if self.conn is None or self.conn.this is None:
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
raise Exception('The connection is closed')
Exception: The connection is closed
======================================================================
[pid:10009]: ERROR: test_layered27.test_layered27.test_drain_remove_insert(palite.large)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered27.py", line 187, in test_drain_remove_insert
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config + 'disaggregated=(role="follower")')
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered30.test_layered30.test_layered30(palite.layered-prefix.one-table)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered30.py", line 109, in test_layered30
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' + self.conn_config)
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered31.test_layered31.test_layered31(palite)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered31.py", line 136, in test_layered31
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' + self.conn_base_config + 'disaggregated=(role="follower")')
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered38.test_layered38.test_gc_ingest_table_with_remove(palite)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered38.py", line 144, in test_gc_ingest_table_with_remove
oplog, t, conn_follow, session_follow = self.setup()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered38.py", line 91, in setup
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config + 'disaggregated=(role="follower")')
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered44.test_layered44.test_layered44
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered44.py", line 111, in test_layered44
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered45.test_layered45.test_prepare_update(palite)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
self.verifyLayered()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
if self.conn is None or self.conn.this is None:
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
raise Exception('The connection is closed')
Exception: The connection is closed
======================================================================
[pid:10009]: ERROR: test_layered47.test_layered47.test_first_gc_with_cursor_on_previous_checkpoint(palite)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 156, in test_first_gc_with_cursor_on_previous_checkpoint
self.setup([uri])
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 62, in setup
self.create_follower()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 82, in create_follower
self.conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered49.test_layered49.test_remove(palite)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered49.py", line 74, in test_remove
self.create_follower()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered49.py", line 66, in create_follower
self.conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
conn = wiredtiger.wiredtiger_open(home, config)
_wiredtiger.WiredTigerError: Device or resource busy
======================================================================
[pid:10009]: ERROR: test_layered27.test_layered27.test_drain_insert_update(palite.small)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
self.verifyLayered()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
if self.conn is None or self.conn.this is None:
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
raise Exception('The connection is closed')
Exception: The connection is closed
======================================================================
[pid:10009]: ERROR: test_layered27.test_layered27.test_drain_remove_insert(palite.small)
----------------------------------------------------------------------
[pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
method()
File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered27.py", line 187, in test_drain_remove_insert
conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config
spinlock-pthread-adaptive-test task_log
Logs:
[1761157251:021610][10013:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy [1761157251:021898][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761157251:021910][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761157251:021916][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761157251:021920][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761157251:021924][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761157251:021929][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761157251:021933][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy [1761157251:021938][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy [1761156951:463779][10011:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy [1761156951:464141][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761156951:464157][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761156951:464166][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761156951:464173][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761156951:464180][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761156951:464187][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found [1761156951:464194][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy [1761156951:464202][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy [1761157037:469443][10010:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy [1761157037:469767][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found [1761157037:469783][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found [1761157037:469789][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found [1761157037:469794][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found [1761157037:469798][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found [1761157037:469803][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found [1761157037:469807][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found [1761157037:469812][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found [1761157037:469816][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found [1761157037:469822][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_remove, 281: Error at src/meta/meta_table.c:281: "cursor->search(cursor)" failed: WT_NOTFOUND: item not found [1761157037:469827][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy [1761157037:469833][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy [1761158291:880015][10011:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy [1761158291:880368][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get_from_config, 911: Error at src/meta/meta_ckpt.c:911: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1761158291:880384][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get, 859: Error at src/meta/meta_ckpt.c:859: "__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)" failed: WT_NOTFOUND: item not found [1761158291:880394][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get_from_config, 911: Error at src/meta/meta_ckpt.c:911: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1761158291:880401][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get, 859: Error at src/meta/meta_ckpt.c:859: "__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)" failed: WT_NOTFOUND: item not found [1761158291:880408][100
spinlock-pthread-adaptive-test task_log
Logs:
Contents from WT_TEST/test_layered47.test_layered47.test_checkpoint_order_mismatch_palite/stderr.txt:
- is related to
-
WT-15520 Dump the error log after a failed test/suite test
-
- Closed
-