failed: spinlock-pthread-adaptive-test on rhel80 [wiredtiger @ e143bc93] [WiredTiger database is already being managed by another thread in this process: Resource busy]

XMLWordPrintableJSON

    spinlock-pthread-adaptive-test on rhel80

    Host: i-02e5a8b5d21ed527e
    Project: wiredtiger
    Commit: e143bc93
    Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

    Task Logs:

    spinlock-pthread-adaptive-test task_log

    Logs:

    [pid:10010]: test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palite): 0.15 seconds
    [pid:10010]: ERROR in test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palite)
    [pid:10009/10010]: test_layered47.test_layered47.test_checkpoint_order_mismatch(palite) ... [pid:10010]: test_layered48.test_layered48.test_layered48 -s 1 (palite.shared): starting
    

    logs

    spinlock-pthread-adaptive-test task_log

    Logs:

    [pid:10009]: ERROR: test_layered27.test_layered27.test_drain_insert_update(palite.large)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
        self.verifyLayered()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
        if self.conn is None or self.conn.this is None:
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
        raise Exception('The connection is closed')
    Exception: The connection is closed
    ======================================================================
    [pid:10009]: ERROR: test_layered27.test_layered27.test_drain_remove_insert(palite.large)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered27.py", line 187, in test_drain_remove_insert
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config + 'disaggregated=(role="follower")')
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered30.test_layered30.test_layered30(palite.layered-prefix.one-table)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered30.py", line 109, in test_layered30
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' + self.conn_config)
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered31.test_layered31.test_layered31(palite)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered31.py", line 136, in test_layered31
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' + self.conn_base_config + 'disaggregated=(role="follower")')
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered38.test_layered38.test_gc_ingest_table_with_remove(palite)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered38.py", line 144, in test_gc_ingest_table_with_remove
        oplog, t, conn_follow, session_follow = self.setup()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered38.py", line 91, in setup
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config + 'disaggregated=(role="follower")')
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered44.test_layered44.test_layered44
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered44.py", line 111, in test_layered44
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered45.test_layered45.test_prepare_update(palite)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
        self.verifyLayered()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
        if self.conn is None or self.conn.this is None:
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
        raise Exception('The connection is closed')
    Exception: The connection is closed
    ======================================================================
    [pid:10009]: ERROR: test_layered47.test_layered47.test_first_gc_with_cursor_on_previous_checkpoint(palite)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 156, in test_first_gc_with_cursor_on_previous_checkpoint
        self.setup([uri])
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 62, in setup
        self.create_follower()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered47.py", line 82, in create_follower
        self.conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered49.test_layered49.test_remove(palite)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered49.py", line 74, in test_remove
        self.create_follower()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered49.py", line 66, in create_follower
        self.conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + ',create,' +
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 421, in wiredtiger_open
        conn = wiredtiger.wiredtiger_open(home, config)
    _wiredtiger.WiredTigerError: Device or resource busy
    ======================================================================
    [pid:10009]: ERROR: test_layered27.test_layered27.test_drain_insert_update(palite.small)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 626, in tearDown
        self.verifyLayered()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 585, in verifyLayered
        if self.conn is None or self.conn.this is None:
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 106, in __getattr__
        raise Exception('The connection is closed')
    Exception: The connection is closed
    ======================================================================
    [pid:10009]: ERROR: test_layered27.test_layered27.test_drain_remove_insert(palite.small)
    ----------------------------------------------------------------------
    [pid:10009]: testtools.testresult.real._StringException: Traceback (most recent call last):
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/wttest.py", line 290, in _callTestMethod
        method()
      File "/data/mci/9a82349020bef8ad826d64891af0729a/wiredtiger/test/suite/test_layered27.py", line 187, in test_drain_remove_insert
        conn_follow = self.wiredtiger_open('follower', self.extensionsConfig() + self.conn_base_config
    

    logs

    spinlock-pthread-adaptive-test task_log

    Logs:

    [1761157251:021610][10013:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy
    [1761157251:021898][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021910][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021916][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021920][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021924][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021929][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761157251:021933][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy
    [1761157251:021938][10013:0x7f7c1818f740], test_layered50.test_layered50.test_evict_on_standby(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy
    [1761156951:463779][10011:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy
    [1761156951:464141][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464157][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464166][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464173][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464180][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464187][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
    [1761156951:464194][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy
    [1761156951:464202][10011:0x7f7c1818f740], test_layered49.test_layered49.test_remove(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy
    [1761157037:469443][10010:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy
    [1761157037:469767][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
    [1761157037:469783][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
    [1761157037:469789][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
    [1761157037:469794][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
    [1761157037:469798][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
    [1761157037:469803][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
    [1761157037:469807][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found
    [1761157037:469812][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found
    [1761157037:469816][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 315: Error at src/cursor/cur_file.c:315: "ret" failed: WT_NOTFOUND: item not found
    [1761157037:469822][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_remove, 281: Error at src/meta/meta_table.c:281: "cursor->search(cursor)" failed: WT_NOTFOUND: item not found
    [1761157037:469827][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_single, 1859: Error at src/conn/conn_api.c:1859: "EBUSY" failed: Device or resource busy
    [1761157037:469833][10010:0x7f7c1818f740], test_layered60.test_layered60.test_layered60(palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: wiredtiger_open, 3075: Error at src/conn/conn_api.c:3075: "__conn_single(session, cfg)" failed: Device or resource busy
    [1761158291:880015][10011:0x7f7c1818f740], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __conn_single, 1859: WiredTiger database is already being managed by another thread in this process: Device or resource busy
    [1761158291:880368][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get_from_config, 911: Error at src/meta/meta_ckpt.c:911: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1761158291:880384][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get, 859: Error at src/meta/meta_ckpt.c:859: "__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)" failed: WT_NOTFOUND: item not found
    [1761158291:880394][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get_from_config, 911: Error at src/meta/meta_ckpt.c:911: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1761158291:880401][10011:0x7f7c1818f740], test_verify_disagg.test_verify_disagg.test_verify_disagg(empty.palite), WT_CONNECTION.dump_error_log: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_meta_ckptlist_get, 859: Error at src/meta/meta_ckpt.c:859: "__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)" failed: WT_NOTFOUND: item not found
    [1761158291:880408][100
    

    logs

    spinlock-pthread-adaptive-test task_log

    Logs:

    Contents from WT_TEST/test_layered47.test_layered47.test_checkpoint_order_mismatch_palite/stderr.txt:
    
    

    logs

          Assignee:
          [DO NOT USE] Backlog - Storage Engines Team
          Reporter:
          xgen-buildbaron-user
          Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

            Created:
            Updated:
            Resolved: