(disagg.mode=switch) Failed to step-up as primary: WT_ROLLBACK

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Test Format

      format-stress-test-disagg-switch-5 on amazon2023-disagg-stress

      Host: i-0f8569b423a628578
      Project: wiredtiger
      Commit: e08d19db
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-5 task_log

      Logs:

      [1758596495:011368][2753845:0xffff88121040], t, layered:T00001, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00001.wt_stable stable table verification failed. : No such file or directory
[1758596495:011425][2753845:0xffff88121040], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00002.wt_stable stable table verification failed. : No such file or directory
[1758596495:057240][2753845:0xffff88121040], t, layered:T00003, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00003.wt_stable stable table verification failed. : No such file or directory
[1758596508:446638][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_DEFAULT][ERROR]: __layered_drain_ingest_tables, 1753: Failed to clear ingest table "file:T00001.wt_ingest": WT_ROLLBACK: conflict between concurrent operations
[1758596508:446672][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_DEFAULT][ERROR]: __wti_disagg_conn_config, 1122: Failed to drain ingest tables: WT_ROLLBACK: conflict between concurrent operations
[1758596508:446682][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446684][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446686][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446687][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446689][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446690][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446691][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446693][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446694][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446696][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446697][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446698][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446700][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446701][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446702][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446706][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446707][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446709][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446710][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446711][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446713][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446714][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446716][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446717][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446718][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446722][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446723][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446725][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446726][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446727][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446729][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446730][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758596508:446732][2753845:0xffff88121040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at src/evict/evict_lru.c:2925: "__evict_get_ref(session

      logs

      format-stress-test-disagg-switch-5 task_log

      Logs:

      #0  0x0000ffff868bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff868bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff868763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff86862264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff86c1e7ac [PAC] in __wt_abort (session=session@entry=0x121dbfc7a000) at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff86cbff70 in __wt_panic_func (session=session@entry=0x121dbfc7a000, error=error@entry=-31800, func=func@entry=0xffff86da7648 <__PRETTY_FUNCTION__.35> "__wti_disagg_conn_config", line=line@entry=1227, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff86d3ae38 "failed to step-up as primary") at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/src/support/err.c:611
#5  0x0000ffff86b50f50 in __wti_disagg_conn_config (session=session@entry=0x121dbfc7a000, cfg=cfg@entry=0xffffc9a36ab8, reconfig=reconfig@entry=true) at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/src/conn/conn_layered.c:1227
#6  0x0000ffff86b58b94 in __wti_conn_reconfig (session=session@entry=0x121dbfc7a000, cfg=cfg@entry=0xffffc9a36ab8) at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/src/conn/conn_reconfig.c:450
#7  0x0000ffff86b36eac in __conn_reconfigure (wt_conn=<optimized out>, config=<optimized out>) at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/src/conn/conn_api.c:1354
#8  0x000000000040b9f0 in disagg_switch_roles () at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/test/format/format_disagg.c:54
#9  0x000000000041a8e0 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/dbc0b90605a6fbaca581ee5415c1ef9b/wiredtiger/test/format/t.c:398

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: