(disagg.mode=switch) failed to step-up as primary: WT_ROLLBACK concurrent conflict

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Not Applicable

      format-stress-test-disagg-switch-2 on amazon2023-disagg-stress

      Host: i-07e2af3690bc7bc44
      Project: wiredtiger
      Commit: 220bd4f3
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-2 task_log

      Logs:

      [1758249043:107628][99544:0xffff9dc9d040], t, layered:T00001, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00001.wt_stable stable table verification failed. : No such file or directory
[1758249043:109306][99544:0xffff9dc9d040], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00002.wt_stable stable table verification failed. : No such file or directory
[1758249043:113189][99544:0xffff9dc9d040], t, layered:T00003, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 115: Verify (layered): file:T00003.wt_stable stable table verification failed. : No such file or directory
[1758249056:947473][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_DEFAULT][ERROR]: __layered_drain_ingest_tables, 1753: Failed to clear ingest table "file:T00001.wt_ingest": WT_ROLLBACK: conflict between concurrent operations
[1758249056:947503][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_DEFAULT][ERROR]: __wti_disagg_conn_config, 1122: Failed to drain ingest tables: WT_ROLLBACK: conflict between concurrent operations
[1758249056:947511][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947513][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947515][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947516][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947517][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947519][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947520][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947521][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947523][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947524][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947526][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947527][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947528][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947531][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947532][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947534][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947535][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947536][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947538][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947542][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947544][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947545][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947546][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947548][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947549][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947550][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947552][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/evict/evict_lru.c:2925: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1758249056:947553][99544:0xffff9dc9d040], t, WT_CONNECTION.reconfigure: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2925: Error at /data

      logs

      format-stress-test-disagg-switch-2 task_log

      Logs:

      #0  0x0000ffff9c2bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff9c2bf9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff9c2763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff9c262264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff9c61fe6c [PAC] in __wt_abort (session=session@entry=0x30bb3fc7a000) at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/os_common/os_abort.c:31
#4  0x0000ffff9c6c20bc in __wt_panic_func (session=session@entry=0x30bb3fc7a000, error=error@entry=-31800, func=func@entry=0xffff9c7a9d80 <__PRETTY_FUNCTION__.35> "__wti_disagg_conn_config", line=line@entry=1227, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff9c73d6c0 "failed to step-up as primary") at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/support/err.c:611
#5  0x0000ffff9c551ac4 in __wti_disagg_conn_config (session=session@entry=0x30bb3fc7a000, cfg=cfg@entry=0xffffcc222298, reconfig=reconfig@entry=true) at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/conn/conn_layered.c:1227
#6  0x0000ffff9c55970c in __wti_conn_reconfig (session=session@entry=0x30bb3fc7a000, cfg=cfg@entry=0xffffcc222298) at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/conn/conn_reconfig.c:450
#7  0x0000ffff9c5378e4 in __conn_reconfigure (wt_conn=<optimized out>, config=<optimized out>) at /data/mci/2163277753be0296564efeb529785745/wiredtiger/src/conn/conn_api.c:1354
#8  0x000000000040b9e8 in disagg_switch_roles () at /data/mci/2163277753be0296564efeb529785745/wiredtiger/test/format/format_disagg.c:54
#9  0x000000000041a8cc in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/2163277753be0296564efeb529785745/wiredtiger/test/format/t.c:398

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: