test/format (disagg.mode=leader) OOO keys detected

XMLWordPrintableJSON

      format-stress-test-disagg-leader-2 on amazon2023-disagg-stress

      Host: i-048bb04ca10e4b34c
      Project: wiredtiger
      Commit: de1746fa
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-2 task_log

      Logs:

      [1763694500:114253][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_OUT_OF_ORDER][ERROR]: WT_CURSOR.prev out-of-order returns: returned key 0000055055.00/opqrstuvwxyzab then key 0000054083.00/opqrs
      [1763694506:003376][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003386][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003388][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003391][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003393][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003395][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003396][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003398][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003399][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003401][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003402][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003404][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003405][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003407][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003409][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003410][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003411][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003413][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003414][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003416][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003417][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003419][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003420][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003423][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003425][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003426][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003427][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003429][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003430][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003434][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003436][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003437][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003438][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near, 860: Error at src/cursor/cur_hs.c:860: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003440][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003442][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003443][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003444][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003446][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003447][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 732: Error at src/cursor/cur_hs.c:732: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1763694506:003449][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_prev, 898: Error at src/btree/bt_curprev.c:898: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1763694506:003450][10536:0xffff17fccbc0], t, file:T00001.wt_stable, WT_CURSOR.prev: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_prev, 246: Error at src/cursor/cur_file.c:246: "ret" failed: WT_NOTFOUND: item not found
      [1763694506:003452][10536:0xffff17fccbc0], t, file:T00001.wt_stable, 
      

      logs

      format-stress-test-disagg-leader-2 task_log

      Logs:

      #0  0x0000ffffa6cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffffa6cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffffa6c763a0 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffffa6c62264 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffffa702e254 [PAC] in __wt_abort (session=session@entry=0x708e7fc99de0) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffffa70d2ad4 in __wt_panic_func (session=session@entry=0x708e7fc99de0, error=error@entry=22, func=func@entry=0xffffa71b1e88 <__PRETTY_FUNCTION__.46> "__cursor_key_order_check_row", line=line@entry=639, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffa711fad0 "found key out-of-order returns") at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/support/err.c:611
      #5  0x0000ffffa6e78be4 in __cursor_key_order_check_row (session=0x708e7fc99de0, cbt=<optimized out>, next=<optimized out>) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curnext.c:639
      #6  0x0000ffffa6e80590 in __wti_cursor_key_order_check (session=session@entry=0x708e7fc99de0, cbt=cbt@entry=0x708e7adf8f00, next=next@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curnext.c:660
      #7  0x0000ffffa6e8ae44 in __wt_btcur_prev (cbt=cbt@entry=0x708e7adf8f00, truncating=truncating@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/btree/bt_curprev.c:950
      #8  0x0000ffffa6f87630 in __curfile_prev (cursor=0x708e7adf8f00) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_file.c:245
      #9  0x0000ffffa6fa3560 in __layered_prev_int (session=session@entry=0x708e7fc99de0, cursor=cursor@entry=0x708e70db2b40) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_layered.c:911
      #10 0x0000ffffa6fa3a0c in __layered_prev (cursor=0x708e70db2b40) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/src/cursor/cur_layered.c:943
      #11 0x0000000000412720 in read_op (cursor=cursor@entry=0x708e70db2b40, op=op@entry=PREV, exactp=exactp@entry=0x0) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/format_inline.h:48
      #12 0x0000000000413458 in nextprev (tinfo=tinfo@entry=0x708e7f3ffc00, next=next@entry=false) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:1753
      #13 0x00000000004150cc in table_op (tinfo=tinfo@entry=0x708e7f3ffc00, intxn=intxn@entry=true, iso_level=iso_level@entry=ISOLATION_SNAPSHOT, op=<optimized out>, op@entry=UPDATE) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:948
      #14 0x0000000000416688 in ops (arg=0x708e7f3ffc00) at /data/mci/d5ade7d273da2fa65074c7c6d578024d/wiredtiger/test/format/ops.c:1351
      #15 0x0000ffffa6cbdb78 in start_thread () from /lib64/libc.so.6
      #16 0x0000ffffa6d2acdc [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: