test/format disagg.multi: leader and follower database hashes diverge on a delete-heavy workload

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      testutil_assert(hash_match) at format_disagg.c:196 - both nodes hashed their databases successfully and disagree, so the leader and follower are holding different data.

      Not WT-18173: its fix 8f86d55d70 is already in 88c35695, and that repro had deletes disabled where this run is delete-heavy (table3 at 79%, stress.failpoint_hs_delete_key_from_ts=1). Likely a different divergence landing on the same check.

      disagg.preserve=0 here — turning it on would give something concrete to diff if it reproduces.

       

      format-stress-test-disagg-multi-2 on ubuntu2004-arm64-nonstandalone

      Host: i-0b442fe4cb2e27523
      Project: wiredtiger
      Commit: 88c35695
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-multi-2 task_log

      Logs:

      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x0000ffffa7fe7aac in __GI_abort () at abort.c:79
      #2  0x0000ffffa84e4718 in __wt_abort (session=session@entry=0x0) at /data/mci/cf3da69b4ba37faf4c4d5f934a13a6d1/wiredtiger/src/os_common/os_abort.c:32
      #3  0x000000000042929c in testutil_die (e=e@entry=0, fmt=fmt@entry=0x42b1a8 "%s/%d: %s") at /data/mci/cf3da69b4ba37faf4c4d5f934a13a6d1/wiredtiger/test/utility/misc.c:75
      #4  0x000000000040e474 in disagg_sync_multi_node (session=0x303bffc96140) at /data/mci/cf3da69b4ba37faf4c4d5f934a13a6d1/wiredtiger/test/format/format_disagg.c:196
      #5  0x00000000004197f8 in operations (ops_seconds=<optimized out>, ops_seconds@entry=7195, run_current=run_current@entry=2, run_total=run_total@entry=3) at /data/mci/cf3da69b4ba37faf4c4d5f934a13a6d1/wiredtiger/test/format/ops.c:581
      #6  0x000000000041ed88 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/cf3da69b4ba37faf4c4d5f934a13a6d1/wiredtiger/test/format/t.c:404
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: