Fix intermittent hang in test_schema_disagg_abort due to rwlock writer starvation

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Engines - Foundations
    • 178.102
    • None
    • None

      Problem

      test_schema_disagg_abort hangs intermittently on longer runs.

      Root cause

      The leader child runs 12 schema-worker threads and 1 checkpoint thread, coordinated by state->lock, a pthread_rwlock initialized with default attributes (workload.c:363):

      pthread_rwlock_init(&state.lock, NULL);   // NULL attr => glibc default = PREFER_READER
      

      The 12 schema workers hold the read lock in a tight loop (create/drop + publish, workload.c:181, 197). The single checkpoint thread needs the write lock (workload.c:270). glibc's default rwlock is reader-preferring: new readers cut ahead of a waiting writer, so with 12 continuous readers the write lock never becomes available.

      Chain to the observed symptom:

      1. Checkpoint thread starves in pthread_rwlock_wrlock – first checkpoint never completes.
      2. READY_FILE sentinel never written (workload.c:283).
      3. Parent busy-waits forever in while (!testutil_exists(...READY_FILE)) (main.c:119).
      4. Log stops right after the CONFIG: line. Hang.

      SIGCHLD does not rescue it – the child is alive (starving), not exiting.

      Amplifier

      Every layered create/drop calls __wt_meta_track_off(need_sync=true) – a synchronous metadata checkpoint + fdatasync – while holding the read lock. This stretches hold times so the lock is essentially always held once all 12 threads warm up, making starvation near-certain. This explains the intermittency: if the checkpoint grabs the write lock before the workers saturate it succeeds; otherwise it starves permanently.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: