ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Prefetch
    • Storage Engines - Persistence
    • 130.134
    • SE Persistence backlog
    • None

      Bug

      __wt_conn_prefetch_queue_push (src/conn/conn_prefetch.c) has two paths into its err label, and the label unconditionally executes WT_REF_SET_STATE(ref, WT_REF_DISK) at line 289. Only one of the two paths has locked the ref.

      • Line 247, WT_ERR(EBUSY) when S2BT(session)->evict_disabled > 0. This is before the DISK-to-LOCKED CAS at line 264.
      • Line 267, WT_ERR(__wt_calloc_one(session, &pe)). This is after the CAS, so the thread does hold the ref locked and the unlock at 289 is correct.

      The other two exits are already handled correctly: the WT_REF_FLAG_PREFETCH check and the failed CAS both goto done, which only drops the prefetch spinlock.

      Consequence

      On the line 247 path, the blind release store occurs without holding the WT_REF lock, which could overwrite another thread's change to the ref state. The session's hazard pointer is on the parent ref, so the WT_REF memory stays valid, but nothing prevents another thread from transitioning the child in the meantime.

      Worked interleaving, using __wt_delete_page (src/btree/bt_delete.c) as the competing thread. T1 is a prefetch-enabled tree walk, T2 a fast truncate on the same ref. T2 never takes the prefetch spinlock, so it provides no mutual exclusion here.

      Step T1 - prefetch T2 - fast truncate ref state
      1 __wti_btree_prefetch reads WT_REF_GET_STATE(next_ref) == WT_REF_DISK unlocked (bt_prefetch.c), calls the queue push   DISK
      2 takes conn->prefetch.lock (line 245)   DISK
      3   __wt_delete_page reads previous_state == WT_REF_DISK (bt_delete.c:126) DISK
      4   CAS WT_REF_DISK to WT_REF_LOCKED succeeds (bt_delete.c:128) LOCKED
      5 sees evict_disabled > 0, WT_ERR(EBUSY) (line 247) - note T1 has not CASed anything   LOCKED
      6   installs ref->page_del, publishes WT_REF_DELETED (bt_delete.c:233) DELETED
      7 err: WT_REF_SET_STATE(ref, WT_REF_DISK) (line 289)   DISK
      8 drops the prefetch spinlock, returns EBUSY   DISK

      After step 7 the truncate has completed and ref->page_del is installed, but the ref reads WT_REF_DISK. A later __wt_page_in_func takes the on-disk path rather than the WT_REF_DELETED case, so it reads the page in and never instantiates the deletion - the truncate is silently dropped for that page, and the ref is left in a state combination (DISK with page_del set) the rest of the tree does not expect.

      Other interleavings can lead to similar outcomes. This is a lost update on the state field, not a lifetime problem. When the ref is still DISK at step 7, the store is a harmless no-op, which is the common case.

      How it got here

      WT-15803 (453f50bf75, "Fix ref not unlocked in error cases") added the two lines at 288-289. That fixed the line 267 path, which genuinely leaked a locked ref, but the label was already shared with the line 247 path.

      The WT_ERR at line 247 dates from WT-12580 (aab2bdf0fd), which converted a WT_RET(EBUSY) into WT_ERR(EBUSY) so that pe would be freed on that exit. pe is no longer allocated before the check - it is calloc'd at line 267 - and neither label frees it today, so that reason no longer applies.

      Fix

      Send the eviction-disabled path to done rather than err:

           __wt_spin_lock(session, &conn->prefetch.lock);
           /* Don't queue pages for trees that have eviction disabled. */
      -    if (S2BT(session)->evict_disabled > 0)
      -        WT_ERR(EBUSY);
      +    if (S2BT(session)->evict_disabled > 0) {
      +        ret = EBUSY;
      +        goto done;
      +    }
      

      That leaves err with a single predecessor, the post-CAS allocation failure, where the unlock is unconditionally correct because the CAS established that the previous state was WT_REF_DISK. It also makes all three "this thread owns nothing" exits consistently jump to done.

      Testing

      WT_TIMING_STRESS_PREFETCH_3 already exists at line 254, immediately before the CAS, to widen this window. A test that runs a prefetch-enabled tree walk against concurrent page-in or fast-truncate on the same refs, with evict_disabled raised, should show the clobber under REF_TRACK or TSan; the ref history from HAVE_REF_TRACK will name __wt_conn_prefetch_queue_push as the writer of the unexpected WT_REF_DISK.

            Assignee:
            Etienne Petrel
            Reporter:
            Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: