-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Prefetch
-
Storage Engines - Persistence
-
130.134
-
SE Persistence backlog
-
None
Bug
__wt_conn_prefetch_queue_push (src/conn/conn_prefetch.c) has two paths into its err label, and the label unconditionally executes WT_REF_SET_STATE(ref, WT_REF_DISK) at line 289. Only one of the two paths has locked the ref.
- Line 247, WT_ERR(EBUSY) when S2BT(session)->evict_disabled > 0. This is before the DISK-to-LOCKED CAS at line 264.
- Line 267, WT_ERR(__wt_calloc_one(session, &pe)). This is after the CAS, so the thread does hold the ref locked and the unlock at 289 is correct.
The other two exits are already handled correctly: the WT_REF_FLAG_PREFETCH check and the failed CAS both goto done, which only drops the prefetch spinlock.
Consequence
On the line 247 path, the blind release store occurs without holding the WT_REF lock, which could overwrite another thread's change to the ref state. The session's hazard pointer is on the parent ref, so the WT_REF memory stays valid, but nothing prevents another thread from transitioning the child in the meantime.
Worked interleaving, using __wt_delete_page (src/btree/bt_delete.c) as the competing thread. T1 is a prefetch-enabled tree walk, T2 a fast truncate on the same ref. T2 never takes the prefetch spinlock, so it provides no mutual exclusion here.
| Step | T1 - prefetch | T2 - fast truncate | ref state |
|---|---|---|---|
| 1 | __wti_btree_prefetch reads WT_REF_GET_STATE(next_ref) == WT_REF_DISK unlocked (bt_prefetch.c), calls the queue push | DISK | |
| 2 | takes conn->prefetch.lock (line 245) | DISK | |
| 3 | __wt_delete_page reads previous_state == WT_REF_DISK (bt_delete.c:126) | DISK | |
| 4 | CAS WT_REF_DISK to WT_REF_LOCKED succeeds (bt_delete.c:128) | LOCKED | |
| 5 | sees evict_disabled > 0, WT_ERR(EBUSY) (line 247) - note T1 has not CASed anything | LOCKED | |
| 6 | installs ref->page_del, publishes WT_REF_DELETED (bt_delete.c:233) | DELETED | |
| 7 | err: WT_REF_SET_STATE(ref, WT_REF_DISK) (line 289) | DISK | |
| 8 | drops the prefetch spinlock, returns EBUSY | DISK |
After step 7 the truncate has completed and ref->page_del is installed, but the ref reads WT_REF_DISK. A later __wt_page_in_func takes the on-disk path rather than the WT_REF_DELETED case, so it reads the page in and never instantiates the deletion - the truncate is silently dropped for that page, and the ref is left in a state combination (DISK with page_del set) the rest of the tree does not expect.
Other interleavings can lead to similar outcomes. This is a lost update on the state field, not a lifetime problem. When the ref is still DISK at step 7, the store is a harmless no-op, which is the common case.
How it got here
WT-15803 (453f50bf75, "Fix ref not unlocked in error cases") added the two lines at 288-289. That fixed the line 267 path, which genuinely leaked a locked ref, but the label was already shared with the line 247 path.
The WT_ERR at line 247 dates from WT-12580 (aab2bdf0fd), which converted a WT_RET(EBUSY) into WT_ERR(EBUSY) so that pe would be freed on that exit. pe is no longer allocated before the check - it is calloc'd at line 267 - and neither label frees it today, so that reason no longer applies.
Fix
Send the eviction-disabled path to done rather than err:
__wt_spin_lock(session, &conn->prefetch.lock);
/* Don't queue pages for trees that have eviction disabled. */
- if (S2BT(session)->evict_disabled > 0)
- WT_ERR(EBUSY);
+ if (S2BT(session)->evict_disabled > 0) {
+ ret = EBUSY;
+ goto done;
+ }
That leaves err with a single predecessor, the post-CAS allocation failure, where the unlock is unconditionally correct because the CAS established that the previous state was WT_REF_DISK. It also makes all three "this thread owns nothing" exits consistently jump to done.
Testing
WT_TIMING_STRESS_PREFETCH_3 already exists at line 254, immediately before the CAS, to widen this window. A test that runs a prefetch-enabled tree walk against concurrent page-in or fast-truncate on the same refs, with evict_disabled raised, should show the clobber under REF_TRACK or TSan; the ref history from HAVE_REF_TRACK will name __wt_conn_prefetch_queue_push as the writer of the unexpected WT_REF_DISK.