Currently we lock the ref before we acquire the prefetch lock when adding a ref to the prefetch queue. This can cause the ref lock be held longer than necessary and increase the latency for users. We should do the opposite and only lock the ref after we have acquired the prefetch lock.