I observed a "cache stuck" situation in a very simple scenario where a single application thread updates records in a single table in the database.

The user thread was getting stuck in __wt_evict_app_assist_worker_check and __wti_evict_app_assist_worker functions, attempting to perform eviction. Meanwhile, eviction workers were in the same situation. The eviction server, however, was active, filling the eviction queue and sleeping intermittently. On the timeline, this is visible as all threads - service threads and the user thread - sleeping simultaneously.

Looking into this, siddhartha.mahajan@mongodb.com mentioned he had encountered an issue where the eviction server would fill the queue with pages, thinking it was making good progress, but in reality, none of those pages could be evicted by workers. Here's the mechanism causing this issue:

1. The eviction server calls __evict_try_queue_page() on an internal page.
2. That function calls __wt_evict_page_urgent() here or here before performing sufficient checks to ensure that the page can actually be evicted (e.g. __wt_page_can_evict().
3. A successful enqueuing of the page is reported to the eviction server.
4. After enqueuing the required number of pages, the eviction server considers its job done and goes to sleep.
5. Eviction workers and application threads are unable to evict any pages.
6. The cycle repeats.

Timeline:

(Click on the attachment to zoom in)