Details
Bug
Major - P3
Description
eviction failing to switch queues leads to starvation
Issue Links
Activity
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2928 Don't give up when the urgent queue is half empty. (#3063)
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/5fb852354745b9efe638183084ee00db821aa132
Author:
{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}Message: WT-2928 Eviction failing to switch queues can lead to starvation (#3060)
There are some configurations where the eviction server waiting for a the current queue to drain before switching in the new queue can lead to starvation. Specifically if in-memory is configured, the cache is full and there is only a single eviction thread. Or if an operation that does not contribute to eviction fills the cache (for example compact).
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/7cc8f4bfbc5b45486e35072771c659e3c9e12ece
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2928 Don't give up when the urgent queue is half empty. (#3063)
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/5fb852354745b9efe638183084ee00db821aa132
Author:
{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}Message: Import wiredtiger: 9cf2f89d6d95e1de797f05ab1fef28695f8bae7b from branch mongodb-3.2
ref: bb18c43915..9cf2f89d6d
for: 3.2.10
WT-2864 Reconfiguring the checkpoint server can lead to hangs
WT-2874 Change test_compact01 to avoid eviction
WT-2918 The dist scripts create C files s_whitespace complains about
WT-2919 Don't mask error returns from style checking scripts
WT-2921 Reduce the WT_SESSION hazard_size when possible
WT-2923 heap-use-after-free on address in compaction
WT-2924 Ensure we are doing eviction when threads are waiting for it
WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag
WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock
WT-2928 Eviction failing to switch queues can lead to starvation
Branch: v3.2
https://github.com/mongodb/mongo/commit/79d9b3ab5ce20f51c272b4411202710a082d0317
Author:
{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}Message: Import wiredtiger: fc0e7abe82595e579573d42448632f7b36a2d154 from branch mongodb-3.4
ref: 5bc03723a7..fc0e7abe82
for: 3.3.15
WT-2864 Reconfiguring the checkpoint server can lead to hangs
WT-2874 Change test_compact01 to avoid eviction
WT-2918 The dist scripts create C files s_whitespace complains about
WT-2919 Don't mask error returns from style checking scripts
WT-2921 Reduce the WT_SESSION hazard_size when possible
WT-2923 heap-use-after-free on address in compaction
WT-2924 Ensure we are doing eviction when threads are waiting for it
WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag
WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock
WT-2928 Eviction failing to switch queues can lead to starvation
Branch: master
https://github.com/mongodb/mongo/commit/9dda827a3ae58beef36d53da1b55554cbd8744c4
Michael Cahill, I've been looking at Sue LoVerso's comment in
WT-2884, where she talks about test_inmem01 failures. Failures in test_inmem01 are pretty easy to reproduce for me, the column-store version of test_inmem01.test_inmem01.test_insert_over_delete_replace fails reliably in under 200 runs.First, I'm not seeing anything like the WT_ROLLBACK failures described in
WT-2884, so I'm opening a new ticket and I'll add a comment inWT-2884.What I'm seeing is the server never empties the current eviction queue and so never switches to the other queue.
It's easy to get into a state in test_inmem01 where all of the dirty leaf pages are in the "other" queue, and there are a couple of internal pages that can't be evicted in the "current" queue. If we don't switch queues, all we do is hammer on the "current" queue, adding the same useless pages to the queue over and over and failing to evict them each time, ignoring all of the useful pages in the "other" queue, and eventually eviction times out.
I suspect we'd be OK if there was even a single eviction worker (because it should empty the current queue and cause a switch), but I haven't actually tested that theory. If that's correct, this isn't a MongoDB problem as it always configures eviction workers.
I'm going to push a branch that fixes test_inmem01 to clarify what I'm seeing; those changes make test_inmem01 run reliably for me. Don't hesitate to just discard that branch if I'm going in the wrong direction, I still don't fully understand the queue switching algorithms.
cc: Alexander Gorrod