Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7852

Don't release eviction server lock when evicting pages

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.1, 4.4.8, 5.0.2, 5.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 1
    • Storage - Ra 2021-07-26

      As part of debugging WT-6729, I noticed behaviour where the eviction queue is full but workers aren't dequeuing pages and trying to evict them.

      We frequently see this in test/format when the workload finishes. We shut down all threads and execute rollback to stable. With the changes in WT-6729, we wait until eviction quiesces which it never does due to the aforementioned bug.

      This is caused by multiple eviction workers taking on the role of the eviction server. I believe the offending commit is here:

      Our assumption is that since we are the only worker thread in the system, we're free to release the pass lock since there's no thread to acquire it and become the server. However, the current thread count is not an accurate reflection of what is happening in the system.

      When you stop a thread in a thread group, we signal to that thread to stop by clearing WT_THREAD_ACTIVE. It does not stop immediately, it just stops after the next iteration of whatever it is doing. It could easily be at the beginning of the eviction logic in which case it will see that the pass lock is free and acquire it to become the server despite the current thread count being 1.

      If I make this change, I don't see the problem anymore. Obviously, this isn't the fix because we specifically released this to avoid a deadlock but it supports the theory that the releasing of the lock is getting us in trouble.

      @@ -743,9 +742,9 @@ __evict_pass(WT_SESSION_IMPL *session)
                    * is already another server is running.
                   FLD_CLR(session->lock_flags, WT_SESSION_LOCKED_PASS);
      -            __wt_spin_unlock(session, &cache->evict_pass_lock);
                   ret = __evict_lru_pages(session, true);
      -            __wt_spin_lock(session, &cache->evict_pass_lock);
                   FLD_SET(session->lock_flags, WT_SESSION_LOCKED_PASS);


      1. Checkout wt-6729-hs-stop-rts.
      2. Add an assert like so:
        -    if (retries == WT_RTS_EVICT_MAX_RETRIES)
        +    if (retries == WT_RTS_EVICT_MAX_RETRIES) {
                 WT_ERR(__wt_msg(session, "timed out waiting for eviction to quiesce, running rts"));
        +        WT_ASSERT(session, false);
        +    }
      3. Schedule an Evergreen patch and run format stress testing.


      Figure out a solution and fix the issue.

            alex.cameron@mongodb.com Alex Cameron (Inactive)
            alex.cameron@mongodb.com Alex Cameron (Inactive)
            0 Vote for this issue
            5 Start watching this issue