[SERVER-63824] Integrate Flow Control tickets into write concurrent queue Created: 18/Feb/22  Updated: 06/Dec/22  Resolved: 03/Oct/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Daniel Gomez Ferro Assignee: Backlog - Storage Execution Team
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-63822 Modify TicketHolder with a concurrent... Closed
depends on SERVER-63823 Replace write tickets with concurrent... Closed
Assigned Teams:
Storage Execution
Participants:

 Description   

We want a single entry point for all tickets needed by write operations. Flow control tickets and write tickets will be acquired at the same time, and operations that fail either of them will release all tickets and go into the write queue, waiting for enough resources.

Currently flow control ticket are taken here and write tickets are taken here.



 Comments   
Comment by Louis Williams [ 03/Mar/22 ]

daniel.gomezferro, we effectively already have tryLock/tryAcquire APIs for locking if you set the deadline to Date_t::now().

Your solution makes sense to me. I have one question/possible concern. In the case of flow control, tickets are refreshed on a periodic schedule. This means that if many operations are waiting, when tickets are refreshed, suddenly every operation in the queue could become ready at once. But in your solution, there is a "chain reaction" that requires each operation to signal the next in line. In practice, this has the effect of serializing waiting operations instead of scheduling them all at once. This could turn out to not be a problem, but in the current flow control implementation, many threads can wait for a flow control ticket and they all get admitted at once. Your algorithm would be a change in behavior. It could in fact be more reasonable because we don't storm the storage engine with new writers. But something to consider.

Comment by Daniel Gomez Ferro [ 03/Mar/22 ]

The tryLock()/tryAcquire() API would only be needed as an optimization, so we could start with:

  1. Each operation that comes in gets assigned and put into a queue
    • If it ends up in the head, it blocks trying to get all of the locks/resources
    • If not, it waits until it gets woken up
  2. When the head of the queue eventually takes all locks/resources, it pops itself out of the queue, wakes up the new head and continues execution
  3. The new head takes all locks/resources in the usual order, blocking (and goes back to step 2)
Comment by Daniel Gomez Ferro [ 02/Mar/22 ]

I’ve been thinking about how to integrate the flow control tickets into the queue, and I believe something like this could be a solution:

  1. Each operation that comes in gets assigned a queue (only if we have more than one queue, e.g. read/write)
  2. If there are operations in the queue, it goes directly into the queue.
  3. If the queue is empty, the operation tries to obtain locks/resources in the usual order (flowControl, PBWM…)
    • If it gets all of them it continues execution (uncontended case)
    • If any of those fails, it goes in the queue
      • If it ends up in the head, it blocks trying to get the rest of the locks/resources
      • If not, it releases all its resources
  4. When the head of the queue eventually takes all locks/resources, it pops itself out of the queue, wakes up the new head and continues execution
  5. The new head takes all locks/resources in the usual order, blocking (and goes back to step 4)

The “only” problem is that we’d need a tryLock()/tryAcquire() API for each resource. The advantage is that we don't change the current lock ordering.

louis.williams do you think this could work?

Generated at Thu Feb 08 05:58:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.