Consider using ThreadPool as the execution resource for ShardingTaskExecutor

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Networking & Observability
    • N&O Availability Backlog
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The ShardingTaskExecutors currently use a NetworkInterfaceThreadPool, which despite its name reuses the single networking reactor thread from the underlying NetworkInterface to run arbitrary tasks. The issue is that this thread is intended to be running an async event loop and non-blocking work, since anything blocking will prevent all other async networking from making progress. For most normal cases, this isn't an issue because the vast majority of work is done using the baton (i.e. on the client thread), but certain tasks that happen in the background (e.g. cursor cleanup) need to use the TaskExecutor's execution resources. We've seen cases where this cleanup work overwhelms the reactor and prevents forward progress, in turn causing more cleanup and eventually total unavailability.

      We should consider changing this execution resource to a ThreadPool so that background work can't interfere with networking.

            Assignee:
            Unassigned
            Reporter:
            Patrick Freed
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: