[CDRIVER-3525] Implement a spec-compliant connection pool Created: 06/Feb/20  Updated: 14/Dec/23  Resolved: 23/Mar/23

Status: Closed
Project: C Driver
Component/s: libmongoc, network
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Major - P3
Reporter: Kevin Albertson Assignee: Ezra Chung
Resolution: Won't Do Votes: 0
Labels: bgtrack, neweng, size-large
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CDRIVER-3214 Ignore ConnectionReadyEvent in CMAP p... Closed
is depended on by CDRIVER-3816 Avoiding connection storms Closed
is depended on by CXX-2133 Avoiding connection storms Closed
is depended on by CDRIVER-2871 Implement Connection Monitoring and P... Closed
is depended on by CDRIVER-4501 Add log messages to CMAP spec Closed
is depended on by CDRIVER-3868 Record command monitoring and CMAP ev... Closed
is depended on by CXX-2166 Record command monitoring and CMAP ev... Closed
is depended on by CDRIVER-3791 Ensure ConnectionCreatedEvents are em... Closed
is depended on by CDRIVER-4513 CMAP connection establishment failure... Closed
Detailed Project Statuses:

Engineer(s): Kevin Albertson, Andrew Witten (Intern)

2020-07-30: Amendment - Updating target date to 2020-09-04 (8 weeks)

  • Andrew has been focusing on completing a POC for this epic over the course of his internship, but the total effort will take approximately 2-4 weeks longer according to the team, updating the end date to reflect the long tail of this estimate for caution’s sake.
  • Andrew is on track to complete the POC by the end of his internship on August 7th, and the core C team will plan to pick up where he leaves off.
  • It has become clear that this effort’s priority has increased as it has been identified as a dependency of DRIVERS-781: Avoiding Connection Storms

2020-07-24: Maintaining target date of 2020-08-07 (4 weeks)

  • This is the intern project that Andrew Witten will be focusing on for the duration of the internship, ending August 7.
  • Work is off to a good start, on track to have POC done by end of internship.

2020-07-10: Setting initial target date of 2020-08-07 (4 weeks)

  • This is the intern project that Andrew Witten will be focusing on for the duration of the internship, ending August 7.
  • Planning to begin work at the start of the coming week.

 Description   
Epic Summary

Summary

libmongoc does not have a notion of a per-server connection pool, but rather a client pool. There may be several benefits to reworking a mongoc_client_t to utilize a connection pool, as described in the Connection Monitoring and Pooling spec.

  • wrapping drivers (C++ in particular) may benefit from a more idiomatic client model.
  • fair waiting for connections (libmongoc's client pool does not have fair waiting)
  • less total sockets created (different threads can utilize a shared pool of sockets, but clients from libmongoc's client pool own their sockets, which necessitates one socket per server per thread)
  • implementation of remaining CMAP monitoring events (e.g. PoolClearedEvent, which could be used by wrapping drivers to detect if a fork occurs)

This would be a large change, likely require refactoring large parts of the driver. And it would increase the maintenance burden. Scoping should involve investigating whether this really is a worthwhile change.

  • Should we test performance of libmongoc's client pool against a prototype mongoc_client_t with connection pool? This would further validate that such a large change is warranted. Since clients from libmongoc's client pool own their sockets, they do not need to lock a shared structure to obtain a socket. Additionally, clients can be popped/pushed back quickly, for the duration of a single operation (which I believe is what Swift does).
  • Do we only make a mongoc_client_t thread safe? Or a mongoc_database_t/mongoc_collection_t as well. 
  • Do we want to support a connection pool for single-threaded mode? CMAP does not require multiple threads. And there may be legitimate uses for a connection pool in a single-threaded environment. For example, it is currently not possible to have more than one socket open against a server on a single threaded mongoc_client_t. That makes it impossible to have more than one exhaust cursor open on a single server. If so, let's also amend the CMAP spec to note that CMAP events are excluded from connections used only for monitoring, but connections shared between monitoring and application use are not excluded.

Improvements to test infrastructure and coverage (CDRIVER-2733) should likely be a prerequisite, and would provide more confidence in such a large change.

Motivation

Cast of Characters

  • Product Owner: Rachelle Palmer
  • Project Lead: Kevin Albertson
  • Program Manager: Alexander Golin

Documentation

Scope Document
[Technical Design Document|]


Generated at Wed Feb 07 21:18:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.