[SERVER-57267] Dereference of invalid iterator in connection pool Created: 27/May/21  Updated: 29/Oct/23  Resolved: 04/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc3, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: George Wangensteen Assignee: George Wangensteen
Resolution: Fixed Votes: 0
Labels: servicearch-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-51476 Upgrade Abseil to 20210324.1 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Steps To Reproduce:

I found this by inspection + looking at the core dump generated by the failure of jstests/sharding/query/merge_write_concern.js in the ASAN build variant of this patch build: https://spruce.mongodb.com/version/6080a2f7a4cf47408799d65c/tasks which was done in the linked server ticket as a test of our ability to upgrade abseil. See the linked SERVER ticket for more details.

Sprint: Service Arch 2021-06-14
Participants:

 Description   

In ConnectionPool::dropConnections here, we used a range-based for loop to iterate through the ConnectionPool's _pools member, a. stdx::unordered_map. Within the loop, we call ConnectionPool::SpecificPool::triggerShutdown on each specific pool we find in the _pools map. This member function will erase this specific pool from the parent ConnectionPool's _pools member here.

According to the std::unordered_map documentation for the "erase" overload set, "References and iterators to the erased elements are invalidated. Other iterators and references are not invalidated." While the abseil documentation is a bit unclear, I think the same applies for the absl::node_hash_map we're actually using (and, in any case, we should probably live by the standard library documentation for stdx:: types). This means that once the specific pool element is erased in triggerShutdown, any iterators to it are invalidated – which implies when that when the ranged-based for loop attempts to advance to the next element in the map by calling next() on the iterator, it will be using/dereferencing an invalid iterator.

Our current version of abseil seems to leave this iterator in a valid state long enough for the for-loop to advance, but this is not guaranteed: when we attempted an upgrade to the newest version of abseil (see SERVER-51476), in at least one test we see that we fail an internal assert inside abseil when attempting to advance the iterator in this range-based for loop, suggesting that the iterator the loop is attempting to advance is invalid.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 17/Jun/21 ]

Author:

{'name': 'George Wangensteen', 'email': 'george.wangensteen@mongodb.com', 'username': 'gewa24'}

Message: SERVER-57267 Fix invalid iterator use in ConnectionPool::dropConnections

(cherry picked from commit 6598a5843ed7318527c8b60bdaaacfe312bd5c51)
Branch: v5.0
https://github.com/mongodb/mongo/commit/6e32e86f4f6340a0697706b9db7fb1d83d03b3c3

Comment by Githook User [ 04/Jun/21 ]

Author:

{'name': 'George Wangensteen', 'email': 'george.wangensteen@mongodb.com', 'username': 'gewa24'}

Message: SERVER-57267 Fix invalid iterator use in ConnectionPool::dropConnections
Branch: master
https://github.com/mongodb/mongo/commit/6598a5843ed7318527c8b60bdaaacfe312bd5c51

Generated at Thu Feb 08 05:41:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.