Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-57267

Dereference of invalid iterator in connection pool

    • Fully Compatible
    • ALL
    • v5.0
    • Hide

      I found this by inspection + looking at the core dump generated by the failure of jstests/sharding/query/merge_write_concern.js in the ASAN build variant of this patch build: https://spruce.mongodb.com/version/6080a2f7a4cf47408799d65c/tasks which was done in the linked server ticket as a test of our ability to upgrade abseil. See the linked SERVER ticket for more details.

      Show
      I found this by inspection + looking at the core dump generated by the failure of jstests/sharding/query/merge_write_concern.js in the ASAN build variant of this patch build: https://spruce.mongodb.com/version/6080a2f7a4cf47408799d65c/tasks which was done in the linked server ticket as a test of our ability to upgrade abseil. See the linked SERVER ticket for more details.
    • Service Arch 2021-06-14

      In ConnectionPool::dropConnections here, we used a range-based for loop to iterate through the ConnectionPool's _pools member, a. stdx::unordered_map. Within the loop, we call ConnectionPool::SpecificPool::triggerShutdown on each specific pool we find in the _pools map. This member function will erase this specific pool from the parent ConnectionPool's _pools member here.

      According to the std::unordered_map documentation for the "erase" overload set, "References and iterators to the erased elements are invalidated. Other iterators and references are not invalidated." While the abseil documentation is a bit unclear, I think the same applies for the absl::node_hash_map we're actually using (and, in any case, we should probably live by the standard library documentation for stdx:: types). This means that once the specific pool element is erased in triggerShutdown, any iterators to it are invalidated – which implies when that when the ranged-based for loop attempts to advance to the next element in the map by calling next() on the iterator, it will be using/dereferencing an invalid iterator.

      Our current version of abseil seems to leave this iterator in a valid state long enough for the for-loop to advance, but this is not guaranteed: when we attempted an upgrade to the newest version of abseil (see SERVER-51476), in at least one test we see that we fail an internal assert inside abseil when attempting to advance the iterator in this range-based for loop, suggesting that the iterator the loop is attempting to advance is invalid.

            Assignee:
            george.wangensteen@mongodb.com George Wangensteen
            Reporter:
            george.wangensteen@mongodb.com George Wangensteen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: