[SERVER-45066] Crashes in AuthorizationManagerImpl when running binaries built with VS2019 Created: 11/Dec/19  Updated: 29/Oct/21  Resolved: 24/Jan/20

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Andrew Morrow (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-43721 Make AuthorizationManager use DistCache Closed
Related
is related to SERVER-45341 InvalidatingLRUCache should remove ac... Closed
is related to SERVER-43721 Make AuthorizationManager use DistCache Closed
Operating System: ALL
Steps To Reproduce:
  • Spawn a windows-64-vs2019-nonprod instance, or just install VS 2019 locally if you have a windows machine.
  • Build master from source, but add MSVC_VERSION=14.2 on the SCons invocation line. I recommend doing a --dbg=on build to get abseil assertions enabled.
  • Run the auth suite.
Sprint: Security 2019-12-16, Dev Platform 2019-12-30, Dev Platform 2020-02-10
Participants:

 Description   

We have recently been doing experiments with building with VS2019, because it comes with a much faster linker. It can save us close to an hour of build time, the majority of which comes from faster linking of unit tests.

However, we are encountering an interesting crash inside the AuthorizationManagerImpl class.

Here are links to two recent instances out of a patch build:

The bottom of the crash always looks like:

[js_test:auth1] 2019-12-10T21:47:03.474+0000 d21770| Assertion failed: it != end(), file C:\data\mci\df6020d8f98282b32443f3e0ae575514\src\src\third_party\abseil-cpp-master\abseil-cpp\absl/container/internal/raw_hash_set.h, line 1269
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| 2019-12-10T21:47:03.687+0000 F  -        [conn1] Got signal: 22 (SIGABRT).
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| mongod.exe     ...\src\mongo\util\stacktrace_windows.cpp(353)                                                                      mongo::printStackTrace+0x4f
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| mongod.exe     ...\src\mongo\util\signal_handlers_synchronous.cpp(242)                                                             mongo::`anonymous namespace'::abruptQuit+0xc3
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| ucrtbased.dll                                                                                                                      raise+0x441
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| ucrtbased.dll                                                                                                                      abort+0x39
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| ucrtbased.dll                                                                                                                      get_wide_winmain_command_line+0x211f
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| ucrtbased.dll                                                                                                                      get_wide_winmain_command_line+0x118
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| ucrtbased.dll                                                                                                                      wassert+0x2f
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| mongod.exe     ...\src\third_party\abseil-cpp-master\abseil-cpp\absl\container\internal\raw_hash_set.h(1270)                       absl::container_internal::raw_hash_set<absl::container_internal::NodeHashMapPolicy<mongo::UserName,std::weak_ptr<mongo::User> >,absl::hash_internal::Hash<mongo::UserName>,std::equal_to<mongo::UserName>,std::allocator<std::pair<mongo::UserName const ,std::weak_ptr<mongo::User> > > >::erase+0x3e
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| mongod.exe     ...\src\mongo\util\invalidating_lru_cache.h(277)                                                                    mongo::InvalidatingLRUCache<mongo::UserName,mongo::User,mongo::AuthorizationManagerImpl::UserCacheInvalidator>::_invalidateKey+0x228
[js_test:auth1] 2019-12-10T21:47:03.687+0000 d21770| mongod.exe     ...\src\mongo\db\auth\authorization_manager_impl.cpp(746)                                                           mongo::AuthorizationManagerImpl::invalidateUserByName+0x194

So this might really be an issue with the InvalidingLRUCache, or with abseil, or it could be a VS2019 issue.

CC shreyas.kalyan and jbreams.



 Comments   
Comment by Andrew Morrow (Inactive) [ 15/Jan/20 ]

MS released VS 2019 16.4.3, but it did not fix this issue.

Comment by Andrew Morrow (Inactive) [ 18/Dec/19 ]

I have reported this issue upstream.

Comment by Shreyas Kalyan [ 12/Dec/19 ]

In our InvalidatingLRUCache class, we are passing calling a function called _invalidateActiveIterator and passing in, among other things, an object of type ActiveIterator. ActiveIterator is an iterator of type ActiveMap::iterator, where ActiveMap is an abseil class absl::node_hash_map. At the bottom of the function, we are calling active_map::erase(iterator++) on the active map function. When calling this function, somehow the iterator increments and we end up passing the value ActiveMap::end as the parameter to the erase function. This seems to be a bug in MSVC 2019.

The relevant code for our InvalidatingLRUCache can be found here:
(ActiveMap and ActiveIterator definitions): https://github.com/mongodb/mongo/blob/a02960827dd9d145292eb32d9745cdd52001ebda/src/mongo/util/invalidating_lru_cache.h#L262-L263
(_invalidateActiveIterator function): https://github.com/mongodb/mongo/blob/a02960827dd9d145292eb32d9745cdd52001ebda/src/mongo/util/invalidating_lru_cache.h#L281-L317

The relevant code in Abseil's classes (node_hash_map relies on base class raw_hash_map that relies on raw_hash_set) can be found here:
(iterator operator++(int)): https://github.com/mongodb/mongo/blob/a02960827dd9d145292eb32d9745cdd52001ebda/src/third_party/abseil-cpp-master/abseil-cpp/absl/container/internal/raw_hash_set.h#L757-L761
(iterator erase(iterator it)): https://github.com/mongodb/mongo/blob/a02960827dd9d145292eb32d9745cdd52001ebda/src/third_party/abseil-cpp-master/abseil-cpp/absl/container/internal/raw_hash_set.h#L1266-L1272

When removing the iterator from the erase function, the tests that were previously failing are passing.
https://evergreen.mongodb.com/version/5df25e71d6d80a467ee1618e

Generated at Thu Feb 08 05:07:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.