[SERVER-10506] authorization_manager_test.cpp without gperftools hangs on linux-64 due to memory corruption Created: 13/Aug/13  Updated: 11/Jul/16  Resolved: 20/Aug/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 2.5.2

Type: Bug Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Run authorization_manager_test without tcmalloc enabled on a modern linux-64 machine.

Participants:

 Description   

http://mci.10gen.com/ui/task/mongodb_mongo_master_cxx11_ubuntu1204_64_d4f670c95a3cc81f4ea297ff0cf809568261dcaf_13_08_13_13_53_06_compile
http://mci.10gen.com/ui/task/mongodb_mongo_master_cxx11_ubuntu1204_64_a58ae77d0e5035cb4b541751dbe042e9173deea5_13_08_13_15_38_07_compile
http://mci.10gen.com/ui/task/mongodb_mongo_master_cxx11_ubuntu1204_64_5c03be3aca41801c807ed49666020c4a8088549d_13_08_13_16_39_07_compile

The compile task stops every time here:

[2013/08/13 18:43:36.681] Tue Aug 13 18:43:36.675 going to run test: VerifyRolesFieldMustBeAnArray



 Comments   
Comment by auto [ 14/Aug/13 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-9518 SERVER-10506 Make sure internal user is never deleted
Branch: master
https://github.com/mongodb/mongo/commit/9b98517497f44c49bc6374ba1d7b5a1480721d1c

Comment by Andrew Morrow (Inactive) [ 14/Aug/13 ]

Note that commenting out the 'initializeAllV1UserData' fixes the test.

Comment by Andrew Morrow (Inactive) [ 14/Aug/13 ]

I can repro the hang. I get some additional info in my console:

 /home/andrew/Documents/10gen/dev/src/mongodb/build/linux2/allocator_system/c++11/release/mongo/db/auth/authorization_manager_test
Tue Aug 13 20:52:54.171 going to run suite: AuthorizationManagerTest
Tue Aug 13 20:52:54.171 	 going to run test: GetPrivilegesFromPrivilegeDocumentCompatible
Tue Aug 13 20:52:54.171 	 going to run test: GetPrivilegesFromPrivilegeDocumentInvalid
Tue Aug 13 20:52:54.171 	 going to run test: DocumentValidationCompatibility
Tue Aug 13 20:52:54.171 	 going to run test: DisableCompatibilityMode
Tue Aug 13 20:52:54.171 	 going to run test: DocumentValidationExtended
Tue Aug 13 20:52:54.171 	 going to run test: testAquireV0User
Tue Aug 13 20:52:54.171 	 going to run test: testAquireV1User
Tue Aug 13 20:52:54.171 	 going to run test: initializeAllV1UserData
Tue Aug 13 20:52:54.171 	 DONE running tests
Tue Aug 13 20:52:54.171 going to run suite: PrivilegeDocumentParsing
Tue Aug 13 20:52:54.171 	 going to run test: VerifyRolesFieldMustBeAnArray
*** Error in `/home/andrew/Documents/10gen/dev/src/mongodb/build/linux2/allocator_system/c++11/release/mongo/db/auth/authorization_manager_test': corrupted double-linked list: 0x0000000000be08f0 ***

A little googling reveals that this is how glibc reports memory corruption: http://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c;h=be472b2ba3885b194f55c29474c4ad5d6a61b729;hb=HEAD#l1428

Don't know why that seems to hang, but maybe that is what malloc does on internal errors?

We notice this in the C++11 builds because they don't use TCMalloc (yet). Not sure why TCMalloc isn't complaning (do we not use the debug tcmalloc in debug build?)

So, anyway running under valgrind gives lots of these:

==23926== Invalid read of size 4
==23926==    at 0x4CDF20: mongo::User::getRefCount() const (user.cpp:51)
==23926==    by 0x472D7F: mongo::(anonymous namespace)::AuthorizationManagerTest::tearDown() (authorization_manager_test.cpp:49)
==23926==    by 0x4735B2: void mongo::unittest::Suite::runTestObject<mongo::(anonymous namespace)::UnitTest__PrivilegeDocumentParsing__VerifyRolesFieldMustBeAnArray>() (unittest.h:296)
==23926==    by 0x502966: mongo::unittest::Suite::run(std::string const&, int) (function_template.hpp:760)
==23926==    by 0x503DFB: mongo::unittest::Suite::run(std::vector<std::string, std::allocator<std::string> > const&, std::string const&, int) (unittest.cpp:228)
==23926==    by 0x471E46: main (unittest_main.cpp:28)
==23926==  Address 0x5d4b2f0 is 160 bytes inside a block of size 168 free'd
==23926==    at 0x4C2B59C: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23926==    by 0x4B6AD8: mongo::AuthorizationManager::~AuthorizationManager() (authorization_manager.cpp:287)
==23926==    by 0x472E3B: mongo::(anonymous namespace)::AuthorizationManagerTest::~AuthorizationManagerTest() (checked_delete.hpp:39)
==23926==    by 0x47318B: void mongo::unittest::Suite::runTestObject<mongo::(anonymous namespace)::UnitTest__AuthorizationManagerTest__initializeAllV1UserData>() (authorization_manager_test.cpp:608)
==23926==    by 0x502966: mongo::unittest::Suite::run(std::string const&, int) (function_template.hpp:760)
==23926==    by 0x503DFB: mongo::unittest::Suite::run(std::vector<std::string, std::allocator<std::string> > const&, std::string const&, int) (unittest.cpp:228)
==23926==    by 0x471E46: main (unittest_main.cpp:28)
==23926==

So this is most likely some sort of memory corruption or use after free problem. Given that Andy is away on vacation, I'd recommend having Spencer take a look.

Comment by Ian Whalen (Inactive) [ 13/Aug/13 ]

acm might have some useful insight here as to why this is happening.

Generated at Thu Feb 08 03:23:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.