[SERVER-10108] smoke ThreadedTests::MongoMutexTest failing intermittently on Nightly Linux 32-bit Created: 05/Jul/13  Updated: 10/Dec/14  Resolved: 11/Jul/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Randolph Tan
Resolution: Duplicate Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

buildbot: Nightly Linux 32-bit


Issue Links:
Duplicate
duplicates SERVER-6020 Exception in MongoMutexTest Closed
Operating System: ALL
Participants:

 Description   

Failed on June 30 and July 4, builds #1383 and #1388

Jul 05 00:00	5f949c19a260...	success	#1389	Build successful
Jul 04 04:50	5f949c19a260...	failure	#1388	Failed test_1
Jul 03 00:00	5741e9a35cfd...	success	#1387	Build successful
Jul 02 07:34	718012dbfc53...	success	#1386	Build successful
Jul 01 19:06	90f1d8947a26...	success	#1385	Build successful
Jul 01 00:00	85ca38c33e4a...	failure	#1384	Failed test_9
Jun 30 02:27	85ca38c33e4a...	failure	#1383	Failed test_1

http://buildlogs.mongodb.org/Nightly%20Linux%2032-bit/builds/1383/test/core/test?mode=raw

Sun Jun 30 03:29:36.054 [testsuite] 	 going to run test: ThreadedTests::List1Test
Sun Jun 30 03:29:36.054 [testsuite] warning: DBException thrown :: caused by :: 14050 List1: item to orphan not in list
0x94f436d 0x94af6bb 0x94afbca 0x8c3fe82 0x8c400ed 0x848b81b 0x94abc54 0x94acc02 0x93126e9 0x85a682b 0x85a695c 0x587390 0x848b461 
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo15printStackTraceERSo+0x2d) [0x94f436d]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo11DBException13traceIfNeededERKS0_+0x12b) [0x94af6bb]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo9uassertedEiPKc+0xba) [0x94afbca]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN13ThreadedTests9List1Test3runEv+0x382) [0x8c3fe82]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite13runTestObjectIN13ThreadedTests9List1TestEEEvv+0x1d) [0x8c400ed]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5boost6detail8function22void_function_invoker0IPFvvEvE6invokeERNS1_15function_bufferE+0xb) [0x848b81b]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite3runERKSsi+0x3c4) [0x94abc54]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite3runERKSt6vectorISsSaISsEERKSsi+0x4c2) [0x94acc02]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo7dbtests10runDbTestsEiPPcSs+0x1b39) [0x93126e9]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_Z11dbtestsMainiPPcS0_+0xcb) [0x85a682b]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(main+0x3c) [0x85a695c]
 /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xe0) [0x587390]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test [0x848b461]

http://buildlogs.mongodb.org/Nightly%20Linux%2032-bit/builds/1388/test/core/test

Thu Jul  4 06:11:15.236 [testsuite] 	 going to run test: ThreadedTests::List1Test
Thu Jul  4 06:11:15.236 [testsuite] warning: DBException thrown :: caused by :: 14050 List1: item to orphan not in list
0x9513f1d 0x94cf26b 0x94cf77a 0x8c4d842 0x8c4daad 0x848f74b 0x94cb804 0x94cc7b2 0x9330be9 0x85aab30 0x85aac6c 0x913390 0x848f391 
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo15printStackTraceERSo+0x2d) [0x9513f1d]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo11DBException13traceIfNeededERKS0_+0x12b) [0x94cf26b]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo9uassertedEiPKc+0xba) [0x94cf77a]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN13ThreadedTests9List1Test3runEv+0x382) [0x8c4d842]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite13runTestObjectIN13ThreadedTests9List1TestEEEvv+0x1d) [0x8c4daad]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5boost6detail8function22void_function_invoker0IPFvvEvE6invokeERNS1_15function_bufferE+0xb) [0x848f74b]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite3runERKSsi+0x3c4) [0x94cb804]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo8unittest5Suite3runERKSt6vectorISsSaISsEERKSsi+0x4c2) [0x94cc7b2]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_ZN5mongo7dbtests10runDbTestsEiPPcSs+0x1b39) [0x9330be9]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(_Z11dbtestsMainiPPcS0_+0x1a0) [0x85aab30]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test(main+0x3c) [0x85aac6c]
 /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xe0) [0x913390]
 /mnt/slaves/Linux_32bit_Nightly/mongo/test [0x848f391]

Successive builds are clear. Huh?



 Comments   
Comment by Matt Kangas [ 11/Jul/13 ]

Thank you for clarifying this redbeard0531. I'm getting better at reading these error outputs...

Comment by Matt Kangas [ 11/Jul/13 ]

Ok, that makes sense. If it's failing because it can't create threads, sounds like a builder-specific issue. Very annoying and distracting but not worth blocking a release over.

Comment by Mathias Stearn [ 11/Jul/13 ]

List1Test isn't failing, we are testing that we get an error (with stacktrace) if we do something illeagal. The real failure is in MongoMutexTest which is failing with thread_resource_error which usually indicates a failure to create a new thread.

Sun Jun 30 03:29:39.240 [testsuite] FAIL: ThreadedTests::MongoMutexTest	 std::exception: boost::thread_resource_error in test ThreadedTests::MongoMutexTest

Comment by Matt Kangas [ 08/Jul/13 ]

Happend again in #1392.

http://buildbot.mongodb.org/builders/Nightly%20Linux%2032-bit/builds/1392/steps/test_1/logs/stdio
http://buildlogs.mongodb.org/Nightly%20Linux%2032-bit/builds/1392/test/core/test

This time I see, preceding the 14050 exception

Mon Jul  8 00:53:25.687 [testsuite] 	 going to run test: SingleTagSet
Mon Jul  8 00:53:25.687 [testsuite]   Assertion failure !_isExhausted src/mongo/client/dbclient_rs.cpp 2025
0x95364ed 0x950bff5 0x94f200e 0x8d6b421 0x8baa2de 0x94ebf3d 0x8bd2309 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
Mon Jul  8 00:53:25.712 [testsuite] 	 going to run test: MultiTagSet
Mon Jul  8 00:53:25.712 [testsuite]   Assertion failure !_isExhausted src/mongo/client/dbclient_rs.cpp 2025
0x95364ed 0x950bff5 0x94f200e 0x8d6b421 0x8ba8afb 0x94ebf3d 0x8bd1189 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
Mon Jul  8 00:53:25.712 [testsuite] 	 going to run test: MultiTagSet
Mon Jul  8 00:53:25.712 [testsuite]   Assertion failure !_isExhausted src/mongo/client/dbclient_rs.cpp 2025
0x95364ed 0x950bff5 0x94f200e 0x8d6b421 0x8ba8afb 0x94ebf3d 0x8bd1189 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 
Mon Jul  8 00:54:40.845 [testsuite] 	 going to run test: CursorTests::ClientCursor::Pin::PinCursor
Mon Jul  8 00:54:40.845 [testsuite] Assertion: 12521:internal error: use of an unlocked ClientCursor
0x95364ed 0x950bff5 0x94f2d50 0x85824a2 0x8582b49 0x85932d6 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 /mnt/slaves/Linux_32bit_Nightly/mongo/test [0x8492511]
Mon Jul  8 00:54:40.853 [testsuite] Assertion: 16089:Cannot kill active cursor 717808979
0x95364ed 0x950bff5 0x94f2d50 0x94f3306 0x8e77ed4 0x8e7800e 0x8582c7b 0x85932d6 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 
Mon Jul  8 00:58:01.697 [testsuite] going to run suite: gridfs
Mon Jul  8 00:58:01.697 [testsuite] 	 going to run test: (anonymous namespace)::SetChunkSizeTest
Mon Jul  8 00:58:01.914 [testsuite] Assertion: 13296:invalid chunk size is specified
0x95364ed 0x950bff5 0x94f2d50 0x8dc19ab 0x86f3195 0x86f3586 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 
Mon Jul  8 00:58:02.327 [testsuite] 	 going to run test: IndexUpdateTests::IndexBuildInProgressTest
Mon Jul  8 00:58:02.328 [testsuite] Assertion: 16574:could not find index being built
0x95364ed 0x950bff5 0x94f2d50 0x9173dbe 0x86ff3d3 0x870cfbd 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 
Mon Jul  8 00:58:03.640 [testsuite] 	 going to run test: JSTests::InvalidStoredJS
Mon Jul  8 00:58:03.710 [testsuite] ReferenceError: invalidstoredjs1 is not defined
Mon Jul  8 00:58:03.738 [testsuite] SyntaxError: Illegal continue statement
Mon Jul  8 00:58:03.738 [testsuite] unable to load stored JavaScript function invalidstoredjs1(): SyntaxError: Illegal continue statement
 
Mon Jul  8 00:58:04.251 [testsuite] going to run suite: jsobj
Mon Jul  8 00:58:04.251 [testsuite] 	 going to run test: JsobjTests::BufBuilderBasic
Mon Jul  8 00:58:04.252 [testsuite] 	 going to run test: JsobjTests::BufBuilderReallocLimit
Mon Jul  8 00:58:04.468 [testsuite] Assertion: 13548:BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.
0x95364ed 0x950bff5 0x94f2d50 0x849c574 0x873505f 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
 
Mon Jul  8 00:58:06.534 [testsuite] 	 going to run test: JsobjTests::ValueStreamTests::Unallowed
Mon Jul  8 00:58:06.534 [testsuite] Assertion: 10336:No subobject started
0x95364ed 0x950bff5 0x94f2d50 0x8739ce5 0x873a04d 0x84928cb 0x94eddd4 0x94eed82 0x9353159 0x85adcb0 0x85addec 0x144390 0x8492511 
...

And the number of assertions printed here KEEPS GOING.

Generated at Thu Feb 08 03:22:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.