[SERVER-8779] 2.2.3, start without --sslPEMKeyPassword but with --sslPEMKeyFile, process zombies, shell hangs, uninterruptible Created: 28/Feb/13  Updated: 08/Mar/13  Resolved: 04/Mar/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Gregor Macadam Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Operating System: ALL
Participants:

 Description   

Repro'd this on two different ubuntu aws instances

ubuntu@ip-10-48-159-19:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ ./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem 
Thu Feb 28 13:16:10 need sslPEMKeyPassword
 
 
^C^C^C^Z^Z^H^H^H^C^C^C^C^Z^Z^Z^Z^Z^Z

ubuntu@ip-10-48-159-19:~$ ps -elf | grep mongo
0 Z ubuntu    5277  5150  0  80   0 -     0 exit   13:16 pts/0    00:00:00 [mongod] <defunct>
0 S ubuntu    5485  5386  0  80   0 -  2026 pipe_w 13:20 pts/1    00:00:00 grep --color=auto mongo



 Comments   
Comment by Gregor Macadam [ 04/Mar/13 ]

Based on the fact that GDB didn't work on ec2-xx-xxx-24-205 in SERVER-8734, and that was one of the machines that we could repro this on, we are putting this down to goosed AWS instances and closing this ticket.

Comment by Gregor Macadam [ 28/Feb/13 ]

This trace shows - current zombie processes.
I create a new clean dir, run strace.
I was able to CTRL-C the strace at the end - but that makes sense right since it was strace I was terminating.
Then I run mongod without strace and it zombies.

ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ ps -elf | grep mongo
0 Z ubuntu    4833     1  0  80   0 -     0 exit   12:42 ?        00:00:00 [mongod] <defunct>
0 Z ubuntu    5213     1  0  80   0 -     0 exit   12:45 ?        00:00:00 [mongod] <defunct>
0 Z ubuntu    5804     1  0  80   0 -     0 exit   13:52 ?        00:00:00 [mongod] <defunct>
0 Z ubuntu    5988     1  0  80   0 -     0 exit   13:53 ?        00:00:00 [mongod] <defunct>
0 S ubuntu    6491  6383  0  80   0 -  2026 pipe_w 15:16 pts/4    00:00:00 grep --color=auto mongo
ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ mkdir data2
ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ strace ./bin/mongod --dbpath ./data2/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem
execve("./bin/mongod", ["./bin/mongod", "--dbpath", "./data2/", "--sslOnNormalPorts", "--sslPEMKeyFile", "../gregorFreeBSD.pem"], [/* 18 vars */]) = 0
brk(0)                                  = 0x21c7000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d33b000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=17252, ...}) = 0
mmap(NULL, 17252, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fce4d336000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=135366, ...}) = 0
mmap(NULL, 2212904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4cefe000
mprotect(0x7fce4cf16000, 2093056, PROT_NONE) = 0
mmap(0x7fce4d115000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fce4d115000
mmap(0x7fce4d117000, 13352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4d117000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libssl.so.0.9.8", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p/\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=338048, ...}) = 0
mmap(NULL, 2433304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4ccab000
mprotect(0x7fce4ccf7000, 2097152, PROT_NONE) = 0
mmap(0x7fce4cef7000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4c000) = 0x7fce4cef7000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcrypto.so.0.9.8", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300]\6\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1612544, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d335000
mmap(NULL, 3723160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4c91e000
mprotect(0x7fce4ca83000, 2093056, PROT_NONE) = 0
mmap(0x7fce4cc82000, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x164000) = 0x7fce4cc82000
mmap(0x7fce4cca8000, 12184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4cca8000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31752, ...}) = 0
mmap(NULL, 2128984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4c716000
mprotect(0x7fce4c71d000, 2093056, PROT_NONE) = 0
mmap(0x7fce4c91c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fce4c91c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libnetsnmpagent.so.15", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\265\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=284008, ...}) = 0
mmap(NULL, 2383264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4c4d0000
mprotect(0x7fce4c513000, 2097152, PROT_NONE) = 0
mmap(0x7fce4c713000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x43000) = 0x7fce4c713000
mmap(0x7fce4c715000, 3488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4c715000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libnetsnmphelpers.so.15", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pq\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=146216, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d334000
mmap(NULL, 2242464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4c2ac000
mprotect(0x7fce4c2cf000, 2093056, PROT_NONE) = 0
mmap(0x7fce4c4ce000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fce4c4ce000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libnetsnmpmibs.so.15", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340B\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1171120, ...}) = 0
mmap(NULL, 3398304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4bf6e000
mprotect(0x7fce4c083000, 2093056, PROT_NONE) = 0
mmap(0x7fce4c282000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x114000) = 0x7fce4c282000
mmap(0x7fce4c28c000, 129696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4c28c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libnetsnmp.so.15", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\27\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=620224, ...}) = 0
mmap(NULL, 2929152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4bca2000
mprotect(0x7fce4bd37000, 2093056, PROT_NONE) = 0
mmap(0x7fce4bf36000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x94000) = 0x7fce4bf36000
mmap(0x7fce4bf39000, 213504, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4bf39000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\241\5\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=962656, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d333000
mmap(NULL, 3142544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4b9a2000
mprotect(0x7fce4ba84000, 2093056, PROT_NONE) = 0
mmap(0x7fce4bc83000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe1000) = 0x7fce4bc83000
mmap(0x7fce4bc8d000, 82832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4bc8d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pU\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1030512, ...}) = 0
mmap(NULL, 3125544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4b6a6000
mprotect(0x7fce4b7a1000, 2093056, PROT_NONE) = 0
mmap(0x7fce4b9a0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xfa000) = 0x7fce4b9a0000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320(\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=88384, ...}) = 0
mmap(NULL, 2184216, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4b490000
mprotect(0x7fce4b4a5000, 2093056, PROT_NONE) = 0
mmap(0x7fce4b6a4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7fce4b6a4000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1811128, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d332000
mmap(NULL, 3925208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4b0d1000
mprotect(0x7fce4b286000, 2093056, PROT_NONE) = 0
mmap(0x7fce4b485000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b4000) = 0x7fce4b485000
mmap(0x7fce4b48b000, 17624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4b48b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14768, ...}) = 0
mmap(NULL, 2109704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4aecd000
mprotect(0x7fce4aecf000, 2097152, PROT_NONE) = 0
mmap(0x7fce4b0cf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fce4b0cf000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=92720, ...}) = 0
mmap(NULL, 2187824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4acb6000
mprotect(0x7fce4accc000, 2093056, PROT_NONE) = 0
mmap(0x7fce4aecb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7fce4aecb000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libwrap.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20,\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=36432, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d331000
mmap(NULL, 2133888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4aaad000
mprotect(0x7fce4aab5000, 2093056, PROT_NONE) = 0
mmap(0x7fce4acb4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fce4acb4000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libperl.so.5.14", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\r\3\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1554200, ...}) = 0
mmap(NULL, 3649832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4a731000
mprotect(0x7fce4a8a4000, 2093056, PROT_NONE) = 0
mmap(0x7fce4aaa3000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x172000) = 0x7fce4aaa3000
mmap(0x7fce4aaac000, 296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4aaac000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libsensors.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0%\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=59704, ...}) = 0
mmap(NULL, 2155528, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4a522000
mprotect(0x7fce4a530000, 2093056, PROT_NONE) = 0
mmap(0x7fce4a72f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x7fce4a72f000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\367\5\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1852792, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d330000
mmap(NULL, 3963320, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce4a15a000
mprotect(0x7fce4a2f9000, 2093056, PROT_NONE) = 0
mmap(0x7fce4a4f8000, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19e000) = 0x7fce4a4f8000
mmap(0x7fce4a51e000, 14776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4a51e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnsl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`@\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=97248, ...}) = 0
mmap(NULL, 2202328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce49f40000
mprotect(0x7fce49f57000, 2093056, PROT_NONE) = 0
mmap(0x7fce4a156000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7fce4a156000
mmap(0x7fce4a158000, 6872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce4a158000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\n\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=43288, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d32f000
mmap(NULL, 2327040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fce49d07000
mprotect(0x7fce49d10000, 2097152, PROT_NONE) = 0
mmap(0x7fce49f10000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7fce49f10000
mmap(0x7fce49f12000, 184832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fce49f12000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d32e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d32d000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d32b000
arch_prctl(ARCH_SET_FS, 0x7fce4d32bb40) = 0
mprotect(0x7fce4b485000, 16384, PROT_READ) = 0
mprotect(0x7fce49f10000, 4096, PROT_READ) = 0
mprotect(0x7fce4a156000, 4096, PROT_READ) = 0
mprotect(0x7fce4b0cf000, 4096, PROT_READ) = 0
mprotect(0x7fce4aecb000, 4096, PROT_READ) = 0
mprotect(0x7fce4a4f8000, 110592, PROT_READ) = 0
mprotect(0x7fce4b9a0000, 4096, PROT_READ) = 0
mprotect(0x7fce4a72f000, 4096, PROT_READ) = 0
mprotect(0x7fce4d115000, 4096, PROT_READ) = 0
mprotect(0x7fce4aaa3000, 16384, PROT_READ) = 0
mprotect(0x7fce4acb4000, 4096, PROT_READ) = 0
mprotect(0x7fce4b6a4000, 4096, PROT_READ) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d32a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d329000
mprotect(0x7fce4bc83000, 32768, PROT_READ) = 0
mprotect(0x7fce4bf36000, 4096, PROT_READ) = 0
mprotect(0x7fce4c713000, 4096, PROT_READ) = 0
mprotect(0x7fce4c4ce000, 4096, PROT_READ) = 0
mprotect(0x7fce4c282000, 8192, PROT_READ) = 0
mprotect(0x7fce4c91c000, 4096, PROT_READ) = 0
mprotect(0x7fce4cc82000, 53248, PROT_READ) = 0
mprotect(0x7fce4cef7000, 4096, PROT_READ) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d327000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d323000
mprotect(0xf20000, 143360, PROT_READ)   = 0
mprotect(0x7fce4d33d000, 4096, PROT_READ) = 0
munmap(0x7fce4d336000, 17252)           = 0
set_tid_address(0x7fce4d32be10)         = 6497
set_robust_list(0x7fce4d32be20, 0x18)   = 0
futex(0x7fff2d89ccec, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fce4d32bb40) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7fce4cf04750, [], SA_RESTORER|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fce4cf047e0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
futex(0x7fce4bc8ded0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x132a520, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d303000
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d2e3000
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d2c3000
mmap(NULL, 540672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d23f000
mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d237000
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d217000
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce49c06000
munmap(0x7fce49d06000, 4096)            = 0
mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d20f000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d20b000
uname({sys="Linux", node="ip-10-227-167-84", ...}) = 0
futex(0x132a300, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fff2d89cc3c, FUTEX_WAKE, 1)    = 0
futex(0x7fff2d89cc3c, FUTEX_WAKE_PRIVATE, 1) = 0
open("/sys/devices/system/cpu/cpu0/tsc_freq_khz", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/cpuinfo", O_RDONLY)         = 3
read(3, "processor\t: 0\nvendor_id\t: Genuin"..., 1023) = 649
read(3, "", 1023)                       = 0
close(3)                                = 0
gettimeofday({1362064649, 588723}, NULL) = 0
futex(0x13221f4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap(NULL, 20983808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce48803000
munmap(0x7fce48803000, 4096)            = 0
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce4d337000
open("/dev/urandom", O_RDONLY)          = 3
mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce48403000
munmap(0x7fce48403000, 4096)            = 0
gettimeofday({1362064649, 623172}, NULL) = 0
read(3, "TFi\212\332\21\300\360b\330\303\2Z\376\214\311\363\244\224bw\204\321yZ\2$\356\253}:\245"..., 8191) = 8191
gettimeofday({1362064649, 624813}, NULL) = 0
rt_sigaction(SIGUSR1, {0x673730, [USR1], SA_RESTORER|SA_RESTART, 0x7fce4b1074a0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, {0x6736a0, [HUP], SA_RESTORER|SA_RESTART, 0x7fce4b1074a0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSEGV, {0x580a10, [], SA_RESTORER|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigaction(SIGBUS, {0x580a10, [], SA_RESTORER|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigaction(SIGILL, {0x580a10, [], SA_RESTORER|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigaction(SIGFPE, {0x580a10, [], SA_RESTORER|SA_SIGINFO, 0x7fce4cf0dcb0}, NULL, 8) = 0
rt_sigaction(SIGABRT, {0x5804a0, [ABRT], SA_RESTORER|SA_RESTART, 0x7fce4b1074a0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {0x5804a0, [QUIT], SA_RESTORER|SA_RESTART, 0x7fce4b1074a0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7fce4b1074a0}, {SIG_DFL, [], 0}, 8) = 0
gettimeofday({1362064649, 627258}, NULL) = 0
getcwd("/home/ubuntu/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3", 1000) = 64
rt_sigprocmask(SIG_SETMASK, [INT TERM], NULL, 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fce47c03000
mprotect(0x7fce47c03000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fce48402bb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fce484039d0, tls=0x7fce48403700, child_tidptr=0x7fce484039d0) = 6498
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce47c02000
read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 118
lseek(4, -62, SEEK_CUR)                 = 56
read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 62
close(4)                                = 0
munmap(0x7fce47c02000, 4096)            = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 4), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fce47c02000
write(1, "Thu Feb 28 15:17:29 need sslPEMK"..., 43Thu Feb 28 15:17:29 need sslPEMKeyPassword
) = 43
exit_group(2)                           = ?
^Cubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ 
ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ 
ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$ 
ubuntu@ip-10-227-167-84:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.2.3$  ./bin/mongod --dbpath ./data2/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem
Thu Feb 28 15:17:43 need sslPEMKeyPassword
^C^C

Comment by Gregor Macadam [ 28/Feb/13 ]

BTW this did repro on two machines (not going crazy!)

EDIT: should mention could not get it to repro on another ~6 AWS instances checked.

Comment by Gregor Macadam [ 28/Feb/13 ]

Yes that's what I mean - status is Z, thread has not been reaped by kernel and in this case shell has also hung. Please stand by

Comment by Eric Milkie [ 28/Feb/13 ]

By "zombied", do you mean the ps display for the process shows Z in the S column?
Can you run mongod with strace? Prepend the command line with "strace ". (Might need to install it first)

Comment by Gregor Macadam [ 28/Feb/13 ]

Does not repro on 2.4.0-rc1. Can only get it to repro with 2.2.3 on one AWS instance.

Comment by Eric Milkie [ 28/Feb/13 ]

Does it reproduce with 2.4?

Generated at Thu Feb 08 03:18:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.