[SERVER-12253] Parent process is not closing after fork Created: 05/Jan/14  Updated: 10/Dec/14  Resolved: 19/Mar/14

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: 2.4.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Pawe? Smoli?ski Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Slackware64 current


Operating System: ALL
Participants:

 Description   

I'm using following small shell script to launch MongoDB:

#!/bin/sh
MONGO_DIR='/home/pawel/bin/mongo'
$MONGO_DIR/bin/mongod --bind_ip 127.0.0.1 --logpath $MONGO_DIR/log/mongo.log --fork --dbpath $MONGO_DIR/db/ --directoryperdb

When I'll launch this, MongoDB process is starting but main process is not closing, i.e. I've following in my console:

$ /home/pawel/bin/mongo/start_mongo.sh
about to fork child process, waiting until server is ready for connections.
forked process: 10044
all output going to: /home/pawel/bin/mongo/log/mongo.log

When checking processes list, I see 3 instances of MongoDB running:

$ ps ax | grep mongo
10041 pts/0 S+ 0:00 /bin/sh /home/pawel/bin/mongo/start_mongo.sh
10042 pts/0 S+ 0:00 /home/pawel/bin/mongo/bin/mongod --bind_ip 127.0.0.1 --logpath /home/pawel/bin/mongo/log/mongo.log --fork --dbpath /home/pawel/bin/mongo/db/ --directoryperdb
10043 ? Ss 0:00 /home/pawel/bin/mongo/bin/mongod --bind_ip 127.0.0.1 --logpath /home/pawel/bin/mongo/log/mongo.log --fork --dbpath /home/pawel/bin/mongo/db/ --directoryperdb
10044 ? Sl 0:02 /home/pawel/bin/mongo/bin/mongod --bind_ip 127.0.0.1 --logpath /home/pawel/bin/mongo/log/mongo.log --fork --dbpath /home/pawel/bin/mongo/db/ --directoryperdb

In log file I've only this:

Sun Jan 5 11:23:02.114 [initandlisten] MongoDB starting : pid=10044 port=27017 dbpath=/home/pawel/bin/mongo/db/ 64-bit host=metis
Sun Jan 5 11:23:02.114 [initandlisten] db version v2.4.8
Sun Jan 5 11:23:02.114 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14
Sun Jan 5 11:23:02.114 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Jan 5 11:23:02.114 [initandlisten] allocator: tcmalloc
Sun Jan 5 11:23:02.114 [initandlisten] options:

Unknown macro: { bind_ip}

Sun Jan 5 11:23:02.322 [initandlisten] journal dir=/home/pawel/bin/mongo/db/journal
Sun Jan 5 11:23:02.340 [initandlisten] recover : no journal files present, no recovery needed
Sun Jan 5 11:23:03.332 [websvr] admin web console waiting for connections on port 28017
Sun Jan 5 11:23:03.333 [initandlisten] waiting for connections on port 27017

When I'll hit Ctrl+C in console, parent process is closing (PID=10042 from the list above) and rest of 2 are still running (PID: 10043, 10044) and I can use MongoDB.
When I'm trying to send SIGTERM to first process (PID=10043) nothing happens, and when I'll send SIGTERM to PID=10044, (with kill 10044 shell command) both processes are closing and I've following in log file:

Sun Jan 5 11:37:05.346 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
Sun Jan 5 11:37:05.346 [signalProcessingThread] now exiting
Sun Jan 5 11:37:05.346 dbexit:
Sun Jan 5 11:37:05.346 [signalProcessingThread] shutdown: going to close listening sockets...
Sun Jan 5 11:37:05.346 [signalProcessingThread] closing listening socket: 9
Sun Jan 5 11:37:05.346 [signalProcessingThread] closing listening socket: 10
Sun Jan 5 11:37:05.346 [signalProcessingThread] closing listening socket: 11
Sun Jan 5 11:37:05.346 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
Sun Jan 5 11:37:05.346 [signalProcessingThread] shutdown: going to flush diaglog...
Sun Jan 5 11:37:05.346 [signalProcessingThread] shutdown: going to close sockets...
Sun Jan 5 11:37:05.346 [signalProcessingThread] shutdown: waiting for fs preallocator...
Sun Jan 5 11:37:05.346 [signalProcessingThread] shutdown: lock for final commit...
Sun Jan 5 11:37:05.347 [signalProcessingThread] shutdown: final commit...
Sun Jan 5 11:37:05.449 [signalProcessingThread] shutdown: closing all files...
Sun Jan 5 11:37:05.450 [signalProcessingThread] closeAllFiles() finished
Sun Jan 5 11:37:05.450 [signalProcessingThread] journalCleanup...
Sun Jan 5 11:37:05.450 [signalProcessingThread] removeJournalFiles
Sun Jan 5 11:37:05.594 [signalProcessingThread] shutdown: removing fs lock...
Sun Jan 5 11:37:05.594 dbexit: really exiting now

And now when I'll start MongoDB server again, the same happens



 Comments   
Comment by Joseph Lietz [ 16/Apr/14 ]

Follow-up to previous comment: it turns out that I might just be impatient; the parent process does exit on its own after, for me, about 40 seconds.

Comment by Joseph Lietz [ 16/Apr/14 ]

I'm experiencing this issue as well with 2.4.9. Some clues:

  • Can reproduce at will on linux. Behavior in OS X is normal.
  • The startup command I'm using is:

    mongod --fork --dbpath $SOME_PATH/data/db --logpath $SOME_PATH/log/mongod.log --pidfilepath $SOME_PATH/mongo.pid --config $SOME_PATH/config/mongodb.conf

  • After that command refuses to return and doing ctrl-c, two processes remain (parent and child). mongod --shutdown then correctly kills both processes.
  • MOST INTERESTING: Running the startup command again (using the same $SOME_PATH) results in normal behavior (parent process exits after fork).
Comment by Pawe? Smoli?ski [ 18/Mar/14 ]

Hi,
some time ago by a mistake I've deleted whole DB on my machine and when I've
created new one, everything is working fine now.


Best regards,
Paweł.
Customer Support (Tech)
OnePageCRM "Simplify Sales"
email: pawel@onepagecrm.com <pawel@onepagecrm.com>
web: http://www.onepagecrm.com

Comment by Stennie Steneker (Inactive) [ 18/Mar/14 ]

Hi Pawel,

Apologies on the delay in follow-up .. are you still experiencing this issue or were you able to find a solution?

Regards,
Stephen

Comment by Pawe? Smoli?ski [ 11/Jan/14 ]

I've launched MongoDB with GDB and after it launched, I've sent SIGSEGV signal to parent process. In result GDB printed backtrace for this process, which should help with identification of the place where program stopped

Sat Jan 11 18:08:17.565 Invalid access at address: 0x3e800004d3f from thread:

Sat Jan 11 18:08:17.565 Got signal: 11 (Segmentation fault).

Sat Jan 11 18:08:17.568 Backtrace:
0xde05e1 0x6d0559 0x6d0ae2 0x7ffff7bcb670 0x7ffff7bcb1dc 0x9eec01 0x6dd6b0 0x6df459 0x7ffff6df2d85 0x6cf1c9
/home/pawel/bin/mongo/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/home/pawel/bin/mongo/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
/home/pawel/bin/mongo/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x6d0ae2]
/lib64/libpthread.so.0(+0xf670) [0x7ffff7bcb670]
/lib64/libpthread.so.0(waitpid+0x1c) [0x7ffff7bcb1dc]
/home/pawel/bin/mongo/bin/mongod(_ZN5mongo27initializeServerGlobalStateEb+0x461) [0x9eec01]
/home/pawel/bin/mongo/bin/mongod() [0x6dd6b0]
/home/pawel/bin/mongo/bin/mongod(main+0x9) [0x6df459]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ffff6df2d85]
/home/pawel/bin/mongo/bin/mongod(__gxx_personality_v0+0x499) [0x6cf1c9]

Comment by Pawe? Smoli?ski [ 11/Jan/14 ]

... and if it'll help, content of some files from /proc/<PARENT PROCESS PID>/ directory
File stack:

[<ffffffff8105caca>] do_wait+0x1ea/0x240
[<ffffffff8105cf24>] SyS_wait4+0x64/0xe0
[<ffffffff81b6f786>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff

File maps:

00400000-014fc000 r-xp 00000000 08:02 3804598 /home/pawel/bin/mongo/bin/mongod
016fc000-01774000 rw-p 010fc000 08:02 3804598 /home/pawel/bin/mongo/bin/mongod
01774000-01b75000 rw-p 00000000 00:00 0
7f8e1bc6c000-7f8e1d872000 rw-p 00000000 00:00 0
7f8e1d872000-7f8e1da31000 r-xp 00000000 08:01 554946 /lib64/libc-2.17.so
7f8e1da31000-7f8e1dc30000 ---p 001bf000 08:01 554946 /lib64/libc-2.17.so
7f8e1dc30000-7f8e1dc34000 r--p 001be000 08:01 554946 /lib64/libc-2.17.so
7f8e1dc34000-7f8e1dc36000 rw-p 001c2000 08:01 554946 /lib64/libc-2.17.so
7f8e1dc36000-7f8e1dc3b000 rw-p 00000000 00:00 0
7f8e1dc3b000-7f8e1dc50000 r-xp 00000000 08:01 264217 /usr/lib64/libgcc_s.so.1
7f8e1dc50000-7f8e1de50000 ---p 00015000 08:01 264217 /usr/lib64/libgcc_s.so.1
7f8e1de50000-7f8e1de51000 rw-p 00015000 08:01 264217 /usr/lib64/libgcc_s.so.1
7f8e1de51000-7f8e1df52000 r-xp 00000000 08:01 554960 /lib64/libm-2.17.so
7f8e1df52000-7f8e1e151000 ---p 00101000 08:01 554960 /lib64/libm-2.17.so
7f8e1e151000-7f8e1e152000 r--p 00100000 08:01 554960 /lib64/libm-2.17.so
7f8e1e152000-7f8e1e153000 rw-p 00101000 08:01 554960 /lib64/libm-2.17.so
7f8e1e153000-7f8e1e237000 r-xp 00000000 08:01 281509 /usr/lib64/libstdc++.so.6.0.18
7f8e1e237000-7f8e1e436000 ---p 000e4000 08:01 281509 /usr/lib64/libstdc++.so.6.0.18
7f8e1e436000-7f8e1e43e000 r--p 000e3000 08:01 281509 /usr/lib64/libstdc++.so.6.0.18
7f8e1e43e000-7f8e1e440000 rw-p 000eb000 08:01 281509 /usr/lib64/libstdc++.so.6.0.18
7f8e1e440000-7f8e1e455000 rw-p 00000000 00:00 0
7f8e1e455000-7f8e1e45c000 r-xp 00000000 08:01 555123 /lib64/librt-2.17.so
7f8e1e45c000-7f8e1e65b000 ---p 00007000 08:01 555123 /lib64/librt-2.17.so
7f8e1e65b000-7f8e1e65c000 r--p 00006000 08:01 555123 /lib64/librt-2.17.so
7f8e1e65c000-7f8e1e65d000 rw-p 00007000 08:01 555123 /lib64/librt-2.17.so
7f8e1e65d000-7f8e1e674000 r-xp 00000000 08:01 555095 /lib64/libpthread-2.17.so
7f8e1e674000-7f8e1e874000 ---p 00017000 08:01 555095 /lib64/libpthread-2.17.so
7f8e1e874000-7f8e1e875000 r--p 00017000 08:01 555095 /lib64/libpthread-2.17.so
7f8e1e875000-7f8e1e876000 rw-p 00018000 08:01 555095 /lib64/libpthread-2.17.so
7f8e1e876000-7f8e1e87a000 rw-p 00000000 00:00 0
7f8e1e87a000-7f8e1e89d000 r-xp 00000000 08:01 556136 /lib64/ld-2.17.so
7f8e1e974000-7f8e1ea5c000 rw-p 00000000 00:00 0
7f8e1ea5c000-7f8e1ea9d000 rw-p 00000000 00:00 0
7f8e1ea9d000-7f8e1ea9e000 r--p 00023000 08:01 556136 /lib64/ld-2.17.so
7f8e1ea9e000-7f8e1eaa0000 rw-p 00024000 08:01 556136 /lib64/ld-2.17.so
7fffeb38e000-7fffeb3af000 rw-p 00000000 00:00 0 [stack]
7fffeb3ff000-7fffeb400000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

File limits:

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 23978 23978 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 23978 23978 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Comment by Pawe? Smoli?ski [ 11/Jan/14 ]

Shell environment variables and limits follow

pawel(17:46:14):~$ set
ANT_HOME=/usr/lib/ant
BASH=/bin/bash
BASHOPTS=cmdhist:expand_aliases:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="2" [2]="45" [3]="2" [4]="release" [5]="x86_64-slackware-linux-gnu")
BASH_VERSION='4.2.45(2)-release'
BROWSER=/opt/firefox/firefox
COLORFGBG='15;0'
COLUMNS=180
CPLUS_INCLUDE_PATH=/usr/lib64/qt/include
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-YKRI8r4UpG,guid=f526edf0e5e6d5f008d264f152c4195c
DESKTOP_SESSION=default
DIRSTACK=()
DISPLAY=:0
DM_CONTROL=/var/run/xdmctl
EUID=1000
GDK_USE_XFT=1
GIT_PAGER=
GROUPS=()
GS_LIB=/home/pawel/.fonts
GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/pawel/.gtkrc-2.0:/home/pawel/.kde/share/config/gtkrc-2.0
GTK_RC_FILES=/etc/gtk/gtkrc:/home/pawel/.gtkrc:/home/pawel/.kde/share/config/gtkrc
G_BROKEN_FILENAMES=1
HISTFILE=/home/pawel/.bash_history
HISTFILESIZE=500
HISTSIZE=500
HOME=/home/pawel
HOSTNAME=metis
HOSTTYPE=x86_64
IFS=$' \t\n'
INPUTRC=/etc/inputrc
JAVA_HOME=/opt/java
KDEDIRS=/usr
KDE_FULL_SESSION=true
KDE_MULTIHEAD=false
KDE_SESSION_UID=1000
KDE_SESSION_VERSION=4
KONSOLE_DBUS_SERVICE=:1.1283
KONSOLE_DBUS_SESSION=/Sessions/1
KONSOLE_DBUS_WINDOW=/Windows/1
KONSOLE_PROFILE_NAME=Powłoka
LANG=pl_PL
LANGUAGE=
LC_ALL=pl_PL
LC_COLLATE=pl_PL
LESS=-M
LESSOPEN='|lesspipe.sh %s'
LINES=50
LOGNAME=pawel
LS_COLORS=
LS_OPTIONS='-F -b -T 0 --color=auto'
MACHTYPE=x86_64-slackware-linux-gnu
MAILCHECK=60
MANPATH=/usr/local/man:/usr/man
MINICOM='-c on'
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/ant/bin:/opt/java/bin:/usr/lib64/kde4/libexec:/usr/lib64/qt/bin:.
PIPESTATUS=([0]="0")
PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/lib64/pkgconfig
PPID=18895
PROFILEHOME=
PS1='\u(\t):\W$ '
PS2='> '
PS4='+ '
PWD=/home/pawel
QT4DIR=/usr/lib64/qt
QT_PLUGIN_PATH=/home/pawel/.kde/lib64/kde4/plugins/:/usr/lib64/kde4/plugins/
SESSION_MANAGER=local/metis:@/tmp/.ICE-unix/1036,unix/metis:/tmp/.ICE-unix/1036
SHELL=/bin/bash
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor
SHELL_SESSION_ID=fb46bc0c17214000bf997d53e99ea1f7
SHLVL=2
T1LIB_CONFIG=/usr/share/t1lib/t1lib.config
TERM=xterm
UID=1000
USER=pawel
WINDOWID=56623131
WINDOWPATH=7
XCURSOR_THEME=ZipoTheme
XDG_CONFIG_DIRS=/etc/xdg:/etc/kde/xdg
XDG_CURRENT_DESKTOP=KDE
XDG_DATA_DIRS=/usr/share:/usr/share:/usr/local/share
XDG_SESSION_COOKIE=966ba2714e585a14d1ecc35c00000016-1388583258.981937-1915499397
XDM_MANAGED=method=classic
_=echo
__GL_YIELD=USLEEP

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 23978
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 23978
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Comment by Eliot Horowitz (Inactive) [ 08/Jan/14 ]

Can you send all shell environment setting and limits?
This is how most people start mongod, so nothing jumping out as to why its not working for you yet.

Generated at Thu Feb 08 03:28:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.