[SERVER-31225] The mongod process forks before listening for connections Created: 22/Sep/17  Updated: 30/Oct/23  Resolved: 01/Dec/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.9
Fix Version/s: 3.4.11, 3.6.1, 3.7.1

Type: Bug Priority: Major - P3
Reporter: Mark Agarunov Assignee: Mathew Robinson (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Duplicate
is duplicated by SERVER-31206 SystemD services depending on mongod ... Closed
Problem/Incident
causes SERVER-33319 Systemd startup should not always be ... Closed
Related
is related to SERVER-16292 mongod isn't ready for connections wh... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6, v3.4
Steps To Reproduce:

Create a systemd service to test if the socket is listening:

[Unit]
Type=oneshot
Description=TestService
After=mongod.service
Wants=mongod.service
 
[Service]
ExecStart=/usr/bin/testsocket
Restart=on-abort
 
[Install]
WantedBy=multi-user.target

With the following script (/usr/bin/testsocket)

#!/bin/sh
/bin/nc localhost 27017 </dev/null >/dev/null && echo "Listening" || echo "Not listening"

Stop mongod, and start the service which should start mongod first:

[root@localhost ~]# systemctl stop mongod
[root@localhost ~]# ps aux|egrep '[m]ongo'    
[root@localhost ~]# systemctl start testservice
[root@localhost ~]# systemctl status testservice
● testservice.service - TestService
 
Sep 21 19:49:02 localhost.localdomain systemd[1]: Started TestService.
Sep 21 19:49:02 localhost.localdomain systemd[1]: Starting TestService...
Sep 21 19:49:02 localhost.localdomain testsocket[3830]: Ncat: Connection refused.
Sep 21 19:49:02 localhost.localdomain testsocket[3830]: Not listening

Add a 1 second sleep via ExecStartPre in the testservice unit file, or to the beginning of /usr/bin/testsocket and mongod is listening by the time it runs:

[root@localhost ~]# systemctl stop mongod       
[root@localhost ~]# ps aux|egrep '[m]ongo'    
[root@localhost ~]# systemctl start testservice       
[root@localhost ~]# systemctl status testservice        
● testservice.service - TestService
 
Sep 21 19:52:13 localhost.localdomain systemd[1]: Started TestService.
Sep 21 19:52:13 localhost.localdomain systemd[1]: Starting TestService...
Sep 21 19:52:14 localhost.localdomain testsocket[3882]: Listening

Sprint: Build Ready
Participants:

 Description   

When starting mongod as a systemd service, the process forks before mongod has opened the socket and starts listening for connections. This can cause dependent services to fail to start, as systemd considers mongod started once the process has forked.



 Comments   
Comment by Githook User [ 05/Dec/17 ]

Author:

{'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com', 'name': 'Mathew Robinson'}

Message: SERVER-31225 Add Type=forking to systemd service files
Branch: v3.6
https://github.com/mongodb/mongo/commit/7aa806063d4279a728a65fc9202a3928e36cd6f1

Comment by Githook User [ 30/Nov/17 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-31225 Add Type=forking to systemd service files
Branch: v3.4
https://github.com/mongodb/mongo/commit/42e14f07cb17a5619e54354de471a9b5d4f6e5fa

Comment by Githook User [ 30/Nov/17 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-31225 Add Type=forking to systemd service files
Branch: master
https://github.com/mongodb/mongo/commit/3607059cd791b273ccaefd879d765dcd9365ebd6

Comment by Mathew Robinson (Inactive) [ 29/Nov/17 ]

I was able to reproduce using the systemd files from our packages:

[root@ip-10-30-199-253 system]# systemctl start mongod && mongo
MongoDB shell version v3.4.10-40-g2e9c0d0
connecting to: mongodb://127.0.0.1:27017
2017-11-29T23:07:03.452+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2017-11-29T23:07:03.453+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:240:13
@(connect):1:6
exception: connect failed
[root@ip-10-30-199-253 system]#

With Type=forking:

[root@ip-10-30-199-253 system]# systemctl start mongod && mongo
MongoDB shell version v3.4.10-40-g2e9c0d0
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.10-40-g2e9c0d0
Server has startup warnings:
2017-11-29T23:07:38.472+0000 I STORAGE  [initandlisten]
2017-11-29T23:07:38.472+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-11-29T23:07:38.472+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-11-29T23:07:46.919+0000 I CONTROL  [initandlisten]
2017-11-29T23:07:46.919+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-11-29T23:07:46.919+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-11-29T23:07:46.919+0000 I CONTROL  [initandlisten]
MongoDB Enterprise >

Comment by Mark Agarunov [ 06/Oct/17 ]

ernie.hershey, I confirmed that mongod does start listening before forking. This should more or less match the behavior of systemd with Type=forking:

[vagrant@localhost ~]$ mongod --logpath log --dbpath data --fork && testsocket 
about to fork child process, waiting until server is ready for connections.
forked process: 1978
child process started successfully, parent exiting
Listening

I believe that the Type=forking will fix this, as without it systemd will treat mongod as Type=simple:

If set to simple (the default if neither Type= nor BusName=, but ExecStart= are specified), it is expected that the process configured with ExecStart= is the main process of the service. In this mode, if the process offers functionality to other processes on the system, its communication channels should be installed before the daemon is started up (e.g. sockets set up by systemd, via socket activation), as systemd will immediately proceed starting follow-up units.

I reproduced this by launching mongod in a subshell:

[vagrant@localhost ~]$ mongod --logpath log --dbpath data --fork & testsocket 
[1] 2006
about to fork child process, waiting until server is ready for connections.
Ncat: Connection refused.
Not listening
[vagrant@localhost ~]$ forked process: 2010
child process started successfully, parent exiting

which matches the behavior without Type=forking.

Comment by Ernie Hershey [ 06/Oct/17 ]

henrik.edin are you certain that doing that will make Mark's reproduction case succeed? I think it will still fork early.

Comment by Henrik Edin [ 02/Oct/17 ]

I've been investigating this with mark.agarunov and it seems that the core issue is that the default mongod.service file doesn't have

Type=forking

By default systemd is using a 'simple' type where the process is forked automatically at start which is what causing this issue.

While looking into this it also seems like the default debian mongod.service and mongod.conf doesn't set fork: true in the processManagement block.

Generated at Thu Feb 08 04:26:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.