[SERVER-33319] Systemd startup should not always be "forking". Created: 14/Feb/18  Updated: 19/Mar/18  Resolved: 19/Mar/18

Status: Closed
Project: Core Server
Component/s: Packaging
Affects Version/s: 3.4.11, 3.4.12, 3.4.13
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Max Allan Assignee: Mathew Robinson (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by SERVER-31225 The mongod process forks before liste... Closed
Related
related to SERVER-33320 Changes to systemd script but no relo... Closed
Operating System: ALL
Steps To Reproduce:

Install an affected version of mongo. Set "fork: false" in your mongod.conf and start mongo with systemd. Wait a few minutes and systemd will kill it.

Sprint: Build 2018-03-12, Build 2018-03-26
Participants:

 Description   

In ticket SERVER-31225 it is suggested that mongo should be set to "type=forking" and this has been applied globally to all mongo deployments. However, it is not impossible that in your /etc/mongod.conf you can have :

processManagement:
  fork: false

And then your server starts and after a while systemd notices it hasn't forked and kills it.
I think startup=simple (or leave it to default) should be the default in the release and if people want to customise their own startup type, the RPM package should not overwrite customised systemd scripts.



 Comments   
Comment by Max Allan [ 19/Mar/18 ]

"We have to ship our service file assuming that you're using our default configurations."
And "Type=simple" works with default and customised configurations.
It's a good job no other package on the planet has this "default configuration" requirement. "Want your webserver to serve content from other than /var/www/html, well you need to customise your startup script. Oh, and by the way when you do that, you won't get future updates to the startup script with a yum update, you need to manually hack them in"

Changing the default startup script when people will already have an unmodified startup script and working deployment will only cause you support hassle not me. It's up to you if you want to maintain this

I'm not sure why you're trying to demonstrate that a modified systemd script will not be changed, I know, I haven't modified the script (I didn't need to, before you made this change, it worked) , so it gets upgraded by yum. This is a good thing, any new changes you add to the script will get made during updates. By forcing some people to change their startup script because they have made valid config changes to the product, they are now going to have to compare all future changes you make and retrofit them to their own startup script. This is probably not great.

[root@ip-10-66-2-175 yum.repos.d]# yum install  mongodb-org-server-3.4.10
.... yum stuff .....
[root@ip-10-66-2-175 yum.repos.d]# grep -i type /etc/systemd/system/multi-user.target.wants/mongod.service 
[root@ip-10-66-2-175 yum.repos.d]# yum -y update mongodb-org-server
..... yum stuff .....
[root@ip-10-66-2-175 yum.repos.d]# grep -i type /etc/systemd/system/multi-user.target.wants/mongod.service 
Type=forking
[root@ip-10-66-2-175 yum.repos.d]# systemctl start mongod
Warning: mongod.service changed on disk. Run 'systemctl daemon-reload' to reload units.

yum WILL update an unmodified systemd script. That's its job. So your statement "Second, our RPM's do not overwrite existing systemd service files" is wrong (and your apostrophe is superfluous).

Are you seriously saying customers should not change mongod.conf ?

Comment by Mathew Robinson (Inactive) [ 19/Mar/18 ]

Hey Max, so a couple of quick notes. First, we ship with fork: true because we have to support some older init systems for our older RHEL and SUSE versions. Second, our RPMs do not overwrite existing systemd service files or existing /etc/mongod.conf files.

As an example I spun up a RHEL 7.0 server and installed MongoDB 3.4:

[root@ip-10-122-78-109 ~]# touch /etc/mongod.conf
[root@ip-10-122-78-109 ~]# echo "[Unit]
Description=Not a real service file
 
[Service]
ExecStart=/usr/bin/echo "TOTALLY STARTING MONGOD"
 
[Install]
WantedBy=multi-user.target" > /etc/systemd/system/mongod.service
[root@ip-10-122-78-109 ~]# yum install mongodb-org
Loaded plugins: langpacks, product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
mongodb-org-3.4                                                                                                                                                   | 2.5 kB  00:00:00     
mongodb-org-3.4/7Server/primary_db                                                                                                                                |  59 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package mongodb-org.x86_64 0:3.4.13-1.el7 will be installed
--> Processing Dependency: mongodb-org-tools = 3.4.13 for package: mongodb-org-3.4.13-1.el7.x86_64
--> Processing Dependency: mongodb-org-shell = 3.4.13 for package: mongodb-org-3.4.13-1.el7.x86_64
--> Processing Dependency: mongodb-org-server = 3.4.13 for package: mongodb-org-3.4.13-1.el7.x86_64
--> Processing Dependency: mongodb-org-mongos = 3.4.13 for package: mongodb-org-3.4.13-1.el7.x86_64
--> Running transaction check
---> Package mongodb-org-mongos.x86_64 0:3.4.13-1.el7 will be installed
---> Package mongodb-org-server.x86_64 0:3.4.13-1.el7 will be installed
---> Package mongodb-org-shell.x86_64 0:3.4.13-1.el7 will be installed
---> Package mongodb-org-tools.x86_64 0:3.4.13-1.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=========================================================================================================================================================================================
 Package                                          Arch                                 Version                                       Repository                                     Size
=========================================================================================================================================================================================
Installing:
 mongodb-org                                      x86_64                               3.4.13-1.el7                                  mongodb-org-3.4                               5.8 k
Installing for dependencies:
 mongodb-org-mongos                               x86_64                               3.4.13-1.el7                                  mongodb-org-3.4                                12 M
 mongodb-org-server                               x86_64                               3.4.13-1.el7                                  mongodb-org-3.4                                20 M
 mongodb-org-shell                                x86_64                               3.4.13-1.el7                                  mongodb-org-3.4                                11 M
 mongodb-org-tools                                x86_64                               3.4.13-1.el7                                  mongodb-org-3.4                                49 M
 
Transaction Summary
=========================================================================================================================================================================================
Install  1 Package (+4 Dependent packages)
 
Total download size: 91 M
Installed size: 258 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): mongodb-org-3.4.13-1.el7.x86_64.rpm                                                                                                                        | 5.8 kB  00:00:00     
(2/5): mongodb-org-mongos-3.4.13-1.el7.x86_64.rpm                                                                                                                 |  12 MB  00:00:00     
(3/5): mongodb-org-shell-3.4.13-1.el7.x86_64.rpm                                                                                                                  |  11 MB  00:00:00     
(4/5): mongodb-org-server-3.4.13-1.el7.x86_64.rpm                                                                                                                 |  20 MB  00:00:01     
(5/5): mongodb-org-tools-3.4.13-1.el7.x86_64.rpm                                                                                                                  |  49 MB  00:00:01     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                     43 MB/s |  91 MB  00:00:02     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mongodb-org-server-3.4.13-1.el7.x86_64                                                                                                                                1/5 
warning: /etc/mongod.conf created as /etc/mongod.conf.rpmnew
Failed to execute operation: Invalid argument
warning: %post(mongodb-org-server-3.4.13-1.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package mongodb-org-server-3.4.13-1.el7.x86_64
  Installing : mongodb-org-mongos-3.4.13-1.el7.x86_64                                                                                                                                2/5 
  Installing : mongodb-org-tools-3.4.13-1.el7.x86_64                                                                                                                                 3/5 
  Installing : mongodb-org-shell-3.4.13-1.el7.x86_64                                                                                                                                 4/5 
  Installing : mongodb-org-3.4.13-1.el7.x86_64                                                                                                                                       5/5 
  Verifying  : mongodb-org-shell-3.4.13-1.el7.x86_64                                                                                                                                 1/5 
  Verifying  : mongodb-org-tools-3.4.13-1.el7.x86_64                                                                                                                                 2/5 
  Verifying  : mongodb-org-mongos-3.4.13-1.el7.x86_64                                                                                                                                3/5 
  Verifying  : mongodb-org-server-3.4.13-1.el7.x86_64                                                                                                                                4/5 
  Verifying  : mongodb-org-3.4.13-1.el7.x86_64                                                                                                                                       5/5 
 
Installed:
  mongodb-org.x86_64 0:3.4.13-1.el7                                                                                                                                                      
 
Dependency Installed:
  mongodb-org-mongos.x86_64 0:3.4.13-1.el7      mongodb-org-server.x86_64 0:3.4.13-1.el7      mongodb-org-shell.x86_64 0:3.4.13-1.el7      mongodb-org-tools.x86_64 0:3.4.13-1.el7     
 
Complete!

As you can see our fake files in /etc/systemd/system and /etc/mongod.conf, are unchanged:

[root@ip-10-122-78-109 system]# cat mongod.service 
[Unit]
Description=Not a real service file
 
[Service]
ExecStart=/usr/bin/echo "TOTALLY STARTING MONGOD"
 
[Install]
WantedBy=multi-user.target
[root@ip-10-122-78-109 system]# cat /etc/mongod.conf
[root@ip-10-122-78-109 system]# 

and just to verify myself I went ahead and upgraded from MongoDB 3.4 to 3.6:

[root@ip-10-122-78-109 ~]# vim /etc/yum.repos.d/mongodb-org-3.6.repo 
[root@ip-10-122-78-109 ~]# yum upgrade mongodb*
Loaded plugins: langpacks, product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
mongodb-org-3.6                                                                                                                                                   | 2.5 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package mongodb-org.x86_64 0:3.4.13-1.el7 will be updated
---> Package mongodb-org.x86_64 0:3.6.3-1.el7 will be an update
---> Package mongodb-org-mongos.x86_64 0:3.4.13-1.el7 will be updated
---> Package mongodb-org-mongos.x86_64 0:3.6.3-1.el7 will be an update
---> Package mongodb-org-server.x86_64 0:3.4.13-1.el7 will be updated
---> Package mongodb-org-server.x86_64 0:3.6.3-1.el7 will be an update
---> Package mongodb-org-shell.x86_64 0:3.4.13-1.el7 will be updated
---> Package mongodb-org-shell.x86_64 0:3.6.3-1.el7 will be an update
---> Package mongodb-org-tools.x86_64 0:3.4.13-1.el7 will be updated
---> Package mongodb-org-tools.x86_64 0:3.6.3-1.el7 will be an update
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=========================================================================================================================================================================================
 Package                                           Arch                                  Version                                    Repository                                      Size
=========================================================================================================================================================================================
Updating:
 mongodb-org                                       x86_64                                3.6.3-1.el7                                mongodb-org-3.6                                5.8 k
 mongodb-org-mongos                                x86_64                                3.6.3-1.el7                                mongodb-org-3.6                                 12 M
 mongodb-org-server                                x86_64                                3.6.3-1.el7                                mongodb-org-3.6                                 20 M
 mongodb-org-shell                                 x86_64                                3.6.3-1.el7                                mongodb-org-3.6                                 12 M
 mongodb-org-tools                                 x86_64                                3.6.3-1.el7                                mongodb-org-3.6                                 46 M
 
Transaction Summary
=========================================================================================================================================================================================
Upgrade  5 Packages
 
Total download size: 90 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for mongodb-org-3.6
(1/5): mongodb-org-3.6.3-1.el7.x86_64.rpm                                                                                                                         | 5.8 kB  00:00:00     
(2/5): mongodb-org-mongos-3.6.3-1.el7.x86_64.rpm                                                                                                                  |  12 MB  00:00:00     
(3/5): mongodb-org-server-3.6.3-1.el7.x86_64.rpm                                                                                                                  |  20 MB  00:00:00     
(4/5): mongodb-org-shell-3.6.3-1.el7.x86_64.rpm                                                                                                                   |  12 MB  00:00:00     
(5/5): mongodb-org-tools-3.6.3-1.el7.x86_64.rpm                                                                                                                   |  46 MB  00:00:01     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                     52 MB/s |  90 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : mongodb-org-server-3.6.3-1.el7.x86_64                                                                                                                                1/10 
warning: /etc/mongod.conf created as /etc/mongod.conf.rpmnew
  Updating   : mongodb-org-tools-3.6.3-1.el7.x86_64                                                                                                                                 2/10 
  Updating   : mongodb-org-shell-3.6.3-1.el7.x86_64                                                                                                                                 3/10 
  Updating   : mongodb-org-mongos-3.6.3-1.el7.x86_64                                                                                                                                4/10 
  Updating   : mongodb-org-3.6.3-1.el7.x86_64                                                                                                                                       5/10 
  Cleanup    : mongodb-org-3.4.13-1.el7.x86_64                                                                                                                                      6/10 
  Cleanup    : mongodb-org-server-3.4.13-1.el7.x86_64                                                                                                                               7/10 
  Cleanup    : mongodb-org-shell-3.4.13-1.el7.x86_64                                                                                                                                8/10 
  Cleanup    : mongodb-org-mongos-3.4.13-1.el7.x86_64                                                                                                                               9/10 
  Cleanup    : mongodb-org-tools-3.4.13-1.el7.x86_64                                                                                                                               10/10 
  Verifying  : mongodb-org-mongos-3.6.3-1.el7.x86_64                                                                                                                                1/10 
  Verifying  : mongodb-org-shell-3.6.3-1.el7.x86_64                                                                                                                                 2/10 
  Verifying  : mongodb-org-tools-3.6.3-1.el7.x86_64                                                                                                                                 3/10 
  Verifying  : mongodb-org-3.6.3-1.el7.x86_64                                                                                                                                       4/10 
  Verifying  : mongodb-org-server-3.6.3-1.el7.x86_64                                                                                                                                5/10 
  Verifying  : mongodb-org-tools-3.4.13-1.el7.x86_64                                                                                                                                6/10 
  Verifying  : mongodb-org-mongos-3.4.13-1.el7.x86_64                                                                                                                               7/10 
  Verifying  : mongodb-org-shell-3.4.13-1.el7.x86_64                                                                                                                                8/10 
  Verifying  : mongodb-org-server-3.4.13-1.el7.x86_64                                                                                                                               9/10 
  Verifying  : mongodb-org-3.4.13-1.el7.x86_64                                                                                                                                     10/10 
 
Updated:
  mongodb-org.x86_64 0:3.6.3-1.el7             mongodb-org-mongos.x86_64 0:3.6.3-1.el7       mongodb-org-server.x86_64 0:3.6.3-1.el7       mongodb-org-shell.x86_64 0:3.6.3-1.el7      
  mongodb-org-tools.x86_64 0:3.6.3-1.el7      
 
Complete!
[root@ip-10-122-78-109 ~]# 

and the files were still unchanged:

[root@ip-10-122-78-109 system]# cat mongod.service 
[Unit]
Description=Not a real service file
 
[Service]
ExecStart=/usr/bin/echo "TOTALLY STARTING MONGOD"
 
[Install]
WantedBy=multi-user.target
[root@ip-10-122-78-109 system]# cat /etc/mongod.conf
[root@ip-10-122-78-109 system]# 

If you have a custom service file in /etc/systemd/system it will take precedence over /usr/lib/systemd/system, so you can see if I run systemctl start mongod:

[root@ip-10-122-78-109 system]# systemctl start mongod
[root@ip-10-122-78-109 system]# systemctl status mongod
● mongod.service - Not a real service file
   Loaded: loaded (/etc/systemd/system/mongod.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
 
Mar 19 14:12:53 ip-10-122-78-109.ec2.internal systemd[1]: mongod.service failed.
Mar 19 14:15:16 ip-10-122-78-109.ec2.internal systemd[1]: Started Not a real service file.
Mar 19 14:15:16 ip-10-122-78-109.ec2.internal systemd[1]: Starting Not a real service file...
Mar 19 14:15:16 ip-10-122-78-109.ec2.internal echo[12188]: TOTALLY STARTING MONGOD
[root@ip-10-122-78-109 system]# ls /etc/systemd/system
basic.target.wants                         default.target        mongod.service           remote-fs.target.wants  system-update.target.wants
dbus-org.fedoraproject.FirewallD1.service  default.target.wants  multi-user.target.wants  sockets.target.wants
dbus-org.freedesktop.Avahi.service         getty.target.wants    printer.target.wants     sysinit.target.wants
[root@ip-10-122-78-109 system]# cat /etc/systemd/system/mongod.service 
[Unit]
Description=Not a real service file
 
[Service]
ExecStart=/usr/bin/echo "TOTALLY STARTING MONGOD"
 
[Install]
WantedBy=multi-user.target
[root@ip-10-122-78-109 system]# 

It started my fake service file instead of the one that we ship which installs in /usr/lib/systemd/system/mongod.service.

[root@ip-10-122-78-109 system]# cat /usr/lib/systemd/system/mongod.service 
[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
Documentation=https://docs.mongodb.org/manual
 
[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongod.conf"
ExecStart=/usr/bin/mongod $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
 
[Install]
WantedBy=multi-user.target
[root@ip-10-122-78-109 system]# 

When there is a conflict between an existing file and a file inside the package, rpm creates a new version (you can see this in the rpm output) called filename.rpmnew and leaves the existing file alone. So in short, since we aren't overriding any user files I see no reason to change our systemd service file. If you are customizing your systemd service file it's best practice to put it in /etc/systemd/system per the systemd docs and if you're customizing your mongod.conf but not the service file we leave that to our customers to resolve. We have to ship our service file assuming that you're using our default configurations.

Comment by Max Allan [ 14/Feb/18 ]

Apologies. I have no edit permission and couldn't set the OS field to "RedHat" when creating the ticket. Could someone with privs update it?

Generated at Thu Feb 08 04:33:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.