[SERVER-7285] Support systemd in future compatible distributions Created: 05/Oct/12  Updated: 23/Nov/16  Resolved: 16/Feb/16

Status: Closed
Project: Core Server
Component/s: Packaging, Usability
Affects Version/s: 2.2.0
Fix Version/s: 3.2.9, 3.3.2

Type: Bug Priority: Major - P3
Reporter: Edouard Perov Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 17
Labels: code-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Fedora >= 15, RHEL7


Issue Links:
Depends
depends on SERVER-14679 (CentOS 7/RHEL 7) init.d script shoul... Closed
is depended on by SERVER-22024 Mongod crash after restart (missing s... Closed
Duplicate
duplicates SERVER-13107 systemd init scripts Closed
is duplicated by SERVER-11238 starting mongod service on fedora fai... Closed
is duplicated by SERVER-15724 Cannot write pid file to /var/run/mon... Closed
is duplicated by SERVER-18162 Fail to start with non-existing /var/... Closed
is duplicated by SERVER-24539 Ubuntu 16.04 packages for 3.2 do not ... Closed
is duplicated by SERVER-9202 Bad pid file configuration in rpm Closed
is duplicated by SERVER-12514 mongo-10gen-server rpm/yum init scrip... Closed
Related
related to SERVER-16219 Fedora: cannot start service after in... Closed
related to SERVER-18439 Unable to start MongoDB 3.0.2 service... Closed
is related to SERVER-17742 Support for Ubuntu 15.04 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Build 10 (02/19/16)
Participants:

 Description   

Hello,
I cannot start mongodb 2.2.0 on Fedora 17
got the error
Oct 05 12:59:50 Evkalipt runuser[1787]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
Oct 05 12:59:50 Evkalipt mongod[1783]: Starting mongod: can't open [/var/log/mongo/mongod.log] for log file: errno:13 Permission denied
Oct 05 12:59:50 Evkalipt runuser[1787]: pam_unix(runuser:session): session closed for user mongod

permissions are ok for mongod user

drwxr-xr-x. 2 mongod mongod 4096 Oct 5 12:41 /var/log/mongo
rw-r----. 1 mongod mongod 0 Aug 28 22:42 mongod.log

Thanks,
Edouard



 Comments   
Comment by Ramon Fernandez Marina [ 30/Jul/16 ]

cloudkitten_, can you please open a separate ticket and provide the diff of the changes you made? That will help investigating this issue.

Thanks,
Ramón.

Comment by Ewald van Geffen [ 29/Jul/16 ]

I experience problems on CentOS7 with the sysvinit compability generators. As quickfix I simply copied the broken systemd generated file and manually fixed and copied it into the right place. The problem was with over 100 PHP-FPM pools the After= statement would be so long it somehow resulted strange issues.

● mongod.service - SYSV: Mongo is a scalable, document-oriented database.
   Loaded: error (Reason: Bad message)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

Jul 29 14:08:13 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:8] Failed to add dependency on php-fp, ignoring: Invalid argument
Jul 29 14:08:13 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:9] Missing '='.
Jul 29 14:23:25 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:8] Failed to add dependency on php-fpm-crokycup, ignoring: Invalid argument
Jul 29 14:23:25 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:9] Missing '='.
Jul 29 15:35:57 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:8] Failed to add dependency on p, ignoring: Invalid argument
Jul 29 15:35:57 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:9] Missing '='.
Jul 29 15:45:55 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:8] Failed to add dependency on php-fpm-remu, ignoring: Invalid argument
Jul 29 15:45:55 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:9] Missing '='.
Jul 29 15:47:26 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:8] Failed to add dependency on php-fpm-wi, ignoring: Invalid argument
Jul 29 15:47:26 kitty.be systemd[1]: [/run/systemd/generator.late/mongod.service:9] Missing '='.

I feel it's better if we can provide native upstart files. Contact me if you need more additional information as this is very much reproducible.

Comment by Githook User [ 15/Jul/16 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-7285: future proof ubuntu packaging with regards to systemd

(cherry picked from commit f7414114d89d7c07ee1f2282290897ddbd27985c)
Branch: v3.2
https://github.com/mongodb/mongo/commit/1f08e9e1df52253cb303f20028c2f7089ab97d16

Comment by Githook User [ 13/Jun/16 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-7285 SERVER-18329: add service file for debian packages

(cherry picked from commit 6f01e1c7555e39e3618953c3fc0a5c5c216be09b)
Branch: v3.2
https://github.com/mongodb/mongo/commit/d7d5d7a1cc444e39c645631044fb1663cee46f4a

Comment by Kanstantsin Shautsou [ 07/Jun/16 ]

RHEL7 and SUSE12 packages rely on the sysvinit->systemd compatibility layer. These processes start and stop MongoDB using the init script but the process is managed, to some extent, by systemd. As I understand it, the installation and upgrade process for RPM packages do not keep track of the state of the program, and as a result, there's no reliable or supported upgrade path that includes transitioning between init systems. As a result RHEL7 and SUSE12 packages will continue to use the sysvinit compatibility system.

http://www.rpm.org/max-rpm/s1-rpm-inside-scripts.html#S3-RPM-INSIDE-PREUN-SCRIPT and https://fedoraproject.org/wiki/Packaging:Scriptlets
Sysv script is really unreliable, it redirects any errors into dev/null. And the only one single sysv script that i have on all servers today. Would be very glad if el7 packaging would stop shipping it.
Is there anything i can help with?

Comment by Githook User [ 16/Feb/16 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-7285: future proof ubuntu packaging with regards to systemd
Branch: master
https://github.com/mongodb/mongo/commit/f7414114d89d7c07ee1f2282290897ddbd27985c

Comment by Sam Kleinman (Inactive) [ 11/Feb/16 ]

After a thorough review of our own packaging infrastructure and the recommendations in packaging guidelines I have a couple of comments regarding this case:

  • The new packages for Debian 8 (jessie) that are currently in progress and covered by SERVER-18329 will use systemd service files to control the process, and will be a completely native systemd package.
  • RHEL7 and SUSE12 packages rely on the sysvinit->systemd compatibility layer. These processes start and stop MongoDB using the init script but the process is managed, to some extent, by systemd. As I understand it, the installation and upgrade process for RPM packages do not keep track of the state of the program, and as a result, there's no reliable or supported upgrade path that includes transitioning between init systems. As a result RHEL7 and SUSE12 packages will continue to use the sysvinit compatibility system.
  • In the future when we create packages for RHEL8 and SUSE13 and any other RPM distro that we package with that may use systemd, we can (and should) transition these packages to use native systemd process management. Additionally, following from the the work on SERVER-18329, we can and will package native systemd process control for all new Debian and Ubuntu based distributions moving forward.

Based on this I'm going to go ahead and close this ticket, thanks for your patience!

Comment by Githook User [ 09/Feb/16 ]

Author:

{u'username': u'tychoish', u'name': u'Sam Kleinman', u'email': u'samk@10gen.com'}

Message: SERVER-7285 SERVER-18329: add service file for debian packages
Branch: master
https://github.com/mongodb/mongo/commit/6f01e1c7555e39e3618953c3fc0a5c5c216be09b

Comment by Alex [ 07/Dec/15 ]

systemd is now the default even on Debian 8

Comment by Ernie Hershey [ 25/Mar/15 ]

andreibarsan we're working to prioritize this. You're right that systemd is becoming the default in more and more places and the sooner we improve our support for it the easier it will make using MongoDB on those distributions.

octoquad - we'll probably have additional issues to work through before our packages work in Ubuntu 15.04. I've created a separate ticket to track that work - SERVER-17742

Comment by Bruce Pieterse [ 24/Mar/15 ]

To extend on Andrei Barsan's comment, Ubuntu Vivid Vervet (15.04) has made the switch to systemd a few weeks back. As a tester for Ubuntu Gnome, I upgraded earlier to catch any distribution upgrade bugs from 14.10. Unfortunately this also rendered my installation of MongoDB 2.6.x with the instructions followed here: http://docs.mongodb.org/v2.6/tutorial/install-mongodb-on-ubuntu/ now unusable:

» sudo service mongod start
Failed to start mongod.service: Unit mongod.service failed to load: No such file or directory.

» systemctl start mongod
Failed to start mongod.service: Unit mongod.service failed to load: No such file or directory.

If I need to file a seperate report for this, please let me know.

Thanks

Comment by Andrei Barsan [ 06/Mar/15 ]

Any news regarding this issue? There are already many Linux server distros which use systemd by default, such as Fedora, RHEL, OpenLogic and CentOS.

Comment by Githook User [ 11/Aug/14 ]

Author:

{u'username': u'mikemaccana', u'name': u'Mike MacCana', u'email': u'mike.maccana@gmail.com'}

Message: SERVER-7285 Add mongod.service file for Linux

Add a systemd .service file. This is used in place of System V init scripts on most current Linux distributions.

Closes #740

Signed-off-by: Benety Goh <benety@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/1663f45b4e33e058a0203a81f50574a870882c34

Comment by Ernie Hershey [ 11/Jul/14 ]

FWIW - I just verified that on a stock rhel 7.0 machine with systemd, our community RPM with non-systemd init script installs cleanly and lets you control the service via the 'service' command, according to the standard docs here - http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/

Comment by Tim Hawkins [ 28/Dec/12 ]

The fix i have given only really works for a standard default install, if the user reconfigures the server to have its data-files in a different place, the problem will resurface.

Im not sure how best to fix this, putting the pid file into a fixed place like /var/run/mongod.pid would cause problems with multiple instances, which are quite common on mongo installations.

Perhaps having the init.d script set the pid path by the command-line setting, as /var/run/mongod.pid, is acceptable, so folks can set the datadir to whatever they like via the conf file, so long as documentation is made available for troubleshooting if it should stall.

The problem is really in systemd, it should never stall, and should time out with a suitable error message. The fact that you can get you system into a non bootable state with this is not good.

Comment by Angel Rivera [ 27/Dec/12 ]

Tim

I can confirm your fix works. I just had this same issue with my new Fedora install. I changed the commented line to pidfile: /var/lib/mongo/mongod.lock and ran the sudo systemctl --system daemon-reload command and it fixed the issue.

I downloaded the mongodb 2.2.2 from the 10gen repos using yum and the boot problem occurred immediately after I rebooted.

Comment by Tim Hawkins [ 04/Dec/12 ]

I'm not sure where this issue is coming from, but my experiences while not ideal are quite different on Fedora 17

I have alsways installed all my copies of Mongo on F16 and F17 from the mongodb Repo using "yum install mongo-10gen mongo-10gen-server"

There IS a failure mode, but for me its not related to permissions, systemd when presented with a new init.d script, reads the script and creates the systemd unit file automaticaly. The mongod init.d script has an incorrect pidfile path which points to a non-existant pidfile.

changing this comment to (for the default install)

  1. pidfile: /var/lib/mongo/mongod.lock

and rerunning

sudo systemctl --system daemon-reload

causes systemd to reimport the init.d script and rebuild the systemd unit file.

So now systemd can find the mongod.lock file which contains the running server pid.

Without this two things will happen.

1. Starting mongod with
service mongod start

will stall, as its waiting for the pid file to be created, mongod actualy starts up, but the command never returns.

2. If you have executed
sudo chkconfig mongod on

to get mongo to startup at boot, then your system will hard hang on restart, and you will have to use the emergency console to recover your system.

Comment by Mark porter [ 11/Oct/12 ]

Hi Edouard,

Thanks for your update and that's what I thought and explains why 2.0.7 works via "service".

I will update this ticket with our progress.

Thanks

Mark

Comment by Edouard Perov [ 10/Oct/12 ]

Hi Mark,
I installed 2.0.7 from EPEL.
Actually, I do not need a fix for that because can start the MongoDB directly with the command.
Thanks for your help.
Edouard

Comment by Mark porter [ 10/Oct/12 ]

Hi Edouard,

I believe that in Fedora 15, Red Hat switched from upstart to systemd as the default init replacement. None of our RPMs support systemd and I believe that your issue has been a problem for quite some time. I suspect that this may not have come up before because most folk use the EPEL repo, which does the correct thing whereas ours doesn't. EPEL do not have 2.2 in their repo yet as far as I know.

Can I confirm the you installed MongoDB 2.0.7 from our repo or from another repo?

If the supposition that I have detailed above is correct, we do not believe that the fix is trivial so we are currently discussing what is best to do regarding a solution.

Thanks

Mark

Comment by Edouard Perov [ 10/Oct/12 ]

Hi Mark,
Is there any update?
Thanks,
Edouard

Comment by Mark porter [ 09/Oct/12 ]

Hi Edouard,

I can reproduce your issue. I am still investigating and will update you with more details as soon as I have something more detailed.

Thanks

Mark

Comment by Gianfranco Palumbo [ 09/Oct/12 ]

So it was working on 2.0.7 ?

Can you check the permission for your data file directory as well as post the /etc/mongod.conf?

Comment by Edouard Perov [ 07/Oct/12 ]

I have found that strafing like this works:
mongod -f /etc/mongod.conf

starting via service does not.

Comment by Edouard Perov [ 06/Oct/12 ]

Installed 2.2.0 back and providing snapshots to show that I start it as root as the official note recommends
==========================================
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-redhat-centos-or-fedora-linux/
Exerpt:
Start the mongod process by issuing the following command (as root, or with sudo):

UNIX> service mongod start
===========================================

This is my screen snapshot

1. List of installed mongodb RPMs
[root@Evkalipt ~]# rpm -qa|grep mongo
mongo-10gen-server-2.2.0-mongodb_1.x86_64
mongo-10gen-2.2.0-mongodb_1.x86_64

2. Current user - root
[root@Evkalipt ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

3. Start server
root@Evkalipt ~]# service mongod start
Starting mongod (via systemctl): Job failed. See system journal and 'systemctl status' for details.
[FAILED]
4. Error stack
[root@Evkalipt ~]# systemctl status mongod.service
mongod.service - SYSV: Mongo is a scalable, document-oriented database.
Loaded: loaded (/etc/rc.d/init.d/mongod)
Active: failed (Result: exit-code) since Sat, 06 Oct 2012 10:23:10 -0700; 6min ago
Process: 2094 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/mongod.service

Oct 06 10:23:10 Evkalipt runuser[2098]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
Oct 06 10:23:10 Evkalipt mongod[2094]: Starting mongod: can't open [/var/log/mongo/mongod.log] for log file: errno:13 Permission denied
Oct 06 10:23:10 Evkalipt runuser[2098]: pam_unix(runuser:session): session closed for user mongod
Oct 06 10:23:10 Evkalipt mongod[2094]: [FAILED]

5. List/owners/permissions of the directories

[root@Evkalipt ~]# ls -al /var//mong
/var/lib/mongo:
total 8
drwxr-xr-x. 2 mongod mongod 4096 Aug 28 22:42 .
drwxr-xr-x. 40 root root 4096 Oct 6 10:20 ..

/var/log/mongo:
total 8
drwxr-xr-x. 2 mongod mongod 4096 Oct 6 10:20 .
drwxr-xr-x. 12 root root 4096 Oct 6 10:20 ..
rw-r----. 1 mongod mongod 0 Aug 28 22:42 mongod.log

Comment by Edouard Perov [ 06/Oct/12 ]

Yes, I start it as root.
I de-installed 2.2.0 (from mongodb.org/downloads) and installed 2.0.7 using Fedora/updates and everything is ok.
I noticed the difference in directory names
2.2.0 - /var/log/mongo/
2.0.7 - /var/log/mongodb/

Might be /var/log/mongodb is "hardcoded" in 2.2.0 but it does on exist.

Comment by Eliot Horowitz (Inactive) [ 06/Oct/12 ]

The error is relatively clear, so are you sure you're starting mongo as the right user?

Generated at Thu Feb 08 03:14:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.