[SERVER-18439] Unable to start MongoDB 3.0.2 service on CentOS 7 Created: 12/May/15  Updated: 01/Jun/22  Resolved: 01/Jun/22

Status: Closed
Project: Core Server
Component/s: Admin, Packaging
Affects Version/s: 3.0.2
Fix Version/s: 4.1 Desired

Type: Bug Priority: Major - P3
Reporter: Shashank Agrawal Assignee: Alex Neben
Resolution: Won't Fix Votes: 3
Labels: centos, service
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

centos7, amazon ec2


Issue Links:
Related
is related to SERVER-7285 Support systemd in future compatible ... Closed
Operating System: Linux
Steps To Reproduce:

Given in description

Participants:

 Description   

We are setting up a MongoDB server for the production environment on Amazon EC2 instance, but could not able to start the service. I've followed this documentation for setup. Here are the steps, I've taken for setting up the server:

Added following to `/etc/yum.repos.d/mongodb-org-3.0.repo`

[mongodb-org-3.0]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1

And installed MongoDB 3.0.2 using `sudo yum install -y mongodb-org-3.0.2`

Created three partitions for data, journal & log:

sudo mkdir /mongo
sudo mkdir /mongo/data
sudo mkdir /mongo/log
sudo mkdir /mongo/journal

Created file system for three separate partitions:

sudo mkfs.ext4 /dev/xvdb
sudo mkfs.ext4 /dev/xvdc
sudo mkfs.ext4 /dev/xvdd

Created entry in `fstab` for reboot:

echo '/dev/xvdb /mongo/data ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdc /mongo/journal ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdd /mongo/log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab

And mounted the partitions:

sudo mount /mongo/data
sudo mount /mongo/journal
sudo mount /mongo/log

Given the permissions and created link

sudo chown mongod:mongod /mongo/data /mongo/journal /mongo/log
sudo ln -s /mongo/journal /mongo/data/journal

Configured `ulimit` & read ahead settings as given in the documentation link above. Verified permissions and partitions:

[deployer@prod-mongo ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      8.0G  1.3G  6.8G  16% /
devtmpfs        3.6G     0  3.6G   0% /dev
tmpfs           3.5G     0  3.5G   0% /dev/shm
tmpfs           3.5G   57M  3.4G   2% /run
tmpfs           3.5G     0  3.5G   0% /sys/fs/cgroup
/dev/xvdc       7.8G   36M  7.3G   1% /mongo/journal
/dev/xvdb       150G   51M  149G   1% /mongo/data
/dev/xvdd       3.9G   16M  3.6G   1% /mongo/log

Permissions:

[deployer@prod-mongo ~]$ ll /
total 32
lrwxrwxrwx.   1 root   root      7 Sep 29  2014 bin -> usr/bin
dr-xr-xr-x.   4 root   root   4096 Sep 29  2014 boot
drwxr-xr-x.  17 root   root   2860 May 11 12:11 dev
lrwxrwxrwx.   1 root   root      7 Sep 29  2014 lib -> usr/lib
lrwxrwxrwx.   1 root   root      9 Sep 29  2014 lib64 -> usr/lib64
drwxr-xr-x.   2 root   root      6 Jun 10  2014 mnt
drwxr-xr-x.   5 mongod mongod   41 May 11 05:06 mongo
drwxr-xr-x.  21 root   root    660 May 11 12:47 run
lrwxrwxrwx.   1 root   root      8 Sep 29  2014 sbin -> usr/sbin

Inside `/mongo`

[deployer@prod-mongo ~]$ ll /mongo/
total 12
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:33 data
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:31 journal
drwxr-xr-x. 3 mongod mongod 4096 May 11 08:58 log

After changing the configurations inside `/etc/mongodb.conf`

logpath=/mongo/log/mongod.log
dbpath=/mongo/data

Also, changed the SELinux setting to `permissive`. Now when I'm doing: `sudo service mongod start`, I'm getting this error:

Starting mongod (via systemctl): Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[FAILED]

Further logging:

[deployer@prod-mongo ~]$ sudo systemctl status mongod.service

mongod.service - SYSV: Mongo is a scalable, document-oriented database.
Loaded: loaded (/etc/rc.d/init.d/mongod)
Active: failed (Result: exit-code) since Tue 2015-05-12 04:42:10 UTC; 42s ago
Process: 22881 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

May 11 04:42:10 ip-xx-xx-xx-xx.local runuser[22887]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 11 04:42:10 ip-xx-xx-xx-xx.localdomain runuser[22887]: pam_unix(runuser:session): session closed for user mongod
May 11 04:42:10 ip-xx-xx-xx-xx.local mongod[22881]: Starting mongod: [FAILED]
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: mongod.service: control process exited, code=exited status=1
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Unit mongod.service entered failed state.

Kernal version from uname -a:

3.10.0-229.1.2.el7.x86_64

If I directly run the mongodb service from the normal user something like this: sudo mongod --config /etc/mongod.conf, then this service is starting properly but not as the system service?

I've followed various articles and blog posts and stackexchange answers, but didn't get any solution.

http://stackoverflow.com/questions/30182016/unable-to-start-mongodb-3-0-2-service-on-centos-7



 Comments   
Comment by Shashank Agrawal [ 19/May/15 ]

Thank you, Ernie for your effort. The server on which we are facing problem has gone live for production. So I can't really test your solution on that. We are going to launch a same server and will do the same thing on that server and will get back to you on this.

Comment by Ernie Hershey [ 15/May/15 ]

I was able to reproduce the problem. I think it's either an oddity with systemd's sysvinit compatibility or just with systemd. As a workaround you can move your pid file back to a location under /var/run, or add "SYSTEMCTL_SKIP_REDIRECT=1" to the top of the init script, which will bypass the systemd call being made which is hanging.

Comment by Shashank Agrawal [ 14/May/15 ]

No, the content in the file /etc/sysconfig/mongod is empty and only contains one commented line.

Also, I'm using command sudo service mongod start and it is freezing after printing the logs I provided in my previous comment.

I tried again after rebooting our that database server and I'm still facing that issue. Not sure what is going on.

Comment by Ernie Hershey [ 14/May/15 ]

It all looks okay to me. And mongod is starting fine, which you can tell from this line:

2015-05-14T03:42:18.536+0000 I NETWORK  [initandlisten] waiting for connections on port 27017

But it shouldn't freeze. It should return.

Do you have anything in /etc/sysconfig/mongod?

One thing you can try is turning off "fork: true" although it seems to me more like having it inadvertently turned off already would be causing the problem you're seeing. Can you try that and see if it makes a difference?

Also sorry to ask for you to repeat yourself but when you see the freezing behavior can you tell me exactly what command you're using when you see that?

Thanks,
Ernie

Comment by Shashank Agrawal [ 14/May/15 ]

Hi Ernie, thank you for the quick reply on this.

Previously, when I was doing sudo service mongod start nothing was getting logged in the log file, but when I did it today after your comment I'm getting following log:

2015-05-14T03:30:30.120+0000 I CONTROL  ERROR: Cannot write pid file to /data/mongod.pid: Permission denied

Looks like a problem with permission, so I deleted the old content inside the data directory & fixed the permission issue since we were running the mongod process directly with the root user as described in my question.

Now when I'm starting the service again it's getting freeze at the state:

Starting mongod (via systemctl):

with the following logs in mongod.log

2015-05-14T03:42:18.160+0000 I JOURNAL  [initandlisten] journal dir=/data/journal
2015-05-14T03:42:18.161+0000 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2015-05-14T03:42:18.529+0000 I JOURNAL  [durability] Durability thread started
2015-05-14T03:42:18.530+0000 I JOURNAL  [journal writer] Journal writer thread started
2015-05-14T03:42:18.531+0000 I CONTROL  [initandlisten] MongoDB starting : pid=2596 port=27017 dbpath=/data 64-bit host=ip-10-0-0-181.localdomain
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten]
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten]
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten]
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] db version v3.0.2
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] git version: 6201872043ecbbc0a4cc169b5482dcf385fc464f
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] build info: Linux ip-10-169-109-10 3.10.0-121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-05-14T03:42:18.532+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", processManagement: { fork: true, pidFilePath: "/data/mongod.pid" }, storage: { dbPath: "/data" }, systemLog: { destination: "file", logAppend: true, path: "/log/mongod.log" } }
2015-05-14T03:42:18.536+0000 I NETWORK  [initandlisten] waiting for connections on port 27017

The data at /etc/mongodb.conf is having only following data (rest are commented):

logpath=/log/mongod.log
 
logappend=true
 
# fork and run in background
fork=true
 
#port=27017
 
dbpath=/data
 
pidfilepath=/data/mongod.pid

Also, content of /etc/init.d/mongod is:

#!/bin/bash
 
# mongod - Startup script for mongod
 
# chkconfig: 35 85 15
# description: Mongo is a scalable, document-oriented database.
# processname: mongod
# config: /etc/mongod.conf
# pidfile: /var/run/mongodb/mongod.pid
 
. /etc/rc.d/init.d/functions
 
# things from mongod.conf get there by mongod reading it
 
 
# NOTE: if you change any OPTIONS here, you get what you pay for:
# this script assumes all options are in the config file.
CONFIGFILE="/etc/mongod.conf"
OPTIONS=" -f $CONFIGFILE"
SYSCONFIG="/etc/sysconfig/mongod"
 
PIDFILEPATH=`awk -F'[:=]' -v IGNORECASE=1 '/^[[:blank:]]*(processManagement\.)?pidfilepath[[:blank:]]*[:=][[:blank:]]*/{print $2}' "$CONFIGFILE" | tr -d "[:blank:]\"'"`
 
mongod=${MONGOD-/usr/bin/mongod}
 
MONGO_USER=mongod
MONGO_GROUP=mongod
 
if [ -f "$SYSCONFIG" ]; then
    . "$SYSCONFIG"
fi
 
PIDDIR=`dirname $PIDFILEPATH`
 
# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
    NUMACTL="numactl $NUMACTL_ARGS"
else
    NUMACTL=""
fi
 
start()
{
  # Make sure the default pidfile directory exists
  if [ ! -d $PIDDIR ]; then
    install -d -m 0755 -o $MONGO_USER -g $MONGO_GROUP $PIDDIR
  fi
 
  # Recommended ulimit values for mongod or mongos
  # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
  #
  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 32000
 
  echo -n $"Starting mongod: "
  daemon --user "$MONGO_USER" --check $mongod "$NUMACTL $mongod $OPTIONS >/dev/null 2>&1"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/mongod
}
 
stop()
{
  echo -n $"Stopping mongod: "
  mongo_killproc "$PIDFILEPATH" $mongod
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/mongod
}
 
restart () {
	stop
	start
}
 
# Send TERM signal to process and wait up to 300 seconds for process to go away.
# If process is still alive after 300 seconds, send KILL signal.
# Built-in killproc() (found in /etc/init.d/functions) is on certain versions of Linux
# where it sleeps for the full $delay seconds if process does not respond fast enough to
# the initial TERM signal.
mongo_killproc()
{
  local pid_file=$1
  local procname=$2
  local -i delay=300
  local -i duration=10
  local pid=`pidofproc -p "${pid_file}" ${procname}`
 
  kill -TERM $pid >/dev/null 2>&1
  usleep 100000
  local -i x=0
  while [ $x -le $delay ] && checkpid $pid; do
    sleep $duration
    x=$(( $x + $duration))
  done
 
  kill -KILL $pid >/dev/null 2>&1
  usleep 100000
 
  rm -f "${pid_file}"
 
  checkpid $pid
  local RC=$?
  [ "$RC" -eq 0 ] && failure "${procname} shutdown" || success "${procname} shutdown"
  RC=$((! $RC))
  return $RC
}
 
RETVAL=0
 
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f /var/lock/subsys/mongod ] && restart || :
    ;;
  status)
    status $mongod
    RETVAL=$?
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    RETVAL=1
esac
 
exit $RETVAL

As you asked to run: bash -x /etc/rc.d/init.d/mongod start the output is:

+ . /etc/rc.d/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' 2014 -ne 1 -a -z '' ']'
++ /bin/mountpoint -q /cgroup/systemd
++ /bin/mountpoint -q /sys/fs/cgroup/systemd
++ case "$0" in
++ _use_systemctl=1
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
++ '[' -c /dev/stderr -a -r /dev/stderr ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -z '' ']'
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
++ '[' 1 = 1 ']'
++ '[' xstart = xstart -o xstart = xstop -o xstart = xrestart -o xstart = xreload -o xstart = xtry-restart -o xstart = xforce-reload -o xstart = xcondrestart ']'
++ systemctl_redirect /etc/rc.d/init.d/mongod start
++ local s
++ local prog=mongod
++ local command=start
++ local options=
++ case "$command" in
++ s='Starting mongod (via systemctl): '
++ '[' -n '' ']'
++ action 'Starting mongod (via systemctl): ' /bin/systemctl start mongod.service
++ local STRING rc
++ STRING='Starting mongod (via systemctl): '
++ echo -n 'Starting mongod (via systemctl):  '
Starting mongod (via systemctl):  ++ shift
++ /bin/systemctl start mongod.service
Failed to issue method call: Access denied
++ failure 'Starting mongod (via systemctl): '
++ local rc=4
++ '[' color '!=' verbose -a -z '' ']'
++ echo_failure
++ '[' color = color ']'
++ echo -en '\033[60G'
                                                           ++ echo -n '['
[++ '[' color = color ']'
++ echo -en '\033[0;31m'
++ echo -n FAILED
FAILED++ '[' color = color ']'
++ echo -en '\033[0;39m'
++ echo -n ']'
]++ echo -ne '\r'
++ return 1
++ '[' -x /bin/plymouth ']'
++ return 4
++ rc=4
++ echo
 
++ return 4
++ exit 4

Please note: I've mentioned in the question that we have created three separate partition mounted on /mongo/data, /mongo/journal and /mongo/log but we had changed it to simply /data, /journal and /log to further debug the problem before adding question on Jira but permissions were correct at that time also since I've added the real data on the question.

Comment by Ernie Hershey [ 13/May/15 ]

Hi sagrawal14 - Can you clear out your mongod.log and see if anything is in it after trying to start the service?

Also If you're comfortable with it, can you attach your mongod.conf and /etc/init.d/mongod?

And run this and share the output:

bash -x /etc/rc.d/init.d/mongod start

Thanks,
Ernie

Generated at Thu Feb 08 03:47:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.