[SERVER-42443] 4.0.11 mongodb-org Install Breaks Non-Root Logins Created: 26/Jul/19  Updated: 02/Aug/19  Resolved: 02/Aug/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.11
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Kenneth Kolano Assignee: Mathew Robinson (Inactive)
Resolution: Cannot Reproduce Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

On Ubuntu 19.04...

sudo apt-get install -y mongodb-org

reboot

Sprint: Dev Tools 2019-08-12
Participants:

 Description   

The 4.0.11 install results in...

"System is booting up. See pam_nologin(8)"

...errors when rebooting on Ubuntu 19.04.



 Comments   
Comment by Kenneth Kolano [ 02/Aug/19 ]

I do normally have a number of services that are dependent on MongoDB. Though I believe I was able to see the issue occur on an VM that had nothing but MongoDB installed,

 

I'll likely try returning the MongoDB service to "After=network.target" per SERVER-42603 to see if that resolves this.

Comment by Connecting Media [ 02/Aug/19 ]

It would also be interesting to know if you have any service (files) which depend on "mongod.service" (as currently being investigated in SERVER-42603).

Comment by Mathew Robinson (Inactive) [ 02/Aug/19 ]

Hey kolano@gmail.com,

I'm closing this as Cannot Reproduce. I've tried to get this to happen on my VM and have been unable to. Note that my VM is a clean Ubuntu install with no additional software installed or running so I'm not sure if there is a conflict there.

Since the issue is intermittent on your end it's hard to know what's causing it, but we exercise our packages on every commit on clean EC2 instances to verify that everything works as expected.

If you can get it to reproduce then do the following and provide the output:

  • Run ssh with the -vvv flag for a user who is locked out
  • Gain root access and run grep nologin /etc/passwd and collect the logs in /var/log/messages

You can gain root access even if PermitRootLogin is disabled since this is a VM you can boot the Ubuntu installation in single user mode. Instructions for how to boot in single-user mode can be found here:
https://askubuntu.com/questions/132965/how-do-i-boot-into-single-user-mode-from-grub

Comment by Connecting Media [ 02/Aug/19 ]

Possibly related to or even caused by SERVER-42603

Comment by Kenneth Kolano [ 31/Jul/19 ]

Was able to get this to re-occur today. Unclear why it's inconsistent. Of course I didn't allow logins for root, so now I'm locked out.

I can at least confirm that this issue effects non-SSH logins directly into the server.

Comment by Kenneth Kolano [ 31/Jul/19 ]

I was able to find a note that indicated the nologin was at...

/run/nologin

 

This is a separate issue, but I'm also noting that /w 4.0.11 the mongod service doesn't keep running, even though it seems MongoDB still is.

Comment by Mathew Robinson (Inactive) [ 31/Jul/19 ]

Given that I'm unable to replicate the issue and you also now cannot replicate it I'm not sure how we can proceed on this issue. I'm inclined to close this as gone away unless something has changed on your side Kenneth?

As for the presence of nologin files, I would expect your system to have a shell called nologin /usr/sbin/nologin or /bin/nologin perhaps (you can determine the path with which nologin. I would also expect your /etc/passwd file to have many nologin entries including our own service user that our packages create so getting specifics about the nologin entries in your passwd file is crucial, but only useful if the issue is occuring.

Comment by Kenneth Kolano [ 30/Jul/19 ]
  • This wasn't effecting 18.04. I'm using my own server /w the OS installed from the default Ubuntu 19.04 ISO.
  • Yes SSH logins were restricted, but I'd think the presence of nologin files would also effect non-SSH logins.
  • I haven't tried /w SSH keys, though I'm not clear we should expect such to not be effected by the presence of nologin

I could get in as root, with the revisions made in my first comment. A nologin file was left present, though I don't recall it's specific location.

Though this was occurring consistently on Jul 26, I'm currently unable to replicate it now for some reason, so I can't confirm the specific location of the remaining nologin file. Unclear why I can no longer replicate, as my test VM hasn't changed nor did the Mongo repo (or it was without revising the date at https://repo.mongodb.org/apt/ubuntu/dists/bionic/mongodb-org/4.0/Release).

Comment by Mathew Robinson (Inactive) [ 30/Jul/19 ]

Hey kolano@gmail.com,

I just want to confirm some things:

  • Are you running stock Ubuntu 18.04 server or is this some providers custom image? For example DigialOcean does not use the basic Ubuntu server ISO so if you didn't install this server yourself and it isn't an internal VM installation can you tell me who your VPS is for this server.
  • Looks like you mean "cannot SSH login" based on your comments. If you have access to a console for this server can you try a non-network login?
  • Also it looks like you might be using passwords over SSH for non-root users do SSH key non-root users also get kicked?

If you can get into the system as any user with 4.0.11 installed can you verify that no user has nologin set in /etc/passwd that shouldn't? You can use grep nologin /etc/passwd to see a list.

Comment by Kenneth Kolano [ 26/Jul/19 ]

I don't have a convenient VM for 18.04, but issue is not seen on 18.10; so I'd guess it works on 18.04.

Issue is only seen with 4.0.11.

For others arriving here with issues, the 4.0.10 release can be installed, as mentioned above, by specifying version details for both mongodb-org, as well as all it's sub-packages...

sudo apt-get install -y mongodb-org=4.0.10 mongodb-org-server=4.0.10 mongodb-org-shell=4.0.10 mongodb-org-mongos=4.0.10 mongodb-org-tools=4.0.10

Comment by Danny Hatcher (Inactive) [ 26/Jul/19 ]

MongoDB only supports Ubuntu 16.04 and 18.04. You're seeing this problem on a 19.04 system; could you try to reproduce on a 18.04? Do you see this issue only on 4.0.11 or do you see it on 4.0.10 as well?

Comment by Kenneth Kolano [ 26/Jul/19 ]

Forgot the package depot update steps in Steps To Reproduce: and apparently all dependencies need to be listed for version installs, so should be revised to...

 
On Ubuntu 19.04...

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list >/dev/null
sudo apt-get update
sudo apt-get install -y mongodb-org=4.0.11 mongodb-org-server=4.0.11 mongodb-org-shell=4.0.11 mongodb-org-mongos=4.0.11 mongodb-org-tools=4.0.11
sudo reboot

All non-root logins result in:

System is booting up. See pam_nologin(8)

No login will be possible, unless root login has previously been enabled...

sudo passwd root
sudo passwd -u root
sudo sed -i -E "s/# Authentication:/# Authentication:\nPermitRootLogin yes/" /etc/ssh/sshd_config

Generated at Thu Feb 08 05:00:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.