[SERVER-45237] After an upgrade mongod will not start on reboot Created: 18/Dec/19  Updated: 27/Jan/20  Resolved: 27/Jan/20

Status: Closed
Project: Core Server
Component/s: Upgrade/Downgrade
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Marshall Giguere Assignee: Mathew Robinson (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod-log.txt    
Operating System: ALL
Steps To Reproduce:

on Ubunt 18.04
upgrade to new mongo packages.
reboot, mongod will not have started.

change file: /lib/systemd/system/mongod.server
After=network.target

to
After=netword.target multi-user.target

Reboot, mongod now starts.

Sprint: Dev Platform 2020-01-13, Dev Platform 2020-01-27
Participants:

 Description   

After a mongodb package update mongod fails to start on system reboot. After a reboot mongod tries to start then bails out with and error "code 48". See attached log file excerpt.

I should note this only happens on update on 18.04. I have no similar issue on 16.04.

The problem seems to be with the systemd config. If I add the additional target"multi-user.target" to the "After=network.target" mongod starts on reboot.

I have looked at other incidents similar and the all relate to WiredTiger IIRC.

It's a minor but annoying problem, especially if I forget to modify the service file after and upgrade.

Mongodb version: 4.2.2
Distro: Ubuntu 18.04
CPU: AMD Ryzen6
RAM: 32Gb



 Comments   
Comment by April Schoffer [ 27/Jan/20 ]

Hi marsh.giguere@gmail.com despite repeated attempts, we've been unable to reproduce this issue. We suspect this issue is likely specific to your environment. Please feel free to reach out with more details if there is anything else you believe we can do to help.

Comment by Marshall Giguere [ 06/Jan/20 ]

Matt;

I don't recall doing anything unusual other than installing 4.2.2.

Here's the analysis reports

mongod.service @4.790s
`-network.target @4.788s
`-NetworkManager.service @4.460s +327ms
`-dbus.service @4.453s
`-basic.target @4.448s
`-sockets.target @4.448s
`-snapd.socket @4.447s +913us
`-sysinit.target @4.441s
`-systemd-timesyncd.service @4.234s +207ms
`-systemd-tmpfiles-setup.service @4.136s +95ms
`-systemd-journal-flush.service @1.465s +2.670s
`-var.mount @1.387s +75ms

`-systemd-fsck@dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.service
@536ms +850ms

`-dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.device
@535ms

network.target @4.788s
`-NetworkManager.service @4.460s +327ms
`-dbus.service @4.453s
`-basic.target @4.448s
`-sockets.target @4.448s
`-snapd.socket @4.447s +913us
`-sysinit.target @4.441s
`-systemd-timesyncd.service @4.234s +207ms
`-systemd-tmpfiles-setup.service @4.136s +95ms
`-systemd-journal-flush.service @1.465s +2.670s
`-var.mount @1.387s +75ms

`-systemd-fsck@dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.service
@536ms +850ms

`-dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.device
@535ms

On Mon, Jan 6, 2020 at 9:23 AM Mathew Robinson (Jira) <jira@mongodb.org>


If you can't hear the dogwoods barking you're taking things too seriously.

Comment by Mathew Robinson (Inactive) [ 06/Jan/20 ]

Thanks, that's close enough to our stock SystemD service file for our purposes.

I'll wait for the systemd-analyze results, I'm not able to repro this on my server so I'm suspicious that a dependency cycle has somehow been introduced in your init order.

Comment by Marshall Giguere [ 06/Jan/20 ]

Matt;

I'm not 100% sure this is the original from the deb. I think I only
changed the one line to get things working. I can safely say systemd is
not my favorite initializer. In order to provide you with the requested
analysis output I'll have to recycle my server, it's busy, I'll post as
soon as I can.

-Marsh

file: /lib/systemd/system/mongod.service

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network.target

[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid

  1. file size
    LimitFSIZE=infinity
  2. cpu time
    LimitCPU=infinity
  3. virtual memory size
    LimitAS=infinity
  4. open files
    LimitNOFILE=64000
  5. processes/threads
    LimitNPROC=64000
  6. locked memory
    LimitMEMLOCK=infinity
  7. total threads (user+kernel)
    TasksMax=infinity
    TasksAccounting=false
  1. Recommended limits for for mongod as specified in
  2. http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target

On Sun, Jan 5, 2020 at 6:20 PM Mathew Robinson (Jira) <jira@mongodb.org>


If you can't hear the dogwoods barking you're taking things too seriously.

Comment by Mathew Robinson (Inactive) [ 05/Jan/20 ]

Just for confirmation can you post the service file which produces this error as well? It looks like MongoDB is indeed trying to start before the network is available despite indicating After=network.target

Can you run systemd-analyze critical-chain mongod.service to show the dependency ordering? Might as well check systemd-analyze critical-chain network.target as well.

Comment by Marshall Giguere [ 04/Jan/20 ]
  • mongod.service - MongoDB Database Server
    Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor
    preset: enabled)
    Active: failed (Result: exit-code) since Sat 2020-01-04 10:53:59 EST;
    8min ago
    Docs: https://docs.mongodb.org/manual
    Process: 1313 ExecStart=/usr/bin/mongod --config /etc/mongod.conf
    (code=exited, status=48)
    Main PID: 1313 (code=exited, status=48)

Jan 04 10:53:59 xxxxx.xxxxx.xxx systemd[1]: Started MongoDB Database Server.
Jan 04 10:53:59 xxxxx.xxxxx.xxx systemd[1]: mongod.service: Main process
exited, code=exited, status=48/n/a
Jan 04 10:53:59 xxxxx.xxxxx.xxx systemd[1]: mongod.service: Failed with
result 'exit-code'.

On Thu, Jan 2, 2020 at 2:00 PM Mathew Robinson (Jira) <jira@mongodb.org>


If you can't hear the dogwoods barking you're taking things too seriously.

Comment by Mathew Robinson (Inactive) [ 02/Jan/20 ]

marsh.giguere@gmail.com

Can you post the output of systemctl status mongod on the affected system without your Target=multi-user.target changes?

(Please run the command after reverting your target changes and rebooting)

Comment by Marshall Giguere [ 27/Dec/19 ]

Yes. No issues on 16.04 only 18.04, mongodb 4.2.2

Comment by Danny Hatcher (Inactive) [ 27/Dec/19 ]

Thank you for the report; we'll look into it. To confirm, you are using 4.2.2 on Ubuntu 16.04 with no issues, it's specifically 4.2.2 on Ubuntu 18.04 that you have the problem?

Generated at Thu Feb 08 05:08:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.