[SERVER-45237] After an upgrade mongod will not start on reboot Created: 18/Dec/19 Updated: 27/Jan/20 Resolved: 27/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Upgrade/Downgrade |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Marshall Giguere | Assignee: | Mathew Robinson (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | on Ubunt 18.04 change file: /lib/systemd/system/mongod.server to Reboot, mongod now starts. |
| Sprint: | Dev Platform 2020-01-13, Dev Platform 2020-01-27 |
| Participants: |
| Description |
|
After a mongodb package update mongod fails to start on system reboot. After a reboot mongod tries to start then bails out with and error "code 48". See attached log file excerpt. I should note this only happens on update on 18.04. I have no similar issue on 16.04. The problem seems to be with the systemd config. If I add the additional target"multi-user.target" to the "After=network.target" mongod starts on reboot. I have looked at other incidents similar and the all relate to WiredTiger IIRC. It's a minor but annoying problem, especially if I forget to modify the service file after and upgrade. Mongodb version: 4.2.2 |
| Comments |
| Comment by April Schoffer [ 27/Jan/20 ] |
|
Hi marsh.giguere@gmail.com despite repeated attempts, we've been unable to reproduce this issue. We suspect this issue is likely specific to your environment. Please feel free to reach out with more details if there is anything else you believe we can do to help. |
| Comment by Marshall Giguere [ 06/Jan/20 ] |
|
Matt; I don't recall doing anything unusual other than installing 4.2.2. Here's the analysis reports mongod.service @4.790s `-systemd-fsck@dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.service `-dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.device network.target @4.788s `-systemd-fsck@dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.service `-dev-disk-by\x2duuid-679e4bfc\x2d4f42\x2d4788\x2da019\x2d3f131942a1c7.device On Mon, Jan 6, 2020 at 9:23 AM Mathew Robinson (Jira) <jira@mongodb.org> – |
| Comment by Mathew Robinson (Inactive) [ 06/Jan/20 ] |
|
Thanks, that's close enough to our stock SystemD service file for our purposes. I'll wait for the systemd-analyze results, I'm not able to repro this on my server so I'm suspicious that a dependency cycle has somehow been introduced in your init order. |
| Comment by Marshall Giguere [ 06/Jan/20 ] |
|
Matt; I'm not 100% sure this is the original from the deb. I think I only -Marsh file: /lib/systemd/system/mongod.service [Unit] [Service]
[Install] On Sun, Jan 5, 2020 at 6:20 PM Mathew Robinson (Jira) <jira@mongodb.org> – |
| Comment by Mathew Robinson (Inactive) [ 05/Jan/20 ] |
|
Just for confirmation can you post the service file which produces this error as well? It looks like MongoDB is indeed trying to start before the network is available despite indicating After=network.target Can you run systemd-analyze critical-chain mongod.service to show the dependency ordering? Might as well check systemd-analyze critical-chain network.target as well. |
| Comment by Marshall Giguere [ 04/Jan/20 ] |
Jan 04 10:53:59 xxxxx.xxxxx.xxx systemd[1]: Started MongoDB Database Server. On Thu, Jan 2, 2020 at 2:00 PM Mathew Robinson (Jira) <jira@mongodb.org> – |
| Comment by Mathew Robinson (Inactive) [ 02/Jan/20 ] |
|
Can you post the output of systemctl status mongod on the affected system without your Target=multi-user.target changes? (Please run the command after reverting your target changes and rebooting) |
| Comment by Marshall Giguere [ 27/Dec/19 ] |
|
Yes. No issues on 16.04 only 18.04, mongodb 4.2.2 |
| Comment by Danny Hatcher (Inactive) [ 27/Dec/19 ] |
|
Thank you for the report; we'll look into it. To confirm, you are using 4.2.2 on Ubuntu 16.04 with no issues, it's specifically 4.2.2 on Ubuntu 18.04 that you have the problem? |