[SERVER-38232] Mongod on Ubuntu 16/18 does not create a pid file Created: 22/Nov/18  Updated: 29/Oct/23  Resolved: 10/Feb/23

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 3.4.15, 3.4.18
Fix Version/s: 3.4 Required, 4.4.19, 5.0.15, 7.0.0-rc0, 6.0.5

Type: Bug Priority: Major - P3
Reporter: James Stocker Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
causes SERVER-74345 mongodb-org-server 4.4.19, 5.0.15, 6.... Closed
Related
is related to SERVER-27241 Cannot write pid file to /var/run/mon... Closed
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.0, v5.0, v4.4
Steps To Reproduce:

Install a server with Ubuntu 16 or 18 and install mongodb-org-server 3.4.x from the repo.mongodb.org repos, start it up and it won't create a pid file where it is supposed to according to the systemd config file.

Participants:
Case:
Story Points: 2

 Description   

I was alerted to missing log files on some of our production mongod servers yesterday, the cause was that log rotate was trying to use the pid file to send a signal to mongo to tell it the log files had rotated. But the pid file wasn't where we expected to find it.

Looking in the systemd service file it should be in /var/run/mongodb/mongod.pid but nothing appeared there, even after I created the folder which was missing. Then I added this to my mongod config file:

processManagement:
   pidFilePath: /var/run/mongodb.pid 

Then mongod failed to start up, as it didn't have permission to create. I touched the file and gave mongo permissions and it started, which should fix our issues.

I would say that this should not be fixed by admins, but should be correctly managed by mongod's systemd service file.

I would have expected a warning or error to appear in the logs in this situation, but nothing appears:

2018-11-22T12:25:35.418+0000 I CONTROL  [main] ***** SERVER RESTARTED *****                          
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] MongoDB starting : pid=12211 port=27017 dbpath=/srv/mongodb 64-bit host=some.host.domain.com
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] db version v3.4.15                          
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] git version: 52e5b5fbaa3a2a5b1a217f5e647b5061817475f9
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016  
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] allocator: tcmalloc                          
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] modules: none                               
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] build environment:                           
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604                      
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten]     distarch: x86_64                         
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten]     target_arch: x86_64                      
2018-11-22T12:25:35.427+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { RESTInterfaceEnabled: false, enabled: false } }, replication: { replSetName: "dev" }, storage: { dbPath: "/srv/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { c
ollectionConfig: { blockCompressor: "snappy" }, engineConfig: { journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } }
2018-11-22T12:25:35.454+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,
log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-11-22T12:25:38.052+0000 I STORAGE  [initandlisten] WiredTiger message [1542889538:52406][12211:0x7fbb84eded00], txn-recover: Main recovery loop: starting at 5591/38323200
2018-11-22T12:25:38.174+0000 I STORAGE  [initandlisten] WiredTiger message [1542889538:174123][12211:0x7fbb84eded00], txn-recover: Recovering log 5591 through 5592
2018-11-22T12:25:38.232+0000 I STORAGE  [initandlisten] WiredTiger message [1542889538:232811][12211:0x7fbb84eded00], txn-recover: Recovering log 5592 through 5592
2018-11-22T12:25:55.956+0000 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2018-11-22T12:25:55.956+0000 I STORAGE  [initandlisten] The size storer reports that the oplog contains 39153394 records totaling to 10711736188 bytes
2018-11-22T12:25:55.956+0000 I STORAGE  [initandlisten] Sampling from the oplog between Nov  8 04:49:55:1e and Nov 22 12:25:26:2 to determine where to place markers for truncation
2018-11-22T12:25:55.957+0000 I STORAGE  [initandlisten] Taking 1000 samples and assuming that each section of oplog contains approximately 391418 records totaling to 107085642 bytes

My server here is running 3.4.15 but I tested with 3.14.18 from repo.mongodb.org on my laptop and saw the same issue



 Comments   
Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-38232 Add RuntimeDirectory to mongod.service

(cherry picked from commit 1e5c67d6b9c80eda789fece2e1973a7f7c6cb7b0)
(cherry picked from commit 8d463a6789d68e2fc39b93346152e1b4a626e14a)
(cherry picked from commit 264b8fcfc71376d62ce7a33a4866b1c462569b85)
(cherry picked from commit 75c39e4a6b213ac4a49cb85b5cca932eadc9a56d)
Branch: v4.4
https://github.com/mongodb/mongo/commit/1920823fb8eb199f164ebd4300a911b85f89cb67

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-38232 Add RuntimeDirectory to mongod.service

(cherry picked from commit 1e5c67d6b9c80eda789fece2e1973a7f7c6cb7b0)
(cherry picked from commit 8d463a6789d68e2fc39b93346152e1b4a626e14a)
(cherry picked from commit 264b8fcfc71376d62ce7a33a4866b1c462569b85)
Branch: v5.0
https://github.com/mongodb/mongo/commit/75c39e4a6b213ac4a49cb85b5cca932eadc9a56d

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-38232 Add RuntimeDirectory to mongod.service

(cherry picked from commit 1e5c67d6b9c80eda789fece2e1973a7f7c6cb7b0)
(cherry picked from commit 8d463a6789d68e2fc39b93346152e1b4a626e14a)
Branch: v6.0
https://github.com/mongodb/mongo/commit/264b8fcfc71376d62ce7a33a4866b1c462569b85

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-38232 Add RuntimeDirectory to mongod.service
Branch: master
https://github.com/mongodb/mongo/commit/1e5c67d6b9c80eda789fece2e1973a7f7c6cb7b0

Comment by Ryan Egesdahl (Inactive) [ 08/Feb/23 ]

We're going to backport this change along with a partial backport of SERVER-70121 that included removing type=forking so all of our branches are consistent in their behavior with systemd and PID files. What this means for users is that there will no longer be a PID file available by default on systemd platforms, but you will be able to add pidFilePath to /etc/mongod.conf to create one anyway if you need it for your tooling. SysV Init (those using init scripts in /etc/init.d) and Upstart (which also uses those scripts) will continue to have a PID file available for process tracking.

Note: We are only going to backport to v6.0, v5.0, and v4.4. We have decided against backporting to v4.2 because it's going EOL in a couple of months and we think it's better to not change expectations too much unless it's for a critically important reason.

Comment by Ryan Egesdahl (Inactive) [ 03/Feb/23 ]

We have moved completely over to systemd to manage the service, which makes a PID file superfluous and potentially inaccurate if not configured correctly. It's better to simply ask systemd for PID and service status information:

systemctl show --property MainPID --value mongod.service
systemctl is-active mongod.service

If you absolutely need a PID file for some reason, you can still add the configuration option in mongod.conf to write one (make sure you don't also have fork!) and use systemctl edit mongod.service to add a RuntimeDirectory=mongodb line to make sure the /run directory is created for it. This is not likely to be a common use case, but we can certainly add RuntimeDirectory=mongodb to the default unit file to support it without causing any problems.

Comment by James Stocker [ 29/Nov/18 ]

In 14.04, you didn't need to specify this value, yet the pid file was created, which is not what the documentation suggests should happen.

In 16.04 and 18.04, the pid file is not created automatically without this being set, like the docs suggest. However, even if you set this value, the pid file cannot be created in /var/run because mongodb doesn't have permissions to create that file, you need to create a tmpfile config file yourself otherwise mongodb will fail to start as it cannot create it's pid file

Comment by Eric Sedor [ 28/Nov/18 ]

Thanks a lot James; we've been able to confirm that a pidFilePath value that matches the service config is missing from the debian mongod.conf file in 3.4.18. We think that is the problem, and have marked this issue for scheduling.

You can keep checking this ticket for additional updates.

Comment by James Stocker [ 28/Nov/18 ]

Yes those logs are before I defined it in the config file.

 

But in the systemd.service file installed by the deb, it is defined:

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid 

 

This is a different location than the old default of /var/run/mongodb.pid

Regardless of it being in the systemd.service file the PID is not created.

When we used this on 14.04 we didn't need to define the pid file location in the mongo config, it was created automatically.

Comment by Eric Sedor [ 28/Nov/18 ]

Hi James,

We're not seeing the pidFilePath option showing up in the parsed options log lines:

2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] options: \{ config: "/etc/mongod.conf", net: { http: { RESTInterfaceEnabled: false, enabled: false } }, replication: \{ replSetName: "dev" }, storage: \{ dbPath: "/srv/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: \{ c ollectionConfig: { blockCompressor: "snappy" }, engineConfig: \{ journalCompressor: "snappy" }, indexConfig: \{ prefixCompression: true } } }, systemLog: \{ destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } } 2018-11-22T12:25:35.454+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60, log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress)

Can you please confirm that /etc/mongod.conf contains the option, or let us know if these log lines are from before the pidFilePath option is added?

Thank you!

Comment by James Stocker [ 27/Nov/18 ]

I realised later than /var/run is on a tmpfs, so I had to add this:

$ cat /usr/lib/tmpfiles.d/mongodb.conf
f /var/run/mongodb.pid 0644 mongodb mongodb - - 

 

This file tells the os to create the pid file and assign ownership to mongodb

Generated at Thu Feb 08 04:48:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.