Details
Description
I was alerted to missing log files on some of our production mongod servers yesterday, the cause was that log rotate was trying to use the pid file to send a signal to mongo to tell it the log files had rotated. But the pid file wasn't where we expected to find it.
Looking in the systemd service file it should be in /var/run/mongodb/mongod.pid but nothing appeared there, even after I created the folder which was missing. Then I added this to my mongod config file:
processManagement:
|
pidFilePath: /var/run/mongodb.pid
|
Then mongod failed to start up, as it didn't have permission to create. I touched the file and gave mongo permissions and it started, which should fix our issues.
I would say that this should not be fixed by admins, but should be correctly managed by mongod's systemd service file.
I would have expected a warning or error to appear in the logs in this situation, but nothing appears:
2018-11-22T12:25:35.418+0000 I CONTROL [main] ***** SERVER RESTARTED ***** |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] MongoDB starting : pid=12211 port=27017 dbpath=/srv/mongodb 64-bit host=some.host.domain.com |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] db version v3.4.15 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] git version: 52e5b5fbaa3a2a5b1a217f5e647b5061817475f9 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] allocator: tcmalloc |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] modules: none |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] build environment: |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] distmod: ubuntu1604 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] distarch: x86_64 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] target_arch: x86_64 |
2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { RESTInterfaceEnabled: false, enabled: false } }, replication: { replSetName: "dev" }, storage: { dbPath: "/srv/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { c |
ollectionConfig: { blockCompressor: "snappy" }, engineConfig: { journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } } |
2018-11-22T12:25:35.454+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60, |
log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress), |
2018-11-22T12:25:38.052+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:52406][12211:0x7fbb84eded00], txn-recover: Main recovery loop: starting at 5591/38323200 |
2018-11-22T12:25:38.174+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:174123][12211:0x7fbb84eded00], txn-recover: Recovering log 5591 through 5592 |
2018-11-22T12:25:38.232+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:232811][12211:0x7fbb84eded00], txn-recover: Recovering log 5592 through 5592 |
2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs |
2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 39153394 records totaling to 10711736188 bytes |
2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] Sampling from the oplog between Nov 8 04:49:55:1e and Nov 22 12:25:26:2 to determine where to place markers for truncation |
2018-11-22T12:25:55.957+0000 I STORAGE [initandlisten] Taking 1000 samples and assuming that each section of oplog contains approximately 391418 records totaling to 107085642 bytes |
My server here is running 3.4.15 but I tested with 3.14.18 from repo.mongodb.org on my laptop and saw the same issue
Attachments
Issue Links
- causes
-
SERVER-74345 mongodb-org-server 4.4.19, 5.0.15, 6.0.5 not starting after upgrading from older version (Debian, RPM Packages)
-
- Closed
-
- is related to
-
SERVER-27241 Cannot write pid file to /var/run/mongodb/mongod.pid: No such file or directory
-
- Closed
-