[SERVER-47198] SIGUSR1 can cause MongoDB process crash Created: 31/Mar/20 Updated: 29/Oct/23 Resolved: 29/Jun/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.6.17 |
| Fix Version/s: | 5.1.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tommy Lee | Assignee: | Billy Donahue |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Requested: |
v5.0, v4.4, v3.6
|
||||||||||||
| Steps To Reproduce: |
|
||||||||||||
| Sprint: | Service Arch 2021-06-28, Service Arch 2021-07-12 | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
After deleted the MongoDB log file then send the signal USR1 with kill command, then mongo process will crash Mongo Version: 3.6.17 Platform: CentOS 7 64bit kill -SIGUSR1 <mongod process id> |
| Comments |
| Comment by Githook User [ 28/Jun/21 ] | ||||
|
Author: {'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}Message: | ||||
| Comment by Githook User [ 28/Jun/21 ] | ||||
|
Author: {'name': 'Billy Donahue', 'email': 'billy.donahue@gmail.com', 'username': 'BillyDonahue'}Message: | ||||
| Comment by Billy Donahue [ 14/Jun/21 ] | ||||
|
Sara, I can send you a CR off of my | ||||
| Comment by Billy Donahue [ 22/Sep/20 ] | ||||
|
My recommendation for what it's worth would be to ignore a "not found" error returned by the rename part of the log rotation. If the boost::filesystem::rename step comes back with errc::no_such_file_or_directory, we can calmly assume the old log file was moved aside by an administrative procedure. It happens. We can maybe log a warning about it. We can continue, opening a new log file as we would have done anyway. I think server-47198.diff | ||||
| Comment by Billy Donahue [ 22/Sep/20 ] | ||||
|
Reproduced with master branch code right now. The failure is logged to the moved-away old log file. (killed.log
Core dump shows the problem is that we fassert that logv2::rotateLogs succeeds.
Log shows:
So while that's weird and disappointing, I don't think the server needs to CRASH from it. | ||||
| Comment by Tommy Lee [ 03/Apr/20 ] | ||||
|
We used to use the following command for log rotating kill -SIGUSR1 <mongod process id> Now I changed to MongoDB native log rotate command mongo --port 10001 -u admin -p "`awk -F'"' '{print $6}' /root/.mongorc.js`" --eval "db.adminCommand( { logRotate : 1 } );"
Using MongoDB native log rotate command is more safe, if log rotate fail, the process still there. For SIGUSR1 will cause the process crash.
| ||||
| Comment by Tommy Lee [ 03/Apr/20 ] | ||||
|
Hi Carl, very appreciate to hear your response, if need me provide more information please let me know. We are a huge online game company, using MongoDB a lot, lucky caught the bug in staging environment. And we using the same version in the production environment. That's a potential risk.
Thanks
Regards, Tommy | ||||
| Comment by Carl Champain (Inactive) [ 02/Apr/20 ] | ||||
|
Thank you for the report. Kind regards, |