[SERVER-36991] mongo startup by parameter --syslog: mongod 100% cpu usage under big data insert Created: 05/Sep/18 Updated: 20/Sep/18 Resolved: 20/Sep/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging |
| Affects Version/s: | 3.4.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | wynn zhao | Assignee: | Nick Brewer |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | 1. MongoDB server version: 3.4.2 2. startup mongo with parameter --syslog /usr/local/mongodb/bin/mongod --fork --dbpath=/data/db --syslog --port=27017 --bind_ip=127.0.0.1,136.227.20.70 3. top -H -p PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4. thread of high cpu usage with common output by "pstack 29078" Thread 1 (process 29078):
5. strace -T -tt -e trace=all -s 10000 -fp 29078 [pid 29078] 13:49:53.945831 futex(0x55e854e411a8, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000017>
6. big size data insert mongodb: mongostat insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn time |
| Participants: |
| Description |
|
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
| Comments |
| Comment by Nick Brewer [ 20/Sep/18 ] | ||||||
|
wynn Happy to help. As an aside, I suggest upgrading to the latest MongoDB 3.4 version - currently 3.4.17, as it contains numerous improvements and bug fixes. -Nick | ||||||
| Comment by wynn zhao [ 19/Sep/18 ] | ||||||
|
Hi Nick You can close this issue, if we reproduce it with suitable testing environment, we can reopen it. Thanks for your timely and seriously support. Have a nice day!
Best Regards! Wynn | ||||||
| Comment by Nick Brewer [ 13/Sep/18 ] | ||||||
|
wynn I've attempted to reproduce this, but I haven't been able to. Using MongoDB 3.4.2 on CentOS 7, I started mongod up first with - --syslog:
--logpath:
Do you have any other suggestions as to how I might be able to reproduce what you're seeing? If not, I believe we can close this ticket, as it does not appear to represent an underlying bug in MongoDB. Thanks, | ||||||
| Comment by wynn zhao [ 13/Sep/18 ] | ||||||
| Comment by wynn zhao [ 13/Sep/18 ] | ||||||
|
Hi Nick We have same configuration with MongoDB and Servers.
Best Regards! Wynn | ||||||
| Comment by Nick Brewer [ 12/Sep/18 ] | ||||||
|
wynn I don't see anything unusual there, but this reinforces my belief that this issue is not with the --syslog setting itself, as you should be seeing roughly the same behavior on badash-linux-5 as you are on badash-linux-6. Are there any other differences (in terms of configuration, hardware, etc) between these two machines? It would be useful to see the output of db.getLogComponents() from both machines. Thanks, | ||||||
| Comment by wynn zhao [ 12/Sep/18 ] | ||||||
|
Hi Nick rsyslog.conf attached. And the server version as follows: [####@badash-linux-6 etc]$ cat /etc/os-release CENTOS_MANTISBT_PROJECT="CentOS-7"
Best Regards! Wynn
| ||||||
| Comment by Nick Brewer [ 11/Sep/18 ] | ||||||
|
wynn Have you made any modifications to your system's syslog configuration? It would be helpful to see your rsyslog.conf file for each of the machines you've provided diagnostic data for. It's worth noting that only one of the machines are seeing increased CPU utilization, despite the fact that they're both using the --syslog option with similar workloads. -Nick | ||||||
| Comment by wynn zhao [ 06/Sep/18 ] | ||||||
|
Hi Nick, We just use one database include several collections with multi-thread writing and reading(c++ driver write or read, php driver read mostly). When this core cpu reach 100%, all other operation will be very slowly or hang. For example, read data from collections by PHP driver, some times, no data will be got or "no such server" errors reported.
In addition, we modify the start up parameter, remove --syslog and instead by --logpath /dev/null. Finally, after one day test, the cpu is very low and the cpu high issue not be reproduced, every operation looks good currently.
Best Regards! Wynn
| ||||||
| Comment by Nick Brewer [ 06/Sep/18 ] | ||||||
|
wynn Looking at the data provided, it doesn't appear that there's substantial CPU used before or after the reboot (where the --syslog option was presumably added): I believe the disparity here is due to the fact that, by default, top measures CPU usage percentage as 100% per core, meaning that multi-core systems can easily exceed 100% usage. So this line:
Indicates that mongod is utilizing one core on this 8-core system. -Nick | ||||||
| Comment by wynn zhao [ 06/Sep/18 ] | ||||||
|
Hi Nick, diagnostic.data has been attached. Best Regards! Wynn | ||||||
| Comment by Nick Brewer [ 05/Sep/18 ] | ||||||
|
wynn Could you please archive (tar or zip) and upload the dbpath/diagnostic.data directory to this ticket? Thanks, |