[SERVER-48072] serverStatus very slow only at end Created: 09/May/20 Updated: 27/Oct/23 Resolved: 14/May/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.0.3 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | phoenix Liu | Assignee: | Dmitry Agranat |
| Resolution: | Community Answered | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Participants: | |||||
| Description |
|
hi, I recently encountered a problem about serverStatus. in mongod's log, i can find a log like below with a lot of other slow logs(normal query&update, etc)
Fri May 1 14:01:23.248 I COMMAND [ftdc] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after locks: 0, after logicalSessionRecordCache: 0, after network: 0, after opLatencies: 0, after opcounters: 0, after opcountersRepl: 0, after repl: 0, after shardingStatistics: 0, after storageEngine: 0, after tcmalloc: 0, after transactions: 0, after wiredTiger: 0, at end: 8256 }
in source code, after all sections handled, before print this 'very slow' log,it handles with metrics and warnings. i don't understand why it took so long.
more details:
my question: 1. what does is mean when 'serverStatus' slow only at end? is it reasonable? 2. under what scenarios may cause this problem? 3. how can I locate this problem, is there any recommendation?
Thank you! |
| Comments |
| Comment by phoenix Liu [ 14/May/20 ] | ||||||||||||||||||||||||
|
Hi Dima, Thanks for reply. I will try your suggestion and give a feedback. But still I'm curious how did you come to this conclusion that it's about NUMA conf in machine. Could you please explain it with more details? Here is our machine's NUMA config (before any changes):
And also, i checked all mongod/mongos service, there is no 'numad' daemon at all. We didn't start mongod with 'numactl --interleave=all'. But in this case, memory does not seem to be bottleneck. How can NUMA config affect it?
Thanks, phoenix
| ||||||||||||||||||||||||
| Comment by Dmitry Agranat [ 14/May/20 ] | ||||||||||||||||||||||||
|
Hi phoenixxliu@tencent.com, thank you for providing the requested information. After analyzing the data, this appears to be an environment problem we've seen in the past. To resolve this issue, please do the following:
As this issue is not related to MongoDB, I will go ahead and close this issue. Again, thank you for your report. Regards, | ||||||||||||||||||||||||
| Comment by phoenix Liu [ 11/May/20 ] | ||||||||||||||||||||||||
|
Hi Dima, Sorry that i missed some info in my reply. It will only cause very short system freezes(only a few seconds) and application will be affected and get 'timeout' error.
from mongostat result, we can see these:
BTW, The time zone is GMT+8. And the newest one happened in Mon May 11 00:43:42(GMT +8), you may check this first. Thanks, Phoenix | ||||||||||||||||||||||||
| Comment by Dmitry Agranat [ 11/May/20 ] | ||||||||||||||||||||||||
|
Thanks phoenixxliu@tencent.com, we will take a look at the uploaded data. Could you please clarify if this is causing any observable performance problems? Thanks, | ||||||||||||||||||||||||
| Comment by phoenix Liu [ 11/May/20 ] | ||||||||||||||||||||||||
|
Hi Dima, Thanks for the reply. I have uploaded the relevant files as required, Considering that /diagnostic.data on May 1 is no longer available any more, you may focus on the following points in time: Sun May 10 10:25:34/Sun May 10 06:50:25 Thanks, Phoenix | ||||||||||||||||||||||||
| Comment by Dmitry Agranat [ 10/May/20 ] | ||||||||||||||||||||||||
|
Thank you for the report. Could you please clarify is this is causing any observable performance problems and provide the exact time & timestamp when this issue occurs? Would you also archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Thanks, |