<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:02:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-65162] Improve FTDC performance to avoid stalls</title>
                <link>https://jira.mongodb.org/browse/SERVER-65162</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While working on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-64503&quot; title=&quot;Add workload with 10k threads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-64503&quot;&gt;&lt;del&gt;SERVER-64503&lt;/del&gt;&lt;/a&gt; we discovered that we can&apos;t really see what&apos;s going on in the server during extreme cases of contention. Which would prove to be greatly valuable information in order to debug what the issues are.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/369944/369944_Screenshot+2022-04-01+at+14.08.37.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;During the test run we saw multiple entries in the mongod log that seem to point to the ftdc thread performing really poorly. &lt;b&gt;We&apos;ve attempted to set its thread scheduling priority to use&lt;/b&gt; &lt;tt&gt;SCHED_FIFO&lt;/tt&gt; &lt;b&gt;and the priority to the max value but that didn&apos;t seem to fix the issue&lt;/b&gt;.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:00:43.832+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:5,&quot;after locks&quot;:3303,&quot;after logicalSessionRecordCache&quot;:3303,&quot;after mirroredReads&quot;:3303,&quot;after network&quot;:3303,&quot;after opLatencies&quot;:3303,&quot;after opcounters&quot;:3303,&quot;after opcountersRepl&quot;:3303,&quot;after oplogTruncation&quot;:3303,&quot;after readConcernCounters&quot;:3303,&quot;after repl&quot;:3303,&quot;after scramCache&quot;:3303,&quot;after security&quot;:3303,&quot;after storageEngine&quot;:3303,&quot;after tcmalloc&quot;:3307,&quot;after tenantMigrations&quot;:3307,&quot;after trafficRecording&quot;:3307,&quot;after transactions&quot;:3307,&quot;after transportSecurity&quot;:3307,&quot;after twoPhaseCommitCoordinator&quot;:3307,&quot;after wiredTiger&quot;:3307,&quot;at end&quot;:7832}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:04:06.873+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:25,&quot;after locks&quot;:25,&quot;after logicalSessionRecordCache&quot;:25,&quot;after mirroredReads&quot;:25,&quot;after network&quot;:25,&quot;after opLatencies&quot;:25,&quot;after opcounters&quot;:25,&quot;after opcountersRepl&quot;:25,&quot;after oplogTruncation&quot;:25,&quot;after readConcernCounters&quot;:25,&quot;after repl&quot;:25,&quot;after scramCache&quot;:25,&quot;after security&quot;:25,&quot;after storageEngine&quot;:25,&quot;after tcmalloc&quot;:30,&quot;after tenantMigrations&quot;:30,&quot;after trafficRecording&quot;:30,&quot;after transactions&quot;:30,&quot;after transportSecurity&quot;:30,&quot;after twoPhaseCommitCoordinator&quot;:30,&quot;after wiredTiger&quot;:5303,&quot;at end&quot;:8375}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:04:26.719+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:4,&quot;after locks&quot;:4,&quot;after logicalSessionRecordCache&quot;:4,&quot;after mirroredReads&quot;:4,&quot;after network&quot;:4,&quot;after opLatencies&quot;:6,&quot;after opcounters&quot;:6,&quot;after opcountersRepl&quot;:6,&quot;after oplogTruncation&quot;:6,&quot;after readConcernCounters&quot;:6,&quot;after repl&quot;:6,&quot;after scramCache&quot;:6,&quot;after security&quot;:6,&quot;after storageEngine&quot;:6,&quot;after tcmalloc&quot;:4376,&quot;after tenantMigrations&quot;:4376,&quot;after trafficRecording&quot;:4376,&quot;after transactions&quot;:4376,&quot;after transportSecurity&quot;:4376,&quot;after twoPhaseCommitCoordinator&quot;:4376,&quot;after wiredTiger&quot;:4377,&quot;at end&quot;:8792}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:04:50.684+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:12,&quot;after locks&quot;:12,&quot;after logicalSessionRecordCache&quot;:12,&quot;after mirroredReads&quot;:12,&quot;after network&quot;:12,&quot;after opLatencies&quot;:12,&quot;after opcounters&quot;:12,&quot;after opcountersRepl&quot;:12,&quot;after oplogTruncation&quot;:12,&quot;after readConcernCounters&quot;:12,&quot;after repl&quot;:4881,&quot;after scramCache&quot;:4881,&quot;after security&quot;:4881,&quot;after storageEngine&quot;:4881,&quot;after tcmalloc&quot;:7479,&quot;after tenantMigrations&quot;:7479,&quot;after trafficRecording&quot;:7479,&quot;after transactions&quot;:7479,&quot;after transportSecurity&quot;:7479,&quot;after twoPhaseCommitCoordinator&quot;:7479,&quot;after wiredTiger&quot;:7480,&quot;at end&quot;:7484}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:07:31.799+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:15,&quot;after locks&quot;:15,&quot;after logicalSessionRecordCache&quot;:913,&quot;after mirroredReads&quot;:913,&quot;after network&quot;:913,&quot;after opLatencies&quot;:913,&quot;after opcounters&quot;:913,&quot;after opcountersRepl&quot;:913,&quot;after oplogTruncation&quot;:913,&quot;after readConcernCounters&quot;:913,&quot;after repl&quot;:913,&quot;after scramCache&quot;:913,&quot;after security&quot;:913,&quot;after storageEngine&quot;:913,&quot;after tcmalloc&quot;:920,&quot;after tenantMigrations&quot;:920,&quot;after trafficRecording&quot;:920,&quot;after transactions&quot;:920,&quot;after transportSecurity&quot;:920,&quot;after twoPhaseCommitCoordinator&quot;:920,&quot;after wiredTiger&quot;:920,&quot;at end&quot;:7799}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:08:10.551+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:3,&quot;after flowControl&quot;:3,&quot;after freeMonitoring&quot;:3,&quot;after globalLock&quot;:6,&quot;after locks&quot;:6,&quot;after logicalSessionRecordCache&quot;:6,&quot;after mirroredReads&quot;:6,&quot;after network&quot;:6,&quot;after opLatencies&quot;:6,&quot;after opcounters&quot;:6,&quot;after opcountersRepl&quot;:6,&quot;after oplogTruncation&quot;:2456,&quot;after readConcernCounters&quot;:2456,&quot;after repl&quot;:2456,&quot;after scramCache&quot;:2456,&quot;after security&quot;:2456,&quot;after storageEngine&quot;:2456,&quot;after tcmalloc&quot;:2462,&quot;after tenantMigrations&quot;:2462,&quot;after trafficRecording&quot;:2462,&quot;after transactions&quot;:2462,&quot;after transportSecurity&quot;:2462,&quot;after twoPhaseCommitCoordinator&quot;:2462,&quot;after wiredTiger&quot;:7666,&quot;at end&quot;:10408}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2022-04-01T10:09:36.029+00:00 I  COMMAND  20499   [ftdc] &quot;serverStatus was very slow&quot;,&quot;attr&quot;:{&quot;timeStats&quot;:{&quot;after basic&quot;:0,&quot;after activeIndexBuilds&quot;:0,&quot;after asserts&quot;:0,&quot;after bucketCatalog&quot;:0,&quot;after catalogStats&quot;:0,&quot;after connections&quot;:0,&quot;after electionMetrics&quot;:0,&quot;after extra_info&quot;:2,&quot;after flowControl&quot;:2,&quot;after freeMonitoring&quot;:2,&quot;after globalLock&quot;:334,&quot;after locks&quot;:334,&quot;after logicalSessionRecordCache&quot;:7857,&quot;after mirroredReads&quot;:7857,&quot;after network&quot;:7857,&quot;after opLatencies&quot;:7857,&quot;after opcounters&quot;:7857,&quot;after opcountersRepl&quot;:7857,&quot;after oplogTruncation&quot;:7857,&quot;after readConcernCounters&quot;:7857,&quot;after repl&quot;:7857,&quot;after scramCache&quot;:7857,&quot;after security&quot;:7857,&quot;after storageEngine&quot;:7857,&quot;after tcmalloc&quot;:8195,&quot;after tenantMigrations&quot;:8195,&quot;after trafficRecording&quot;:8195,&quot;after transactions&quot;:8195,&quot;after transportSecurity&quot;:8195,&quot;after twoPhaseCommitCoordinator&quot;:8195,&quot;after wiredTiger&quot;:8196,&quot;at end&quot;:11029}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Note that the numbers are milliseconds. The log seems to indicate that the following points in the serverStatus command are bottlenecks and could do with some performance improvements:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;logicalSessionRecordCache&lt;/li&gt;
	&lt;li&gt;locks&lt;/li&gt;
	&lt;li&gt;tcmalloc&lt;/li&gt;
	&lt;li&gt;repl&lt;/li&gt;
	&lt;li&gt;oplogTruncation&lt;/li&gt;
	&lt;li&gt;wiredTiger&lt;/li&gt;
	&lt;li&gt;at end&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Additionally, after performing some preliminary searches it seems that the Linux system metrics collector is also a bottleneck at points due to the multiple syscalls needed to get the necessary information from the &lt;tt&gt;/proc&lt;/tt&gt; filesystem.&lt;/p&gt;

&lt;p&gt;Attached to the ticket is a flamegraph of a perf recording of the ftdc thread during the benchmark run. The &lt;tt&gt;read()&lt;/tt&gt; syscall can be seen as taking a relatively decent chunk of the processing time.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2014110">SERVER-65162</key>
            <summary>Improve FTDC performance to avoid stalls</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-security">Backlog - Security Team</assignee>
                                    <reporter username="jordi.olivares-provencio@mongodb.com">Jordi Olivares Provencio</reporter>
                        <labels>
                    </labels>
                <created>Fri, 1 Apr 2022 12:42:37 +0000</created>
                <updated>Tue, 7 Nov 2023 18:25:46 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="4926635" author="connie.chen" created="Tue, 25 Oct 2022 19:00:53 +0000"  >&lt;p&gt;Backlog Grooming Notes: Move this into a project and re-evaluate as a project-sized effort&lt;/p&gt;</comment>
                            <comment id="4460864" author="JIRAUSER1264163" created="Wed, 6 Apr 2022 13:11:51 +0000"  >&lt;p&gt;At that scale it is indeed impractical to work with all threads. I did some manual gdb samples (like 5 or so) and they pointed to the&#160;&lt;tt&gt;/proc&lt;/tt&gt; reads and tcmalloc. I&apos;ll try to automate this process with gdb in order to better gauge where the bottlenecks might be.&lt;/p&gt;</comment>
                            <comment id="4460682" author="bruce.lucas@10gen.com" created="Wed, 6 Apr 2022 11:57:01 +0000"  >&lt;p&gt;As mentioned, I suspect that a lot of the delay in FTDC may be due to mutex contention, not CPU contention, and therefore won&apos;t show up in the perf data, so if we only tackle the hot spots seen in perf data we may not make any improvement. The observation that increasing thread priority didn&apos;t help somewhat supports that.&lt;/p&gt;

&lt;p&gt;Because there are a large number of threads, using the normal gdb approach of &quot;bt thread apply all&quot; to get stack trace samples to see both on- and off-cpu time won&apos;t be practical. It&apos;s possible that SIGUSR2 would work although that may also be problematic because of the number of threads.&lt;/p&gt;

&lt;p&gt;However it may be possible to get useful stack trace samples by using gdb to sample just the ftdc thread, i.e. &quot;bt thread apply &lt;span class=&quot;error&quot;&gt;&amp;#91;ftdcthread&amp;#93;&lt;/span&gt;&quot;. I think it might be worth a try to get samples this way, maybe 100 samples at intervals of a couple of seconds, to see what bottlenecks are impeding the ftdc thread, in order to decide what bottlenecks need to be tackled.&lt;/p&gt;</comment>
                            <comment id="4450726" author="bruce.lucas@10gen.com" created="Fri, 1 Apr 2022 15:52:20 +0000"  >&lt;p&gt;I think much of this might be contending with other threads for access to data structures guarded by mutexes, and if so would not show up in the perf data.&lt;/p&gt;

&lt;p&gt;Also note that while the charts show with blank spaces for missing FTDC samples, the values computed still generally reflect correct averages over the intervals between the samples that are collected since generally FTDC metrics are cumulative numbers. You can often get a better visual representation by reducing the number of display samples in the t2 options menu.&lt;/p&gt;</comment>
                            <comment id="4450232" author="JIRAUSER1264163" created="Fri, 1 Apr 2022 13:30:53 +0000"  >&lt;p&gt;tcmalloc in particular seems to repeteadly call the same methods &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4c7f60204736dea74792bcd85f4da816a2fa94cc/src/mongo/util/tcmalloc_server_status_section.cpp#L89&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;over and over with different arguments&lt;/a&gt; and the underlying implementation seems to be &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4c7f60204736dea74792bcd85f4da816a2fa94cc/src/third_party/gperftools/dist/src/tcmalloc.cc#L700&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;incredibly inefficient about it&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4450209" author="louis.williams" created="Fri, 1 Apr 2022 13:22:40 +0000"  >&lt;p&gt;As far as the &quot;globalLock&quot; section goes, it is required to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a4b2452ab7442ade203ca3c778c71cf6ab15cf61/src/mongo/db/stats/lock_server_status_section.cpp#L58-L66&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;iterate through the Lockers of all active Clients&lt;/a&gt;, so that would explain the poor performance at high thread levels.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2329111">SERVER-76723</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="369944" name="Screenshot 2022-04-01 at 14.08.37.png" size="33077" author="jordi.olivares-provencio@mongodb.com" created="Fri, 1 Apr 2022 12:10:48 +0000"/>
                            <attachment id="369943" name="test.svg" size="70705" author="jordi.olivares-provencio@mongodb.com" created="Fri, 1 Apr 2022 12:13:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25129"><![CDATA[Server Security]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 1 Apr 2022 13:22:40 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 15 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2763</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 15 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-security</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>connie.chen@mongodb.com</customfieldvalue>
            <customfieldvalue>jordi.olivares-provencio@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0pef3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0jf9z:cc</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0p0kf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>