<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:18:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-71213] High variance in PriorityTicketHolder microbenchmark runs</title>
                <link>https://jira.mongodb.org/browse/SERVER-71213</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, we are seeing a case where the ticketholder_bm benchmark performance of the PriorityTicketHolder degrades significantly on outlier runs. In general, the PriorityTicketHolder microbenchmarks perform comparable to that of the SemaphoreTicketHolder, consistently surpassing the SemaphoreTicketHolder performance with benchmark runs of 1024 threads.&lt;/p&gt;

&lt;p&gt;After performing some analysis, we discovered that outlier runs with poor performance were spending a significant amount of time (~29% compared to ~2% in standard runs) in _pthread_rwlock_rdlock.&#160;&lt;/p&gt;

&lt;p&gt;We suspect this could be due to the use of the shared_mutex, and sub-optimal waits for shared readers.&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2179514">SERVER-71213</key>
            <summary>High variance in PriorityTicketHolder microbenchmark runs</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="jordi.olivares-provencio@mongodb.com">Jordi Olivares Provencio</assignee>
                                    <reporter username="haley.connelly@mongodb.com">Haley Connelly</reporter>
                        <labels>
                    </labels>
                <created>Wed, 9 Nov 2022 17:41:19 +0000</created>
                <updated>Fri, 27 Oct 2023 13:51:50 +0000</updated>
                            <resolved>Fri, 11 Nov 2022 16:59:44 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4987439" author="haley.connelly" created="Wed, 16 Nov 2022 17:38:56 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gomezferro%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gomezferro@mongodb.com&quot;&gt;daniel.gomezferro@mongodb.com&lt;/a&gt; you make a great point, and thanks for meeting offline to investigate this further.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Conclusion&lt;/b&gt;&lt;br/&gt;
The microbenchmarks are not indicative of the actual PriorityTicketHolder performance in a highly contentious workload.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Recommendation&lt;/b&gt;&lt;br/&gt;
We should try to fix the microbenchmark to be more representative or re-evaluate the impact of how frequently operations &quot;skip the line&quot; in the PriorityTicketHolder.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Analysis&lt;/b&gt;&lt;br/&gt;
We compared the number normal operations &quot;addedToQueue&quot; versus &quot;startedProcessing&quot; to get an idea of how many operations queue in the 1024k benchmark workloads (run locally on a  m6i.2xlarge instance).&lt;br/&gt;
&lt;ins&gt;Average queued:processed ratios in benchmarks&lt;/ins&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;1 : 2 in the SemaphoreTicketHolder benchmark&lt;/li&gt;
	&lt;li&gt;1 : 500 in the PriorityTicketHolder benchmark&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Additionally, we had performance data from the &lt;a href=&quot;https://github.com/mongodb/genny/blob/9fab05fdaef3f58711e00344abdafa31a8fa2c85/src/workloads/execution/BackgroundIndexConstruction.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;background_index_construction&lt;/a&gt; genny workload which aims to test index build performance when there is write contention. To make the comparison accurate, we compared the SemaphoreTicketHolder performance to the PriorityTicketHolder&apos;s performance when index builds are normal priority by default (only normal priority operations were run in the test).&lt;br/&gt;
&lt;ins&gt;Average queued:processed ratios in contentious performance workload&lt;/ins&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;1 : 2 in the SempahoreTicketHolder run&lt;/li&gt;
	&lt;li&gt;1 : ~3.5 in the PriorityTicketHolder run&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Theory&lt;/b&gt;&lt;br/&gt;
We are already aware that incoming operations may grab the next available ticket, &quot;skip the line&quot;, despite there being operations queued (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70391&quot; title=&quot;Address line skipping in PriorityTicketHolder&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70391&quot;&gt;&lt;del&gt;SERVER-70391&lt;/del&gt;&lt;/a&gt;). We believe the microbenchmarks today increase the likelihood of &quot;line skipping&quot; since threads who release a ticket immediately go back to reacquire. &lt;/p&gt;</comment>
                            <comment id="4974815" author="JIRAUSER1264163" created="Fri, 11 Nov 2022 16:58:38 +0000"  >&lt;p&gt;That very well might be the case. Having a bit of contention with the locks might explain it. It would align with the pthread reader lock jumping dramatically in time spent there.&lt;/p&gt;</comment>
                            <comment id="4974301" author="JIRAUSER1263153" created="Fri, 11 Nov 2022 15:00:26 +0000"  >&lt;p&gt;Maybe you don&apos;t even need to get enqueued, it could just be a case of taking the lock almost always un contended VS having some contention, and it appears there&apos;s pretty low contention overall with the low number of enqueues.&lt;/p&gt;</comment>
                            <comment id="4970958" author="JIRAUSER1264163" created="Thu, 10 Nov 2022 16:37:14 +0000"  >&lt;p&gt;This is interesting, but doesn&apos;t seem to be the case unfortunately:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:16 &#160; &#160; &#160; &#160; &#160;45.8 ns &#160; &#160; &#160; &#160; &#160;662 ns &#160; &#160; &#160;1038752 Acquired=1.51068M/s AcquiredPerThread=94.4176k/s Enqueued=0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:128 &#160; &#160; &#160; &#160; 6.53 ns &#160; &#160; &#160; &#160; &#160;632 ns &#160; &#160; &#160;1128832 Acquired=1.58339M/s AcquiredPerThread=12.3703k/s Enqueued=0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:1024 &#160; &#160; &#160; &#160;1646 ns &#160; &#160; &#160; &#160;32513 ns &#160; &#160; &#160; 102400 Acquired=30.7574k/s AcquiredPerThread=30.0365/s Enqueued=210 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-------&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:16 &#160; &#160; &#160; &#160; &#160;46.3 ns &#160; &#160; &#160; &#160; &#160;672 ns &#160; &#160; &#160;1036112 Acquired=1.48738M/s AcquiredPerThread=92.9611k/s Enqueued=0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:128 &#160; &#160; &#160; &#160; 5.89 ns &#160; &#160; &#160; &#160; &#160;625 ns &#160; &#160; &#160;1106560 Acquired=1.59926M/s AcquiredPerThread=12.4942k/s Enqueued=0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;BM_acquireAndRelease&amp;lt;PriorityTicketHolder&amp;gt;/threads:1024 &#160; &#160; &#160; &#160; 183 ns &#160; &#160; &#160; &#160; &#160;990 ns &#160; &#160; &#160; 535552 Acquired=1009.64k/s AcquiredPerThread=985.979/s Enqueued=307&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;I obtained the number for Enqueued from the ticketholder statistics for normal queue which is where all operations are going. It raises a very interesting question though, which is why are we enqueueing so little in the benchmark&lt;/p&gt;</comment>
                            <comment id="4970003" author="JIRAUSER1263153" created="Thu, 10 Nov 2022 12:50:44 +0000"  >&lt;p&gt;Are you monitoring how many times you ended up in the queues? A problem you might be running into is that for fast cases you are (almost) never queueing (a thread takes the ticket, sleeps, releases and reacquires without contention) vs slow cases were you end up queueing and dequeuing a lot.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2189506">SERVER-71476</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 10 Nov 2022 12:50:44 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 12 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2935</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 12 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gomezferro@mongodb.com</customfieldvalue>
            <customfieldvalue>haley.connelly@mongodb.com</customfieldvalue>
            <customfieldvalue>jordi.olivares-provencio@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1hk7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i106gg:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="6492">Execution Team 2022-11-14</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1h6cv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>