<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:10:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-46051] Change stream sys-perf regression due to SERVER-45691</title>
                <link>https://jira.mongodb.org/browse/SERVER-46051</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While running a performance test for the fix to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt;, I discovered that a far more significant regression had been introduced by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-45691&quot; title=&quot;Change Mutex::LockListeners to use a std::vector again&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-45691&quot;&gt;&lt;del&gt;SERVER-45691&lt;/del&gt;&lt;/a&gt;, which changed &lt;tt&gt;Mutex::LockListeners&lt;/tt&gt; to use a vector.&lt;/p&gt;

&lt;p&gt;Attached is a graph of the regression; the Evergreen task &lt;a href=&quot;https://evergreen.mongodb.com/task/sys_perf_linux_shard_lite_change_streams_latency_539cbefbf49451e2fcdc1a54e2b1810ee6160d50_20_01_22_07_21_43&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;can be found here&lt;/a&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1141158">SERVER-46051</key>
            <summary>Change stream sys-perf regression due to SERVER-45691</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="bernard.gorman@mongodb.com">Bernard Gorman</assignee>
                                    <reporter username="bernard.gorman@mongodb.com">Bernard Gorman</reporter>
                        <labels>
                            <label>KP42</label>
                            <label>qexec-team</label>
                    </labels>
                <created>Sat, 8 Feb 2020 23:12:59 +0000</created>
                <updated>Fri, 27 Oct 2023 20:42:22 +0000</updated>
                            <resolved>Tue, 14 Apr 2020 16:01:01 +0000</resolved>
                                                                    <component>Aggregation Framework</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2864676" author="david.daly" created="Tue, 11 Feb 2020 15:50:22 +0000"  >&lt;p&gt;Note we also opened BF-16171 as it represented regression still left over after fixing BF-15717. It looks like a separate regression went in during that period and was hidden by BF-15717.&#160;&lt;/p&gt;</comment>
                            <comment id="2864670" author="david.daly" created="Tue, 11 Feb 2020 15:48:50 +0000"  >&lt;p&gt;The major regression was almost surely BF-15717. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-45691&quot; title=&quot;Change Mutex::LockListeners to use a std::vector again&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-45691&quot;&gt;&lt;del&gt;SERVER-45691&lt;/del&gt;&lt;/a&gt; was supposed to be the fix for it. That ticket made a lot of things slower.&#160;&lt;/p&gt;</comment>
                            <comment id="2864028" author="bernard.gorman" created="Tue, 11 Feb 2020 13:32:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.daly&quot; class=&quot;user-hover&quot; rel=&quot;david.daly&quot;&gt;david.daly&lt;/a&gt;: I assumed there was no additional BF generated because a regression in these tests had already been registered for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt;, and that ticket was still in progress. I&apos;d say it would be better to open a new BF and link it to this ticket, since the cause (broadly speaking) has already been identified - we can use this ticket for further discussion.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ben.caimano&quot; class=&quot;user-hover&quot; rel=&quot;ben.caimano&quot;&gt;ben.caimano&lt;/a&gt;: since you worked on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-45691&quot; title=&quot;Change Mutex::LockListeners to use a std::vector again&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-45691&quot;&gt;&lt;del&gt;SERVER-45691&lt;/del&gt;&lt;/a&gt;, do you have any initial insight into what might have caused this? For context, the graphs above show the average/max time (in seconds) between the time a write is performed and the time the corresponding event is retrieved by change streams. The blue line shows the latency for 60 simultaneous clients performing a mix of inserts, updates, deletes and queries. Each test-case is running a different number of streams alongside these workload clients, so &lt;tt&gt;1_1c&lt;/tt&gt; means one stream reading changes on one collection, &lt;tt&gt;15_1c&lt;/tt&gt; means 15 streams reading changes on one collection, etc.&lt;/p&gt;

&lt;p&gt;There are also a set of corresponding throughput tests; I&apos;ve attached a screenshot of the current graph. These measure the number of operations that the workload threads were able to drive to the cluster given the number of active change streams. As you can see, there was a major regression at some point between December 12th and 17th; I&apos;m currently bisecting in order to find the culprit. Your recent change in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-45691&quot; title=&quot;Change Mutex::LockListeners to use a std::vector again&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-45691&quot;&gt;&lt;del&gt;SERVER-45691&lt;/del&gt;&lt;/a&gt; appears to have &lt;a href=&quot;https://evergreen.mongodb.com/task/sys_perf_linux_shard_lite_change_streams_throughput_539cbefbf49451e2fcdc1a54e2b1810ee6160d50_20_01_22_07_21_43&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;claimed back some of that regression&lt;/a&gt;. It would make sense that this higher write throughput could lead to the increased change stream latency that we see in the former tests, except that &lt;b&gt;the throughput was even higher from August to December, during which time the change stream latency was also much lower.&lt;/b&gt; So it seems that at present we have increased latency &lt;b&gt;despite&lt;/b&gt; a reduction in throughput.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1268844">SERVER-46821</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="1108060">SERVER-45691</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="246785" name="SERVER-45691-regression.png" size="465580" author="bernard.gorman@mongodb.com" created="Sat, 8 Feb 2020 23:14:44 +0000"/>
                            <attachment id="247100" name="SERVER-45691-throughput.png" size="400667" author="bernard.gorman@mongodb.com" created="Tue, 11 Feb 2020 13:24:05 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 10 Feb 2020 16:47:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-46821'>SERVER-46821</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>david.daly@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwpitb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hwdizz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3292">Query 2020-02-24</customfieldvalue>
    <customfieldvalue id="3494">Query 2020-03-09</customfieldvalue>
    <customfieldvalue id="3496">Query 2020-03-23</customfieldvalue>
    <customfieldvalue id="3497">Query 2020-04-06</customfieldvalue>
    <customfieldvalue id="3498">Query 2020-04-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwp52n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>