<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:05:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44356] Change Stream latency is determined by config server write rate</title>
                <link>https://jira.mongodb.org/browse/SERVER-44356</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt;, we introduced an improved mechanism for change streams to detect the addition of new shards to the cluster. This involves opening an internal cursor on the config servers alongside the shard streams, in order to monitor for writes to the &lt;tt&gt;config.shards&lt;/tt&gt; collection. While this approach guarantees that we will always correctly pick up new shards, it has the effect of making the change stream&apos;s latency dependent upon the config server write rate; since the config server cursor is treated as just another shard in the sorted stream from the ARM&apos;s perspective, the latter cannot return results from any shards until the config server writes an oplog entry which surpasses the &lt;tt&gt;clusterTime&lt;/tt&gt; of those events.&lt;/p&gt;

&lt;p&gt;In addition to the writes that the config servers perform to persist metadata changes, each &lt;tt&gt;mongoS&lt;/tt&gt; in the cluster pings the config servers every 10s and every component in the cluster lockpings every 30 seconds. A large and/or busy cluster will therefore not suffer much additional latency. But in the scenario where a cluster has only a single &lt;tt&gt;mongoS&lt;/tt&gt; and is not actively splitting or migrating, our latency guarantees go from a worst-case of ~10s if one of the shards in the cluster is completely inactive, to a minimum of ~10s regardless of how active the shards are.&lt;/p&gt;

&lt;p&gt;There are a couple of ways we could address this:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Ask Sharding if they are prepared to enable &lt;tt&gt;writePeriodicNoops&lt;/tt&gt; on the config servers by default, and set it to a high frequency of maybe 1-2 seconds to minimize the impact on change streams. Each no-op entry is 103 bytes, which at even 1-second frequency is only ~362kB per hour before compression.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Use both the old and new shard-detection mechanisms. The new approach is applicable to all cases, but only strictly necessary for (1) whole-cluster streams, and (2) streams which are opened on a database that does not yet exist. Aside from those cases, we do not need to open a cursor on the config servers at all. In the case of (2), we could also close the &lt;tt&gt;config.shards&lt;/tt&gt; cursor as soon as we see the first event in the stream.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Do nothing and consider this another flavour of activity-dependent latency.&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="987168">SERVER-44356</key>
            <summary>Change Stream latency is determined by config server write rate</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="bernard.gorman@mongodb.com">Bernard Gorman</assignee>
                                    <reporter username="bernard.gorman@mongodb.com">Bernard Gorman</reporter>
                        <labels>
                            <label>qexec-team</label>
                    </labels>
                <created>Fri, 1 Nov 2019 10:37:29 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:00 +0000</updated>
                            <resolved>Sun, 16 Feb 2020 03:06:29 +0000</resolved>
                                                    <fixVersion>4.3.4</fixVersion>
                                    <component>Aggregation Framework</component>
                                        <votes>0</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="2880202" author="xgen-internal-githook" created="Sun, 16 Feb 2020 03:04:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;gormanb&apos;, &apos;name&apos;: &apos;Bernard Gorman&apos;, &apos;email&apos;: &apos;bernard.gorman@gmail.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt; Set config server periodicNoopIntervalSecs to 1&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1d21e89fa5c9d838b2f40045c26fe41f306a8873&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1d21e89fa5c9d838b2f40045c26fe41f306a8873&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2768473" author="kaloian.manassiev" created="Tue, 28 Jan 2020 16:42:18 +0000"  >&lt;p&gt;We just had a meeting about this with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jack.mulrow&quot; class=&quot;user-hover&quot; rel=&quot;jack.mulrow&quot;&gt;jack.mulrow&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;Jack pointed out to me that the gossiped &lt;tt&gt;configOpTime&lt;/tt&gt; is actually the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f76c092162f8bc8f12696064ca6e866c1ba607bf/src/mongo/db/s/sharding_config_optime_gossip.cpp#L51-L54&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;majority committed&lt;/a&gt; opTime on the config server and not the last written, like the clusterTime. As a consequence of this, it means that reads from the config server (which are &lt;tt&gt;w:majority, afterOpTime: configOpTime&lt;/tt&gt;) are less likely to block waiting on the opTime to become majority written. This in turn means that enabling the NoopWriter on the config server is actually not as problematic as I originally thought.&lt;br/&gt;
&#160;&lt;br/&gt;
So in summary, we can just enable the NoopWriter on the config server in order to address this problem and there is no need to rely on the topologyOpTime.&lt;/p&gt;</comment>
                            <comment id="2718221" author="bernard.gorman" created="Fri, 10 Jan 2020 10:30:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;: I&apos;m not sure that will address the original problem, as outlined in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt;. We already have an oplog event &lt;tt&gt;migrateChunkToNewShard&lt;/tt&gt; which is generated whenever a chunk is moved to a shard that doesn&apos;t have any chunks; this was the original way that change streams detected the addition of new shards. The problem identified in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt; is that, if a new shard is added &lt;b&gt;and a new database is created on that shard&lt;/b&gt;, then a whole-cluster change stream will miss all events that occur on the shard until the first chunk migrates to it. The problem is actually somewhat more extensive than this simple example; see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723?focusedCommentId=2404274&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2404274&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this comment&lt;/a&gt; for more details.&lt;/p&gt;

&lt;p&gt;How will the new &lt;tt&gt;clusterTopologyOpTime&lt;/tt&gt; help us to avoid this scenario?&lt;/p&gt;</comment>
                            <comment id="2712217" author="kaloian.manassiev" created="Wed, 8 Jan 2020 16:44:18 +0000"  >&lt;p&gt;Apologies for the delay in getting back to this. I didn&apos;t realise it had impact on driver testing as well and I thought it was about performance only.&lt;/p&gt;

&lt;p&gt;Enabling the NoopWriter on the config server will introduce unnecessary waits for metadata reads, which in turn can have negative and possibly unpredictable impact on the latency of CRUD operations, which end up having to do routing info refresh. One such example is that let&apos;s say a migration commits (with majority) and produces an opTime 100, but before it returns to the caller shard, the NoopWriter advances it up to 101. Now the refresh needs to wait for 101 to become majority committed instead of 100.&lt;/p&gt;

&lt;p&gt;Because of this I am hesitant just making this change.&lt;/p&gt;

&lt;p&gt;Instead, what we are proposing to do is to introduce a &lt;tt&gt;clusterTopologyOpTime&lt;/tt&gt; vector clock component, which gets advanced only when shards are added and removed. This, combined with the causal consistency provided for reads against the metadata, will ensure that if any chunk is removed to a newly added shard, any change streams will see the newly added shard when they see the chunk migration.&lt;/p&gt;

&lt;p&gt;This is not something which is quick to do though and will take at least a couple of iterations.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="888141">SERVER-42723</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2426792">SERVER-80427</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 8 Jan 2020 15:18:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 51 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 51 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw059r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr6rof:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3286">Query 2019-12-02</customfieldvalue>
    <customfieldvalue id="3287">Query 2019-12-16</customfieldvalue>
    <customfieldvalue id="3288">Query 2019-12-30</customfieldvalue>
    <customfieldvalue id="3289">Query 2020-01-13</customfieldvalue>
    <customfieldvalue id="3290">Query 2020-01-27</customfieldvalue>
    <customfieldvalue id="3291">Query 2020-02-10</customfieldvalue>
    <customfieldvalue id="3292">Query 2020-02-24</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvzrj3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>