<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:06:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-24482] Initial sync during high document update/churn causes repl-worker slowness, connection churn</title>
                <link>https://jira.mongodb.org/browse/SERVER-24482</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Summary&lt;/b&gt;&lt;br/&gt;
A replica-set with a high document churn, in particular where documents are frequently updated and deleted (the same document is updated then deleted later), where a member tries to initial sync it will suffer from a low replication rate when it first switches to catching up on the oplog. The replication rate can be so low that the initial sync may eventually fail in this final stage.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Scenario&lt;/b&gt;&lt;br/&gt;
Start with a collection with a few hundred thousand documents in it. Run a script that updates documents somewhat randomly, but also deletes and inserts some - in particular, it is important that documents which are being updated are also candidates for deletion, with new documents being inserted to take their place. The idea is to make the sync&apos;ing member collection copy highly skewed across time such that when it starts to apply the oplog it will have updates for documents that don&apos;t exist in either its own store or in any remote member. This will result in a message like the following:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[repl writer worker 1] missing object not found on source. presumably deleted later in oplog&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;These messages are not fatal, the scenario can be recovered, but the appearance of this message is correlated with both very slow oplog application and connection churning observed on the sync source from the member being sync&apos;ed.&lt;/p&gt;

&lt;p&gt;The oplog application speeds up dramatically once it gets past the crossover point where no more operations are occurring for deleted documents.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Reproduction&lt;/b&gt;&lt;br/&gt;
The attached script can reproduce this as follows:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Create a single member replica-set.&lt;/li&gt;
	&lt;li&gt;Run &lt;tt&gt;tdb.insertUntil(500000)&lt;/tt&gt; in the shell and go read a book (this is safe to run from multiple mongo shells to improve throughput)&lt;/li&gt;
	&lt;li&gt;Add a new member, which should start sync&apos;ing&lt;/li&gt;
	&lt;li&gt;Run &lt;tt&gt;tdb.execChurner(1000,20,50)&lt;/tt&gt; in the shell - this loops 1000 times over the following:
	&lt;ul&gt;
		&lt;li&gt;Find 50 sort-of random documents, make an array of their _id&lt;/li&gt;
		&lt;li&gt;Apply a random update to each document in the _id array, delete last document in the array (remove document from mongodb and pop that array entry), and repeat until the array is empty&lt;/li&gt;
		&lt;li&gt;Sleep for 20ms&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Watch what happens when the sync&apos;ing member switches to applying the oplog.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;If this is monitored closely the sync&apos;ing member is likely to continue to fall further behind as it applies oplog entries too slowly. As the number of &quot;missing object not found on source&quot; messages drop the replication rate increases. Once no more messages of that type occur, replication rate skyrockets.&lt;/p&gt;</description>
                <environment></environment>
        <key id="292613">SERVER-24482</key>
            <summary>Initial sync during high document update/churn causes repl-worker slowness, connection churn</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="evin.roesle@mongodb.com">Evin Roesle</assignee>
                                    <reporter username="andrew.ryder@mongodb.com">Andrew Ryder</reporter>
                        <labels>
                            <label>RF</label>
                            <label>initialSync</label>
                    </labels>
                <created>Thu, 9 Jun 2016 06:17:06 +0000</created>
                <updated>Fri, 3 Jan 2020 21:48:13 +0000</updated>
                            <resolved>Fri, 3 Jan 2020 21:48:13 +0000</resolved>
                                    <version>3.0.12</version>
                                                    <component>Replication</component>
                                        <votes>3</votes>
                                    <watches>23</watches>
                                                                                                                <comments>
                            <comment id="2703429" author="evin.roesle" created="Fri, 3 Jan 2020 21:48:13 +0000"  >&lt;p&gt;This issue has gone away in MongoDB 4.3+ so we are closing this ticket.&lt;/p&gt;</comment>
                            <comment id="1289054" author="scotthernandez" created="Thu, 9 Jun 2016 10:28:46 +0000"  >&lt;p&gt;During the first apply phase (of initial sync) when attempting to get to a consistent point, any failed update due to a missing document must re-fetch the document &amp;#8211; in this case, since it was deleted it could not be found; each of those queries can be costly and are currently run sequentially, and in-line with the applies. Since fetching the new document, even if not found, causes increasing the window of operation which must be applied before initial sync can complete these cases can also lengthen the time before initial sync is complete.&lt;/p&gt;

&lt;p&gt;This is a known performance issue, and I&apos;ve linked a work item related to improving performance in these cases.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="160393">SERVER-15410</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="125553" name="tdb.js" size="6979" author="andrew.ryder@mongodb.com" created="Thu, 9 Jun 2016 06:17:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000V7dorIAB, 500A000000WU4ClIAL, 500A000000cCoCyIAK, 5002K00000dP5ZHQA0, 5002K00000dak3AQAQ]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 9 Jun 2016 10:28:46 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 5 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>evin.roesle@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 5 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.ryder@mongodb.com</customfieldvalue>
            <customfieldvalue>evin.roesle@mongodb.com</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk61j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsmapr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3001">Repl 2019-07-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsevu7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>