<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:04:59 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44092] Orphan documents impact IXSCAN performance</title>
                <link>https://jira.mongodb.org/browse/SERVER-44092</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This happens once in a while. So far, no clear steps were found to reproduce this, but the symptoms are clear enough on their own.&#160;&lt;/p&gt;

&lt;p&gt;This issue looks like this: some of mongod instances start to have an abnormally high CPU usage. They have the same amount of data as other shards, the same count and shape of operations. This may go as bad as near 100% cpu usage, but mongod instance still woks, albeit with increased latency.&#160;&lt;/p&gt;

&lt;p&gt;In all cases we found the same offending query pattern: there are some frequent reads to a collection with, let&apos;s say, &quot;foo == 1&quot; query. The returned documents are updated with &quot;foo = 2&quot; and updated in db, then read query is repeated. Normally, this query often returns zero documents, and this doesn&apos;t upset mongod very much. When the issue takes place, however, this query still returns zero documents, but takes much longer to do so.&lt;/p&gt;

&lt;p&gt;Some prior experience with orphaned documents gave us a degree of insight, so we tried and cleaned up orphans in the problematic collection. All performance effects ceased at once. No other side effects were observed.&#160;&lt;/p&gt;

&lt;p&gt;The effect presented itself several times, and each time this workaround was just as successful. At once such occasion we degraded state lasted some 12 hours, so this is not something that resolves itself - and therefore this doesn&apos;t look like a healthy last stage of chunk balancing, for example. Orphans are left behind for good. Or rather, for the bad.&#160;&lt;/p&gt;

&lt;p&gt;The issue seems random and damaging enough to warrant some sort of user-side workaround, like cron-based orphan cleanup on all collections. Clearly, such measures must not be the only way to fix the issue.&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="972714">SERVER-44092</key>
            <summary>Orphan documents impact IXSCAN performance</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="eric.sedor@mongodb.com">Eric Sedor</assignee>
                                    <reporter username="petr.ivanov.s@gmail.com">Peter Ivanov</reporter>
                        <labels>
                    </labels>
                <created>Fri, 18 Oct 2019 09:26:33 +0000</created>
                <updated>Fri, 6 Dec 2019 22:40:36 +0000</updated>
                            <resolved>Fri, 6 Dec 2019 22:40:36 +0000</resolved>
                                    <version>4.0.10</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="2590861" author="eric.sedor" created="Fri, 6 Dec 2019 22:40:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=petr.ivanov.s%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;petr.ivanov.s@gmail.com&quot;&gt;petr.ivanov.s@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;m going to close this ticket for now but I do encourage you to write back if explain plan information becomes available.&lt;/p&gt;

&lt;p&gt;Thanks a lot!&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="2498607" author="petr.ivanov.s@gmail.com" created="Thu, 24 Oct 2019 06:58:11 +0000"  >&lt;p&gt;Thank you for clarification, Eric. We&apos;ll come back with more info if we can.&#160;&lt;/p&gt;</comment>
                            <comment id="2498203" author="eric.sedor" created="Wed, 23 Oct 2019 20:31:07 +0000"  >&lt;p&gt;To clarify, there is extra work to do to avoid returning orphaned documents in query results (There are some details in a recent ticket, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38972&quot; title=&quot;mongos takes forever for multi shard queries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38972&quot;&gt;&lt;del&gt;SERVER-38972&lt;/del&gt;&lt;/a&gt;). This has contributed to the desire to limit their existence entirely. In the meantime, explain plans might provide us with evidence of specific bugs that we&apos;re not aware of. I&apos;m going to keep this ticket open for a while in case you are able to provide these.&lt;/p&gt;

&lt;p&gt;One potential workaround (before 4.4) aside from the automation of cleanupOrphaned you mentioned is to look into what has prompted the orphans in the deployment to begin with. For assistance troubleshooting the creation of orphans, I encourage you to ask our community by posting on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or on &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Gratefully,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="2496975" author="petr.ivanov.s@gmail.com" created="Wed, 23 Oct 2019 05:55:23 +0000"  >&lt;p&gt;Hi Eric.&#160;&lt;/p&gt;

&lt;p&gt;Possibly, we could. This doesn&apos;t happen all that often, though, and this won&apos;t be possible at all in case we implement some sort of a workaround in a manner described above.&#160;&lt;/p&gt;

&lt;p&gt;But the query is actually exactly as simple as mentioned above. All you need is a index for a field, a query with a specific value or a range for this field. I believe that if you artificially create a lot of orphans in a state when this field matches a query, and then modify the actual moved documents so that they don&apos;t satisfy it any more, then a request via mongos will reproduce exactly the issue I&apos;m describing.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;From user&apos;s perspective, though, a mechanism to properly ensure the cleanup of all orphans would solve this issue completely. It depends, of course, on exactly how &apos;eventually&apos; it will happen.&#160;&lt;/p&gt;</comment>
                            <comment id="2496838" author="eric.sedor" created="Wed, 23 Oct 2019 00:33:49 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=petr.ivanov.s%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;petr.ivanov.s@gmail.com&quot;&gt;petr.ivanov.s@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;This is Eric chiming in to ask for some more information. As my colleague &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat&quot;&gt;dmitry.agranat&lt;/a&gt; already pointed out, we are working to ensure sharded clusters clean up orphans on their own.&lt;/p&gt;

&lt;p&gt;However, I think it would help if we understood exactly where the poor performance is coming from in this case. For that we need to understand exactly what query and index are being impacted by the existence of orphan documents, and in what way.&lt;/p&gt;

&lt;p&gt;If/when this happens again, can you please collect &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/cursor.explain/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;explain(&quot;executionStats&quot;)&lt;/a&gt; output for the query both before and after cleaning up orphaned documents?&lt;/p&gt;

&lt;p&gt;Gratefully,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="2494344" author="dmitry.agranat" created="Tue, 22 Oct 2019 13:04:22 +0000"  >&lt;p&gt;Hi Peter,&lt;/p&gt;

&lt;p&gt;We potentially plan to address this issue starting with MongoDB 4.4, the server will ensure that eventually there are no more orphaned documents.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2493343" author="eric.sedor" created="Mon, 21 Oct 2019 19:43:38 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=petr.ivanov.s%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;petr.ivanov.s@gmail.com&quot;&gt;petr.ivanov.s@gmail.com&lt;/a&gt;; we will take a look at this submission, and may have some follow up questions.&lt;/p&gt;

&lt;p&gt;Eric&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="671627">SERVER-38972</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 21 Oct 2019 19:43:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 9 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/PM-812'>PM-812</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 9 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>petr.ivanov.s@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvy0lr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvmh8f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvxmv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>