<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:19:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-28801] Enhance the index validation code to output details on what is wrong</title>
                <link>https://jira.mongodb.org/browse/SERVER-28801</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, the index validation code only mentions that a corrupt index has too many or too few index entries.  But the code has knowledge of which documents are missing index entries, which index entries are missing documents, and which index entries are incorrectly referencing the same document.  It would be helpful to display this information when the validation fails.&lt;br/&gt;
Currently, for documents that are unindexed for example, all that is displayed is:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;indexDetails&quot; : {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 		&quot;db182.coll182.$_id_&quot; : {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 			&quot;valid&quot; : false&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 		}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;(It doesn&apos;t even mention explicitly why the index is invalid: too many, or too few entries)&lt;/p&gt;</description>
                <environment></environment>
        <key id="374094">SERVER-28801</key>
            <summary>Enhance the index validation code to output details on what is wrong</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="milkie@mongodb.com">Eric Milkie</reporter>
                        <labels>
                            <label>stm</label>
                    </labels>
                <created>Fri, 14 Apr 2017 15:24:59 +0000</created>
                <updated>Tue, 6 Dec 2022 04:03:12 +0000</updated>
                            <resolved>Thu, 2 May 2019 19:32:53 +0000</resolved>
                                                                    <component>Testing Infrastructure</component>
                                        <votes>2</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="2231373" author="max.hirschhorn@10gen.com" created="Thu, 2 May 2019 04:25:42 +0000"  >&lt;p&gt;Pretty sure this is done thanks to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregory.wlodarek&quot; class=&quot;user-hover&quot; rel=&quot;gregory.wlodarek&quot;&gt;gregory.wlodarek&lt;/a&gt; but reassigning to the Storage team to confirm.&lt;/p&gt;</comment>
                            <comment id="1986171" author="geert.bosch" created="Mon, 27 Aug 2018 11:55:22 +0000"  >&lt;p&gt;Yes, it was always the intent to do another pass, but we never got to schedule that work and was not strictly necessary for the primary goal of detecting corruption. &lt;/p&gt;</comment>
                            <comment id="1981207" author="milkie" created="Tue, 21 Aug 2018 14:39:54 +0000"  >&lt;p&gt;The code was designed to not hold locks, as part of the background validation project, so it depends on whether you want to break that or not.&lt;/p&gt;</comment>
                            <comment id="1981077" author="max.hirschhorn@10gen.com" created="Tue, 21 Aug 2018 13:27:28 +0000"  >&lt;blockquote&gt;
&lt;p&gt;Writes to the collection in the interim could throw the results off, but for our automated testing cases that shouldn&apos;t be a problem.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Couldn&apos;t we just hold the &lt;tt&gt;MODE_X&lt;/tt&gt; lock on the collection for even longer to avoid that being an issue?&lt;/p&gt;</comment>
                            <comment id="1981064" author="milkie" created="Tue, 21 Aug 2018 13:17:52 +0000"  >&lt;p&gt;Yes, that could be done.  Writes to the collection in the interim could throw the results off, but for our automated testing cases that shouldn&apos;t be a problem.&lt;/p&gt;</comment>
                            <comment id="1978196" author="max.hirschhorn@10gen.com" created="Thu, 16 Aug 2018 22:24:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, do either of you know whether another pass through the collection or an index could be done to report the documents or index keys which map to the offending hash?&lt;/p&gt;</comment>
                            <comment id="1551753" author="milkie" created="Tue, 18 Apr 2017 17:15:28 +0000"  >&lt;p&gt;In light of these limitations, I&apos;d like to change the work for this ticket to encompass changing the validate() logic itself.  This would be a lot more work, but eventually this sort of diagnosis can be really valuable.&lt;/p&gt;</comment>
                            <comment id="1551702" author="robert.guo" created="Tue, 18 Apr 2017 16:31:20 +0000"  >&lt;p&gt;1. In the case of having more documents than index entries, &lt;tt&gt;validate()&lt;/tt&gt; would confuse users because it only knows when the number of documents has exceeded the number of index entries, but the &quot;leftover&quot; documents may not be the root cause of the inconsistency; the actual cause is likely that previously scanned documents had missing index entries. &lt;/p&gt;

&lt;p&gt;Showing the leftover documents could be useful for our tests where we have relatively few documents and there&apos;s a higher chance of an invalid document being printed; but in the field where the number of inconsistent documents is typically far fewer than the total number of documents, it&apos;s highly unlikely the printed information will be useful.&lt;/p&gt;

&lt;p&gt;(edit: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.5.3/jstests/hooks/validate_collections.js#L81&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;we&apos;re dumping the collection in the JS hook&lt;/a&gt; if validate() fails, so we don&apos;t really gain additional information in our tests from showing these leftover documents...)&lt;/p&gt;

&lt;p&gt;2. For index entries missing documents, we don&apos;t currently keep track of the extra entry, or even which index has extra entries (all we know is that a number in a hash table is &amp;gt;0, which could have been caused by any index); so there&apos;s no extra information to print.&lt;/p&gt;

&lt;p&gt;3. Misordered indexes is one scenario where we do have extra information! In this case, we can actually add the offending entries to the error message. (But I don&apos;t think we&apos;ve ever seen this error in our testing)&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; Given the limitations of points 1 and 2, I don&apos;t think there&apos;s anything more we can print for users without changing the validate() code to do another pass of the collection and invalid index.&lt;/p&gt;</comment>
                            <comment id="1551597" author="milkie" created="Tue, 18 Apr 2017 15:20:48 +0000"  >&lt;p&gt;In addition to helping diagnose build failures, I was hoping to use this new code to help diagnose failures in the field for which we have no reproducer in house.  I&apos;m not sure having the extra information will confuse users in regard to what they need to do next; they will still either resync, or contact MongoDB for assistance.&lt;/p&gt;</comment>
                            <comment id="1551554" author="max.hirschhorn@10gen.com" created="Tue, 18 Apr 2017 14:56:10 +0000"  >&lt;p&gt;After some discussion internally with the TIG team, we are likely to only add the additional logging when the server is running with enableTestCommands=1 to avoid any confusion around why in some cases additional details are included in the response. The details themselves won&apos;t really instruct a user as to the action they need to take since they&apos;ll still want to resync or reIndex their node.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, does that seem reasonable to you?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="408797">SERVER-30356</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 18 Apr 2017 14:56:10 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 40 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 40 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>robert.guo@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht5te7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsy43j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4awf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>