<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 22:28:17 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[COMPASS-1764] Compass handles the crazy keys document series</title>
                <link>https://jira.mongodb.org/browse/COMPASS-1764</link>
                <project id="13182" key="COMPASS">Compass </project>
                    <description>&lt;h4&gt;&lt;a name=&quot;Usage%2Fstepstoreproduce&quot;&gt;&lt;/a&gt;Usage/steps to reproduce&lt;/h4&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# Generate the document&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;python3 mkbson.py &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# Import the document&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongorestore -d perf -c crazy_keys raw_crazy_keys.bson &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Then navigate to the &lt;tt&gt;perf.crazy_keys&lt;/tt&gt; collection in Compass.&lt;/p&gt;

&lt;p&gt;An extended set of documents can be created and imported into MongoDB with this script relatively easily, on my local machine I already have the following numbers of keys and preliminary results:&lt;br/&gt;
32^3=32,768&lt;br/&gt;
48^3=110,592 &lt;br/&gt;
64^3=262,144 (slow, ~5-10s)&lt;br/&gt;
80^3=512,000 (slow, ~30s)&lt;br/&gt;
96^3=884,736&lt;br/&gt;
140^3=2,744,000 (Compass 1.9.x betas crash with a memory exception)&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;Proposedsolutioncapat100kfields&quot;&gt;&lt;/a&gt;Proposed solution - cap at 100k fields&lt;/h4&gt;

&lt;p&gt;#intellectualhonesty - let the data drive your decision.&lt;/p&gt;

&lt;p&gt;Thus Compass, most likely in &lt;a href=&quot;https://github.com/mongodb-js/mongodb-schema&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-schema&lt;/a&gt;, should stop scanning at about 100k fields (configurable as other optimizations are found) in a document and report this Easter egg in the GUI (originally this was Issue 2 in &lt;a href=&quot;https://jira.mongodb.org/browse/COMPASS-1901&quot; title=&quot;Implement CRUD loading spinners/ripples/animations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;COMPASS-1901&quot;&gt;&lt;del&gt;COMPASS-1901&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;Acceptancecriteria&quot;&gt;&lt;/a&gt;Acceptance criteria&lt;/h4&gt;

&lt;ul&gt;
	&lt;li&gt;A mongoimport -able version of these documents, perhaps with easier to understand names like crazy_keys_512000 (might be already done by &lt;a href=&quot;https://jira.mongodb.org/browse/COMPASS-1766&quot; title=&quot;Investigate Compass CRUD render times with increasing numbers of keys&quot; class=&quot;issue-link&quot; data-issue-key=&quot;COMPASS-1766&quot;&gt;&lt;del&gt;COMPASS-1766&lt;/del&gt;&lt;/a&gt;, if so drop up to 2 story points)&lt;/li&gt;
	&lt;li&gt;Cap field or schema processing at 100k fields, reporting this somehow, e.g. with properties such as &quot;totalFieldCount&quot;, &quot;analyzedFieldCount&quot; (working titles to capture the semantic, change the syntax as needed).&lt;/li&gt;
	&lt;li&gt;Message displaying this scenario, e.g. &quot;100,000 fields limit reached, remaining N fields are not displayed&quot;&lt;/li&gt;
	&lt;li&gt;Appropriate unit tests&lt;/li&gt;
&lt;/ul&gt;


&lt;h4&gt;&lt;a name=&quot;Outofscope&quot;&gt;&lt;/a&gt;Out of scope&lt;/h4&gt;

&lt;ul&gt;
	&lt;li&gt;Anything in the schema/indexes/document validation tabs (such as the react-select dropdown to choose an index name), this is just about the Documents Tab.&lt;/li&gt;
&lt;/ul&gt;


&lt;h4&gt;&lt;a name=&quot;Background&quot;&gt;&lt;/a&gt;Background&lt;/h4&gt;

&lt;p&gt;So I let Friday afternoon get to me and challenged myself to create a document with ~2.7 million keys.&lt;/p&gt;

&lt;p&gt;After about 3 minutes, Compass crashes. The mongo shell is still doing something in CPU-land after 8 minutes so far.&lt;/p&gt;

&lt;p&gt;On different data sets, such as documents with ~500,000 fields (i.e. &lt;a href=&quot;https://jira.mongodb.org/browse/COMPASS-1766&quot; title=&quot;Investigate Compass CRUD render times with increasing numbers of keys&quot; class=&quot;issue-link&quot; data-issue-key=&quot;COMPASS-1766&quot;&gt;&lt;del&gt;COMPASS-1766&lt;/del&gt;&lt;/a&gt;), Compass takes ~30 seconds or more to render, or crashes (&lt;a href=&quot;https://jira.mongodb.org/browse/COMPASS-1764&quot; title=&quot;Compass handles the crazy keys document series&quot; class=&quot;issue-link&quot; data-issue-key=&quot;COMPASS-1764&quot;&gt;&lt;del&gt;COMPASS-1764&lt;/del&gt;&lt;/a&gt;). &lt;/p&gt;

&lt;p&gt;Would explicitly limiting Compass to say 100,000 fields with a message like &quot;Compass cannot render documents with more than 100,000 fields&quot; seem reasonable? Yes but only when the user gets down to the 100,001st field.&lt;/p&gt;

&lt;p&gt;For documents under 100k fields, is a loading spinner enough? (i.e. ~6 seconds, so should not need a progress bar?). &lt;/p&gt;

&lt;p&gt;Proposed Solutions: Potentially push to the GPU? Otherwise just drop to &lt;tt&gt;Loading...&lt;/tt&gt; text.&lt;/p&gt;</description>
                <environment></environment>
        <key id="417815">COMPASS-1764</key>
            <summary>Compass handles the crazy keys document series</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="peter.schmidt">Peter Schmidt</reporter>
                        <labels>
                    </labels>
                <created>Fri, 18 Aug 2017 08:14:29 +0000</created>
                <updated>Fri, 27 Oct 2023 20:19:27 +0000</updated>
                            <resolved>Mon, 1 Oct 2018 10:26:24 +0000</resolved>
                                                                    <component>CRUD</component>
                    <component>Performance</component>
                        <due></due>
                            <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                <comments>
                            <comment id="2018843" author="durran.jordan@10gen.com" created="Mon, 1 Oct 2018 10:26:24 +0000"  >&lt;p&gt;Rewriting stories to remove hadron-document&lt;/p&gt;</comment>
                            <comment id="1651185" author="peter.schmidt" created="Fri, 18 Aug 2017 08:15:23 +0000"  >&lt;p&gt;I think it&apos;s safe to say for this synthetic example, wont fix (at least for now, and especially as the Mongo shell doesn&apos;t work either).&lt;/p&gt;

&lt;p&gt;EDIT: Compass should never crash (that would be a bug if reported by a user), so this issue does need to be handled explicitly. Ticket has accordingly been repurposed into a task broken out of discussion which started on &lt;a href=&quot;https://jira.mongodb.org/browse/COMPASS-1901&quot; title=&quot;Implement CRUD loading spinners/ripples/animations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;COMPASS-1901&quot;&gt;&lt;del&gt;COMPASS-1901&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="163517" name="mkbson.py" size="1133" author="peter.schmidt" created="Fri, 18 Aug 2017 08:13:13 +0000"/>
                            <attachment id="163516" name="raw_crazy_keys.bson" size="16464005" author="peter.schmidt" created="Fri, 18 Aug 2017 08:13:21 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 1 Oct 2018 10:26:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 19 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_14262" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>End date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Sep 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>COMPASS-1739</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 19 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>durran.jordan@mongodb.com</customfieldvalue>
            <customfieldvalue>peter.schmidt</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htd7z3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9alb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_14261" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>Start date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 18 Aug 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htcu1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>