<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:55:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20770] Consolidate the two code paths for running find commands in the mongo shell</title>
                <link>https://jira.mongodb.org/browse/SERVER-20770</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The shell uses DBCommandCursor to iterate results returned from the &quot;cursor-generating commands&quot; (find, aggregate, listIndexes, listCollections). The implementation is primarily in Javascript:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/bd76599a88e8062c634eeb6918bb30f868bc6042/src/mongo/shell/query.js#L650-L804&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/bd76599a88e8062c634eeb6918bb30f868bc6042/src/mongo/shell/query.js#L650-L804&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Rewriting this functionality in C++ would most likely make this code run faster. It would also ensure that the shell does not consume excessive memory, since memory required to buffer each batch of results could be reclaimed after the batch is exhausted. Currently, the Javascript engine&apos;s garbage collector may not run frequently enough to reclaim memory in a timely manner, especially for queries returning many megabytes worth of results.&lt;/p&gt;</description>
                <environment></environment>
        <key id="232914">SERVER-20770</key>
            <summary>Consolidate the two code paths for running find commands in the mongo shell</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-query-execution">Backlog - Query Execution</assignee>
                                    <reporter username="david.storch@mongodb.com">David Storch</reporter>
                        <labels>
                            <label>move-stm</label>
                            <label>storch</label>
                            <label>tech-debt</label>
                    </labels>
                <created>Mon, 5 Oct 2015 22:46:54 +0000</created>
                <updated>Thu, 8 Jun 2023 19:11:25 +0000</updated>
                                                                            <component>Querying</component>
                    <component>Shell</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="5486041" author="JIRAUSER1270811" created="Thu, 8 Jun 2023 19:08:29 +0000"  >&lt;p&gt;I have attached file &lt;a href=&quot;https://jira.mongodb.org/secure/attachment/455060/sv20770.tar.gz&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;sv20770.tar.gz&lt;/a&gt; containing Git patch files for the 14 commits on PR 13066. These are against SHA 72c47177dd75d7451d623a04f0423971b55ebf07:&lt;/p&gt;

&lt;p&gt;0001-&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20770&quot; title=&quot;Consolidate the two code paths for running find commands in the mongo shell&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20770&quot;&gt;SERVER-20770&lt;/a&gt;-Consolidate-the-two-code-paths-for-runn.patch&lt;br/&gt;
0002-sv20770-ps02-Experiment-treating-everything-as-a-C-e.patch&lt;br/&gt;
0003-sv20770-ps03-Undo-ps02.patch&lt;br/&gt;
0004-sv20770-ps04-If-slaveOk-option-given-use-readPrefere.patch&lt;br/&gt;
0005-sv20770-ps05-Undo-ps04.patch&lt;br/&gt;
0006-sv20770-ps06-Set-default-readPreference-primaryPrefe.patch&lt;br/&gt;
0007-sv20770-ps07-Code-format.patch&lt;br/&gt;
0008-sv20770-ps08-Make-client-side-C-parser-include-comme.patch&lt;br/&gt;
0009-sv20770-ps09-C-mimic-JS-self-mod-code-set_read_prefe.patch&lt;br/&gt;
0010-sv20770-p10-Remove-instrumentation-other-cleanups.patch&lt;br/&gt;
0011-sv20770-p11-Move-constants-to-make-visible-in-should.patch&lt;br/&gt;
0012-sv20770-ps12-Experiment-leave-ps11-refector-in-but-d.patch&lt;br/&gt;
0013-sv20770-ps13-Reduce-scope-of-ps09-fix-from-global-to.patch&lt;br/&gt;
0014-sv20770-ps14-Add-prepareCommandRequest-call-for-non-.patch&lt;/p&gt;</comment>
                            <comment id="5485832" author="JIRAUSER1270811" created="Thu, 8 Jun 2023 18:03:54 +0000"  >&lt;p&gt;PR (not delivered) of the partial work (against master branch, which was 7.1.0-rc0 at the time) I did before this was paused on 2023-06-08 (dev branch &lt;b&gt;cherkauer_sv20770)&lt;/b&gt;:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/10gen/mongo/pull/13066&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/10gen/mongo/pull/13066&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This has 14 commits on it, but the best results were obtained on the eighth one (labeled &quot;sv20770 ps08...&quot;). Commits after that were attempts to correctly emulate the set_read_preference_secondary.js override used by&lt;/p&gt;

&lt;p&gt;resmoke.py --suite=aggregation_secondary_reads jstests/aggregation/sources/collStats/query_exec_stats.js&lt;/p&gt;

&lt;p&gt;These changes fixed that test but caused about 15 other tests in the suite to start failing in patch builds. (Prior to these changes, query_exec_stats.js was the only test failing in this suite on the C++ code path.)&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Search for the &quot;kjc&quot; comment markers in query.js&lt;/b&gt; to see how I forced all &quot;find&quot; commands to take the C++ code path, without removing the JS code path. This can easily be flipped back and forth by changing whether the marked if() test in &lt;b&gt;DBQuery.prototype._exec&lt;/b&gt; is &quot;if (false)&quot; (forces C++ code path for everything) or &quot;if (!this._isExhaustCursor())&quot; (original behavior of C++ code path for exhaust cursors and JS code path for everything else).&lt;/p&gt;</comment>
                            <comment id="5485711" author="JIRAUSER1270811" created="Thu, 8 Jun 2023 17:30:17 +0000"  >&lt;p&gt;I worked on this for three weeks, but now we are pausing it. Short summary:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Putting all find calls through the C++ path without other changes produces 230 suite failures in patch build.&lt;/li&gt;
	&lt;li&gt;After three weeks of effort I have been able to fix 27 of these.&lt;/li&gt;
	&lt;li&gt;Interpolating, this produces an estimate of 26 weeks to reach &quot;Ready To Review,&quot; which is not the end. (Besides review there will be a long tail of BFs trickling in from tests that get broken but are not part of patch builds.)&lt;/li&gt;
	&lt;li&gt;The main barrier is self-modifying code (&quot;overrides&quot;) in the JS code path (jstests/libs/override_methods/ as well as manual overrides). These are used extensively: about &lt;b&gt;400&lt;/b&gt; JS and YAML files load overrides from this library directory, and there are additional cases of custom overrides done manually.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Profile of this project is &lt;b&gt;high-cost&lt;/b&gt; (lots of effort), &lt;b&gt;high-risk&lt;/b&gt; (potential to break a large number of tests not visible to patch builds), and &lt;b&gt;low-payoff&lt;/b&gt; (if successful, tests might run faster and some JS code might be retired, but new JS code to emulate overrides in the &quot;C++&quot; code path will have been created).&lt;/p&gt;

&lt;p&gt;I will attach artifacts for the work I did up to this point.&lt;/p&gt;</comment>
                            <comment id="4696989" author="david.storch" created="Thu, 21 Jul 2022 21:44:37 +0000"  >&lt;p&gt;I looked at this in a bit more detail this week. It&apos;s going to be easier to implement it in two steps. The first step is to simplify the shell as described by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-68213&quot; title=&quot;Stop generating an OP_QUERY-style query representation in the shell&amp;#39;s find implementation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-68213&quot;&gt;&lt;del&gt;SERVER-68213&lt;/del&gt;&lt;/a&gt;. Once this is complete, it could be easier to unify the exhaust and non-exhaust paths in the shell. I&apos;m going to work on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-68213&quot; title=&quot;Stop generating an OP_QUERY-style query representation in the shell&amp;#39;s find implementation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-68213&quot;&gt;&lt;del&gt;SERVER-68213&lt;/del&gt;&lt;/a&gt; instead of this ticket, so I will put this ticket back into the triage queue.&lt;/p&gt;</comment>
                            <comment id="4657562" author="david.storch" created="Tue, 5 Jul 2022 15:01:37 +0000"  >&lt;p&gt;I think we should consider this ticket as a quick win. Given that the concept of &quot;legacy reads&quot; versus &quot;read commands&quot; has gone away, there is no longer a reason to have two paths for running queries in the shell. At the moment, the C++ path is specifically for exhaust queries and the JS one is for all other find commands. It should be doable to make all commands use the C++ code path that is currently reserved for exhaust queries.&lt;/p&gt;</comment>
                            <comment id="4231798" author="david.storch" created="Mon, 6 Dec 2021 08:45:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=brooke.miller&quot; class=&quot;user-hover&quot; rel=&quot;brooke.miller&quot;&gt;brooke.miller&lt;/a&gt; I&apos;d like to keep this ticket open, but on the Query Execution team&apos;s backlog. I think this remains valuable as a code improvement for a code path which is heavily used in order to test the server. Also, I believe that the changes the Query Execution team did to get rid of OP_QUERY in 5.1 (along with the other legacy op codes) should make this ticket a bit easier.&lt;/p&gt;

&lt;p&gt;Sending to the QE triage queue.&lt;/p&gt;</comment>
                            <comment id="4230092" author="brooke.miller" created="Fri, 3 Dec 2021 18:22:31 +0000"  >&lt;p&gt;The shell is deprecated to external users, so please feel free to re-open if you are a core-db contributor and would like this feature for correctness testing purposes.&lt;/p&gt;</comment>
                            <comment id="1052469" author="xgen-internal-githook" created="Tue, 6 Oct 2015 14:37:22 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;dstorch&apos;, u&apos;name&apos;: u&apos;David Storch&apos;, u&apos;email&apos;: u&apos;david.storch@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20770&quot; title=&quot;Consolidate the two code paths for running find commands in the mongo shell&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20770&quot;&gt;SERVER-20770&lt;/a&gt; garbage collect every 10 MB in itcount() instead of every 10000 documents&lt;/p&gt;

&lt;p&gt;Ensures that the shell will not consume too much memory while using&lt;br/&gt;
DBCommandCursor to iterate a large result set.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a7aa0d48c57a1ce1a82825e5bc6348f88fc10f35&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a7aa0d48c57a1ce1a82825e5bc6348f88fc10f35&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1816654">SERVER-58491</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2033140">SERVER-65955</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2096763">SERVER-68213</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="273453">SERVER-23219</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="455060" name="sv20770.tar.gz" size="8970" author="kevin.cherkauer@mongodb.com" created="Thu, 8 Jun 2023 19:06:48 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 6 Oct 2015 14:37:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        34 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kevin.cherkauer@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            34 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-execution</customfieldvalue>
            <customfieldvalue>brooke.miller@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kevin.cherkauer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrksrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2ion:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7233">QE 2023-05-29</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs9vtb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>