<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:54:52 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-62345] Consider re-evaluating fuzzer testing philosophy</title>
                <link>https://jira.mongodb.org/browse/SERVER-62345</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Our fuzzers are designed to alert us of any change in behavior between different versions or configurations of mongo. They currently generate a BFG every time a query fails on one version and succeeds on another. Often, these are invalid queries that contain some improper value or operation. However, the offending part of the query may be optimized away on one side before it has a chance to error; or it may trigger a different error first because it executes in a different order. &lt;/p&gt;

&lt;p&gt;The problem increased sharply when we turned the SBE engine on by default. Since SBE re-implements MQL expressions from scratch, there are many differences in undocumented behavior from non-SBE versions of mongo. So far, we have addressed these BFs by adding workarounds to the fuzzer to suppress BFGs in specific scenarios. The list of &lt;a href=&quot;https://github.com/10gen/jstestfuzz/commit/fac2666e540d301ab88276154d5c823ef7f19665&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;these&lt;/a&gt; &lt;a href=&quot;https://github.com/10gen/jstestfuzz/blob/05a6a5ea290a65fab9e817b550231433a5bacd05/src/fuzzers/agg/agg_compare_results.js#L230-L299&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;workarounds&lt;/a&gt; in the fuzzers is growing long. &lt;/p&gt;

&lt;p&gt;We dedicate a lot of time and resources to these BFs. Many of them are connected to queries that customers should not be running, since they are invalid MQL. Our documentation does not promise consistent behavior for invalid queries.&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;How likely are customers to ever suffer as a result of these behavior changes in practice?&lt;/li&gt;
	&lt;li&gt;How much value does it provide to spend developer time resolving these types of differences?&lt;/li&gt;
	&lt;li&gt;What behavior changes do we want to prioritize identifying and fixing?&lt;/li&gt;
	&lt;li&gt;How can we most efficiently change the fuzzers to reflect these priorities?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;One proposal is to stop the fuzzers from generating BFGs when one side errors and the other doesn&#8217;t, or when the error messages don&#8217;t match, since this usually happens with queries that are already invalid. In this world, the fuzzer would only generate a BFG when both sides succeed, but return different results for the same query.&lt;/p&gt;

&lt;p&gt;There is a risk that a new version of mongo could introduce a bug where a valid query suddenly errors. This is something the fuzzers would currently catch, but we should discuss how a problem like this would be identified if we were to make this change.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1958171">SERVER-62345</key>
            <summary>Consider re-evaluating fuzzer testing philosophy</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-query-execution">Backlog - Query Execution</assignee>
                                    <reporter username="jennifer.peshansky@mongodb.com">Jennifer Peshansky</reporter>
                        <labels>
                    </labels>
                <created>Tue, 4 Jan 2022 16:14:17 +0000</created>
                <updated>Tue, 6 Dec 2022 00:41:53 +0000</updated>
                            <resolved>Wed, 9 Feb 2022 18:51:05 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="4345881" author="kyle.suarez" created="Wed, 9 Feb 2022 18:51:05 +0000"  >&lt;p&gt;Closing this as Done, as we had several meetings and have a path forward with regard to ignoring certain classes of errors.&lt;/p&gt;</comment>
                            <comment id="4275515" author="max.hirschhorn@10gen.com" created="Wed, 5 Jan 2022 04:09:05 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We dedicate a lot of time and resources to these BFs. Many of them are connected to queries that customers should not be running, since they are invalid MQL.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I&apos;d be curious to see tabulated numbers on this. On the Sharding team there had been a lot of anecdotal evidence that any failure in the sharding&amp;#95;csrs&amp;#95;continuous&amp;#95;config&amp;#95;stepdown Evergreen task was a testing&amp;#45;only problem and would get solved by excluding the offending test. I was surprised to find&#8212;and reported in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59891&quot; title=&quot;Replace the coverage from sharding_continuous_config_stepdown.yml and then delete the test suite&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59891&quot;&gt;SERVER-59891&lt;/a&gt;&#8212;that for every 2 testing&amp;#45;only issues there has still been 1 real code issue. Not a good ratio by any means but much better than what I had anticipated discovering.&lt;/p&gt;

&lt;p&gt;The list of questions in the description to evaluate are spot&amp;#45;on. Testing is about managing risk and that means weighing the severity and likelihood in the context of time scarcity and diminishing returns.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;One proposal is to stop the fuzzers from generating BFGs when one side errors and the other doesn&#8217;t, or when the error messages don&#8217;t match, since this usually happens with queries that are already invalid. In this world, the fuzzer would only generate a BFG when both sides succeed, but return different results for the same query.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Has it ever been the case that the query results from the configuration which didn&apos;t error are actually nonsensical? In other words, has the differential testing from the fuzzer ever detected valid but missing error checking?&lt;/p&gt;

&lt;p&gt;There are testing techniques beyond differential testing which could be applied to databases (e.g. metamorphic testing, pivoted query synthesis). Section 6. Related Work of &lt;a href=&quot;https://arxiv.org/pdf/2007.08292.pdf&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://arxiv.org/pdf/2007.08292.pdf&lt;/a&gt; might be a reasonable place to start for anyone who is interested. It&apos;ll be a fun research project to apply them to aggregation pipelines and not only find queries! I also enjoyed reading Section 3.4. Corner Cases and Limitations as I found it basically summarizes the complexities we&apos;ve hit with the fuzzers over the years. With any of these testing techniques, there&apos;s a tradeoff between the kinds of bugs it can detect due to limiting the inputs/queries/transformations/etc. and how much of an idealized model must be implemented elsewhere to verify the results. The ones mentioned in the paper still accommodate our lack of a ground truth for the system behavior (aka a test oracle) though.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1922508">SERVER-61389</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 5 Jan 2022 04:09:05 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-execution</customfieldvalue>
            <customfieldvalue>jennifer.peshansky@mongodb.com</customfieldvalue>
            <customfieldvalue>kyle.suarez@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0fvl3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2ixb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0fhqf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>