<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:48:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-59889] Add compatibility tests for $group pushdown behavior</title>
                <link>https://jira.mongodb.org/browse/SERVER-59889</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This work adds a JS test suite that compares the behavior between execution of $group in the classic engine vs $group pushdown into sbe. We could consider a randomized testing approach for data generation, not for queries. Randomized queries makes it too hard to investigate a failure.&lt;/p&gt;

&lt;p&gt;The main goal here is to expose incompatible behaviors between the behaviors of $group in classic engine and ones of $group in SBE as soon as possible and make them easy to investigate.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1872000">SERVER-59889</key>
            <summary>Add compatibility tests for $group pushdown behavior</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="yoonsoo.kim@mongodb.com">Yoon Soo Kim</assignee>
                                    <reporter username="eric.cox@mongodb.com">Eric Cox</reporter>
                        <labels>
                            <label>sbe</label>
                    </labels>
                <created>Fri, 10 Sep 2021 23:45:20 +0000</created>
                <updated>Tue, 18 Jan 2022 23:10:42 +0000</updated>
                            <resolved>Tue, 18 Jan 2022 23:10:42 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="4299226" author="JIRAUSER1258488" created="Tue, 18 Jan 2022 23:09:53 +0000"  >&lt;p&gt;I&apos;d rather wait for how the test working group discussion goes. I believe that this ticket is closely related to the discussion because what I was trying to do was to define a combinatorial test framework and on top of that, define test matrix for $group. Will resolve this as &quot;Won&apos;t Fix&quot;.&lt;/p&gt;

&lt;p&gt;But please, feel free to reopen this ticket when the necessity arises.&lt;/p&gt;</comment>
                            <comment id="4077693" author="JIRAUSER1258488" created="Wed, 22 Sep 2021 15:29:51 +0000"  >&lt;p&gt;Sure &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ethan.zhang&quot; class=&quot;user-hover&quot; rel=&quot;ethan.zhang&quot;&gt;ethan.zhang&lt;/a&gt;, will add it.&lt;/p&gt;</comment>
                            <comment id="4076282" author="JIRAUSER1257640" created="Wed, 22 Sep 2021 04:05:14 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoonsoo.kim&quot; class=&quot;user-hover&quot; rel=&quot;yoonsoo.kim&quot;&gt;yoonsoo.kim&lt;/a&gt;&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;no missing value vs missing value(s) vs invalid value(s) for a type/field&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Should we add &quot;undefined&quot; to this as well?&#160;&lt;/p&gt;</comment>
                            <comment id="4074913" author="JIRAUSER1257640" created="Tue, 21 Sep 2021 17:01:44 +0000"  >&lt;p&gt;Just discussed this in standup. The combinations are deterministic, but the test data is not. Yoonsoo will make changes to make it deterministic.&lt;/p&gt;</comment>
                            <comment id="4073391" author="JIRAUSER1258488" created="Tue, 21 Sep 2021 04:37:00 +0000"  >&lt;p&gt;Sure, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=steve.la&quot; class=&quot;user-hover&quot; rel=&quot;steve.la&quot;&gt;steve.la&lt;/a&gt;. I think basically there are 3 groups of tests for $group.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;group-by key(s) tests: jstests/aggregation/sources/group/*.js&lt;/li&gt;
	&lt;li&gt;accumulator tests: jstests/aggregation/accumulators/*.js&lt;/li&gt;
	&lt;li&gt;specific tests for bugs or behaviors: many tests in jstests/aggregation/bugs directory are for $group and accumulators but not all of them.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Yes, existing $group test suites eventually validate expected results for test cases that we are explicitly testing if we run those tests with SBE turned on. The questions are whether they are enough and it would be easy to investigate test failures.&lt;/p&gt;

&lt;p&gt;I think we have the same kind of test suites for find() but AFAIK, we&apos;ve had a long tail of correctness issues for SBE find() many of which have been found by fuzzer tests. Failures with the fuzzer test are notorious for being hard to investigate among query team. I believe that the one reason why we have a long tail for SBE correctness for find() is that fuzzer tests are inherently randomized tests for queries. So, it may take a long time to reveal all possible correctness issues and also it&apos;s very time-consuming to make a minimal repro out of a generated failed fuzzer test file, which is a starting point of fixing an issue.&lt;/p&gt;

&lt;p&gt;This ticket is an attempt to expose such incompatibilities as early as possible by generating full $group spec combinations out of various test dimensions and running those $group specs on both the classic engine and the SBE engine and compare the SBE engine result to the classic engine result. This ticket is also an attempt to help engineers to investigate test failures by attaching a repro script for each test failure. Usually a js test stops running as soon as the first failed test case is encountered but this test suite will keep running until all test combinations of $group spec are completed and report all failed test cases with repro scripts at the end of execution.&lt;/p&gt;

&lt;p&gt;While I have been working on this ticket, I found two incompatibilities though it&apos;s still in the early stage of development. One is related to an error code for a case that a numeric expression like $add fails with non-numeric value at runtime and the other is $sum result for SBE is different from one for classic engine for a certain input data. I think the first one is ok to include to known incompatibilities and the second one is worth investigating further.&lt;/p&gt;

&lt;p&gt;I&apos;m not sure whether this explanation makes sense to you and you think this test suite would be doable and useful. Please, let me know.&lt;/p&gt;

&lt;p&gt;If we think that existing $group test suites and agg fuzzer tests are enough, I can move on to work on other items like improving $group performance.&#160;&lt;/p&gt;</comment>
                            <comment id="4066890" author="JIRAUSER1258488" created="Thu, 16 Sep 2021 19:36:25 +0000"  >&lt;p&gt;Found a bug.&lt;/p&gt;</comment>
                            <comment id="4064257" author="JIRAUSER1258488" created="Wed, 15 Sep 2021 21:09:58 +0000"  >&lt;p&gt;Test dimensions to consider are:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;no accumulator vs one accumulator vs multiple accumulators&lt;/li&gt;
	&lt;li&gt;null _id vs scalar _id vs document _id with one field vs document _id with multiple fields&lt;/li&gt;
	&lt;li&gt;a simple field path vs a complex field path vs expression for each accumulator&lt;/li&gt;
	&lt;li&gt;no missing value vs missing value(s) vs invalid value(s) for a type/field vs undefined for a field&lt;/li&gt;
	&lt;li&gt;data types: int vs long vs decimal vs small string vs long string vs date vs timestamp vs array vs mixed-type array vs boundary values (MIN/MAX per type, NaN, infinity) vs mixed types&lt;/li&gt;
	&lt;li&gt;default collator vs explicit collator&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;For dimension 1 &amp;amp; 2 &amp;amp; 3, we could generate $group stages. This list should cover as many combinations as possible, avoiding redundant test cases. For 4 &amp;amp; 5 &amp;amp; 6, we can leverage random data generation and still full combinations should be covered.&lt;/p&gt;

&lt;p&gt;In the context of this test suite,&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;not being supported in SBE is NOT an error.&lt;/li&gt;
	&lt;li&gt;we should report an easy repro step (or steps) with generated data, the classic engine&apos;s result, the SBE&apos;s result, the winning plan, SBE PlanStage tree, and $group stage when a failure happens and the test suite should make it as easy as possible to investigate a test failure.&lt;/li&gt;
	&lt;li&gt;The classic engine&apos;s result is always considered as &quot;correct expected behavior&quot; and this test suite can&apos;t verify whether the classic engine&apos;s behavior is correct or not.&lt;/li&gt;
	&lt;li&gt;if a SBE result is different from the classic engine&apos;s result in any way, it&apos;s an error.&lt;/li&gt;
	&lt;li&gt;Whenever we decide that a difference is an intended behavior, we add a known difference as a custom compare function so that the comparison can succeed.&lt;/li&gt;
	&lt;li&gt;This test suite only verifies $group pipeline stage behavior.&lt;/li&gt;
	&lt;li&gt;This test suite does not test whether $group is pushed down or not.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1876358">SERVER-60030</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 15 Sep 2021 21:09:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 3 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-60030'>SERVER-60030</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2267</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>yoonsoo.kim@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 3 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eric.cox@mongodb.com</customfieldvalue>
            <customfieldvalue>ethan.zhang@mongodb.com</customfieldvalue>
            <customfieldvalue>yoonsoo.kim@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i018vz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2bz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4719">QE 2021-09-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i00v1b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>