<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:23:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[DRIVERS-1652] Clarify events.json and result.json produced by workload executor</title>
                <link>https://jira.mongodb.org/browse/DRIVERS-1652</link>
                <project id="10980" key="DRIVERS">Drivers</project>
                    <description>&lt;p&gt;This issue has been repurposed to clarify expected output for &lt;tt&gt;events.json&lt;/tt&gt; and &lt;tt&gt;results.json&lt;/tt&gt; in all cases:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;tt&gt;results.json&lt;/tt&gt; and &lt;tt&gt;events.json&lt;/tt&gt; should always be created by the workload executor (even if the test runner propagates an error not caught in a loop)&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;events.json&lt;/tt&gt; must always report arrays for &lt;tt&gt;events&lt;/tt&gt;, &lt;tt&gt;errors&lt;/tt&gt;, and &lt;tt&gt;failures&lt;/tt&gt;. Those arrays may be empty. If any events, errors, or failures are obtained from the entity map, they will be appended to those arrays. If the unified test runner propagates an error (e.g. not caught by a loop), the workload executor is expected to report the error/failure and append it to an array accordingly.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;numErrors&lt;/tt&gt; and &lt;tt&gt;numFailures&lt;/tt&gt; in &lt;tt&gt;results.json&lt;/tt&gt; will always report the size of the respective array in &lt;tt&gt;events.json&lt;/tt&gt; and never be unset (-1).&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;numSuccesses&lt;/tt&gt; and &lt;tt&gt;numIterations&lt;/tt&gt; in &lt;tt&gt;results.json&lt;/tt&gt; may be unset (-1) if their respective values cannot be obtained from the entity map.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;These changes will be incorporated into the workload executor spec document and the validation tests will be revised accordingly.&lt;/p&gt;

&lt;p&gt;For additional context, this issue&apos;s original summary and description follows:&lt;/p&gt;

&lt;hr /&gt;

&lt;h4&gt;&lt;a name=&quot;Incorrectassertionsforresult.jsoninValidateWorkloadExecutor&quot;&gt;&lt;/a&gt;Incorrect assertions for result.json in ValidateWorkloadExecutor&lt;/h4&gt;

&lt;p&gt;&lt;a href=&quot;https://mongodb-labs.github.io/drivers-atlas-testing/spec-workload-executor.html#behavioral-description&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Behavioral Description: #4&lt;/a&gt; states:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If the unified test runner raises an error while executing the workload, the error MUST be reported using the same format as errors handled by the unified test runner, as described in the unified test runner specification under the loop operation. Errors handled by the workload executor MUST be included in the calculated (and reported) error count.&lt;/p&gt;

&lt;p&gt;If the unified test runner reports a failure while executing the workload, the failure MUST be reported using the same format as failures handled by the unified test runner, as described in the unified test runner specification under the loop operation. Failures handled by the workload executor MUST be included in the calculated (and reported) failure count. If the driver&#8217;s unified test runner is intended to handle all failures internally, failures that propagate out of the unified test runner MAY be treated as errors by the workload executor.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;If the &lt;tt&gt;loop&lt;/tt&gt; operation does not store errors or failures in an entity, those exceptions are expected to interrupt the loop and propagate to the test runner. My understanding of the text above is that the workload executor is expected to capture and report those errors in the same manner &amp;#8211; and thus product &lt;tt&gt;results.json&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;I&apos;m confused as to why the &lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/blob/master/tests/validator-numFailures-not-captured.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validator-numFailures-not-captured.yml&lt;/a&gt; test expects &lt;tt&gt;-1&lt;/tt&gt; to be reported for &lt;tt&gt;numFailures&lt;/tt&gt; in that case (and likewise for &lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/blob/master/tests/validator-numErrors-not-captured.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validator-numErrors-not-captured.yml&lt;/a&gt;). If anything, &lt;tt&gt;ValidateWorkloadExecutor&apos;s.test_num_failures_not_captured&lt;/tt&gt; should expect &lt;tt&gt;1&lt;/tt&gt; for &lt;tt&gt;numFailures&lt;/tt&gt; (and possibly &lt;tt&gt;-1&lt;/tt&gt; for &lt;tt&gt;numErrors&lt;/tt&gt;) since an uncaught failure will abort the loop on its first iteration and the workload executor should report that single failure on its own.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://mongodb-labs.github.io/drivers-atlas-testing/spec-workload-executor.html#behavioral-description&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Behavior Description: #8&lt;/a&gt; states:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;MUST calculate the aggregate counts of errors (numErrors) and failures (numFailures) from the error and failure lists. If the errors or failures were not reported by the test runner, such as because the respective options were not specified in the test scenario, the workload executor MUST use -1 as the value for the respective counts.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This does not agree with the expectations for ValidateWorkloadExecutor&apos;s &quot;simple test&quot;, which does &lt;em&gt;not&lt;/em&gt; specify &lt;tt&gt;storeErrorsAsEntity&lt;/tt&gt; or &lt;tt&gt;storeFailuresAsEntity&lt;/tt&gt; but later asserts that &lt;tt&gt;results.json&lt;/tt&gt; does not have any &lt;tt&gt;-1&lt;/tt&gt; values. Per the quoted text, a workload executor should use -1 for the error/failure counts.&lt;/p&gt;

&lt;p&gt;This issue extends to other ValidateWorkloadExecutor tests, which specify either &lt;tt&gt;storeErrorsAsEntity&lt;/tt&gt; or &lt;tt&gt;storeFailuresAsEntity&lt;/tt&gt; (but not both). For example, &lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/blob/master/tests/validator-numErrors.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validator-numErrors.yml&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/blob/master/tests/validator-numFailures-as-errors.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validator-numFailures-as-errors.yml&lt;/a&gt; only use &lt;tt&gt;storeErrorsAsEntity&lt;/tt&gt;. I would expect &lt;tt&gt;results.json&lt;/tt&gt; to produce an actual count for &lt;tt&gt;numErrors&lt;/tt&gt; (based on the size of the &lt;tt&gt;errors&lt;/tt&gt; array in &lt;tt&gt;events.json&lt;/tt&gt;) and leave &lt;tt&gt;numFailures&lt;/tt&gt; unset (i.e. -1); however, this conflicts with the assertions in ValidateWorkloadExecutor.run_test:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        if any(val &amp;lt; 0 for val in stats.values()):&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            self.fail(&quot;The workload executor reported incorrect execution &quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                      &quot;statistics. Reported statistics MUST NOT be negative.&quot;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I&apos;m not sure what needs to change here, nor do I understand how existing implementations are currently passing the ValidateWorkloadExecutor tests. Does the documented behavior for the workload executor need to be changed, or are the assertions in ValidateWorkloadExecutor incorrect?&lt;/p&gt;</description>
                <environment></environment>
        <key id="1672915">DRIVERS-1652</key>
            <summary>Clarify events.json and result.json produced by workload executor</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="13453" iconUrl="https://jira.mongodb.org/images/icons/status_generic.gif" description="">Implementing</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="jmikola@mongodb.com">Jeremy Mikola</assignee>
                                    <reporter username="jmikola@mongodb.com">Jeremy Mikola</reporter>
                        <labels>
                    </labels>
                <created>Wed, 7 Apr 2021 14:00:01 +0000</created>
                <updated>Wed, 1 Sep 2021 20:53:10 +0000</updated>
                                                                <component>Atlas Testing</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="3731147" author="xgen-internal-githook" created="Wed, 21 Apr 2021 18:20:40 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jeremy Mikola&apos;, &apos;email&apos;: &apos;jmikola@gmail.com&apos;, &apos;username&apos;: &apos;jmikola&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1652&quot; title=&quot;Clarify events.json and result.json produced by workload executor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1652&quot;&gt;DRIVERS-1652&lt;/a&gt;: Clarify events.json and result.json produced by workload executor (#120)&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1652&quot; title=&quot;Clarify events.json and result.json produced by workload executor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1652&quot;&gt;DRIVERS-1652&lt;/a&gt;: Clarify JSON produced by workload executor&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This rewrites portions of the workload executor spec to clearly define how events.json and results.json are produced. This also improves instructions for when the test runner propagates and error/failures, which must then be reported by the workload executor in the same format.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1652&quot; title=&quot;Clarify events.json and result.json produced by workload executor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1652&quot;&gt;DRIVERS-1652&lt;/a&gt;: Make ValidateWorkloadExecutor consistent with spec&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Removes the placeholder stats if results.json cannot be read. Per the spec, the workload executor should always produce results.json (and events.json).&lt;/p&gt;

&lt;p&gt;Always assert presence and types of required fields in results.json and events.json. Improve events.json assertions for errors and failures.&lt;/p&gt;

&lt;p&gt;Add numErrors-as-failures test.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Always remove events.json before spawning workload executor&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Fix wrong comment wording&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Co-authored-by: Andreas Braun &amp;lt;alcaeus@users.noreply.github.com&amp;gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/commit/481a6c0b3323280276d593cf5cf54e26958ec6f2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb-labs/drivers-atlas-testing/commit/481a6c0b3323280276d593cf5cf54e26958ec6f2&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3717837" author="jmikola@gmail.com" created="Wed, 14 Apr 2021 14:02:07 +0000"  >&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb-labs/drivers-atlas-testing/pull/120&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb-labs/drivers-atlas-testing/pull/120&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10620">
                    <name>Issue split</name>
                                            <outwardlinks description="split to">
                                        <issuelink>
            <issuekey id="1865682">PHPLIB-714</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1685940">RUBY-2587</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865677">CSHARP-3826</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865676">CXX-2363</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865678">GODRIVER-2144</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865679">NODE-3584</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1685938">JAVA-4114</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865675">CDRIVER-4144</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865680">MOTOR-816</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865681">PYTHON-2892</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1865683">RUST-1010</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                            <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10257" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Documentation Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10250"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10951" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Driver Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10748"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_23952" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Driver Compliance</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<style type='text/css'>
         #scriptField, #scriptField *{
                border: 1px solid black;
            }

            #scriptField{
                border-collapse: collapse;
            }

            #scriptField td {
                text-align: center; /* Center-align text in table cells */
            }

            #scriptField td.key {
                text-align: left; /* Left-align text in the Key column */
            }

            #scriptField a {
                text-decoration: none; /* Remove underlines from links */
                border: none; /* Remove border from links */
            }
            
            /* Add green background color to cells with FixVersion */
            #scriptField td.hasFixVersion {
                background-color: #00FF00; /* Green color code */
            }

            /* Center-align the first row headers */
            #scriptField th {
                text-align: center;
            }
        </style>
<table id='scriptField'>
  <tr>
    <th>Key</th>
    <th>Status/Resolution</th>
    <th>FixVersion</th>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/JAVA-4114'>JAVA-4114</a>
    </td>
    <td>Backlog</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/RUBY-2587'>RUBY-2587</a>
    </td>
    <td>Done</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/CDRIVER-4144'>CDRIVER-4144</a>
    </td>
    <td>Won't Do</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/CXX-2363'>CXX-2363</a>
    </td>
    <td>Backlog</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/CSHARP-3826'>CSHARP-3826</a>
    </td>
    <td>Backlog</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/GODRIVER-2144'>GODRIVER-2144</a>
    </td>
    <td>Backlog</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/NODE-3584'>NODE-3584</a>
    </td>
    <td>Backlog</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/MOTOR-816'>MOTOR-816</a>
    </td>
    <td>Duplicate</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/PYTHON-2892'>PYTHON-2892</a>
    </td>
    <td>Duplicate</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/PHPLIB-714'>PHPLIB-714</a>
    </td>
    <td>Duplicate</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/RUST-1010'>RUST-1010</a>
    </td>
    <td>Duplicate</td>
    <td class=''></td>
  </tr>
  <tr>
    <td class='key'>
      <a href='https://jira.mongodb.org/browse/SWIFT-1342'>SWIFT-1342</a>
    </td>
    <td>Won't Do</td>
    <td class=''></td>
  </tr>
</table>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyomzb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>