<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:50:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-38852] Failing $merge can leave zombie writes in the cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-38852</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;If an $merge fails on one shard, the process of cleaning up another shard&apos;s writes can leave some writes cancelled, but still very much alive in the network. Upon any error all other shards participating in an aggregation will be interrupted. Because &lt;a href=&quot;https://github.com/mongodb/mongo/blob/62076130c3321f842f3ce4fb426aefc078074206/src/mongo/s/async_requests_sender.cpp#L109&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;an interrupted call to &lt;tt&gt;AsyncRequestsSender::next()&lt;/tt&gt; will cancel the callbacks and return without waiting for the response of inserts or updates we&apos;ve already sent&lt;/a&gt;, we can end up returning control to the user while there are still active writes which were generated on their behalf. This is causing problems in some of our tests, but is generally surprising and probably undesirable.&lt;/p&gt;

&lt;p&gt;In more detail, consider a sharded cluster which has collections &lt;tt&gt;test.input&lt;/tt&gt; and &lt;tt&gt;test.output&lt;/tt&gt;, both sharded by {_id: 1}. Suppose that &lt;tt&gt;test.input&lt;/tt&gt; is distributed like so:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;chunks [0,5), [15, 20) on shard 1&lt;/li&gt;
	&lt;li&gt;chunk [5, 15) on shard 2&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;and suppose &lt;tt&gt;test.output&lt;/tt&gt; is distributed like so:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;chunk [0, 10) on shard 1&lt;/li&gt;
	&lt;li&gt;chunk [10, 20) on shard 2&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If we had 20 documents in test.input with _ids =[0,20) and we executed&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.input.aggregate([{$merge: {to: &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;output&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, mode: &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;insertDocuments&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}}])&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;then we would end up with each shard sending 5 documents each to itself and the other shard: [0,5) would go from shard 1 to itself, [5,10) would go from shard 2 to shard 1, [10, 15) would go from shard 2 to itself, and [15, 20) would go from shard 1 to shard 2.&lt;/p&gt;

&lt;p&gt;Then suppose that when performing the inserts from shard 1, some of them return with an error, maybe a duplicate key. The aggregate on shard 1 will return with an error to mongos. When that error gets back to mongos, mongos will send a killCursors to the aggregate (with the $merge) running on shard 2. Upon receiving that killCursors, the $merge stage &lt;em&gt;may&lt;/em&gt; be in a state where it has scheduled both the writes but hasn&apos;t heard back from one or both of them. In this scenario, the &lt;tt&gt;AsyncRequestsSender&lt;/tt&gt; will cancel its callbacks as described above and then the aggregate and its cursor will be destroyed and we will respond to mongos and then to the user with the error without verifying that the writes have finished (successfully or not).&lt;/p&gt;

&lt;p&gt;In some of our tests, we perform an $merge which is expected to fail after partially completing, then issue a subsequent &lt;tt&gt;db.target.remove({});&lt;/tt&gt; and expecting that &lt;tt&gt;remove&lt;/tt&gt; to get rid of everything in the collection. Because there can be lingering writes from the $merge, this remove may succeed but there will very shortly be some documents inserted into the collection, causing a subsequent assertion to fail.&lt;/p&gt;


&lt;p&gt;I&apos;ve also attached &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/205526/205526_repro_no_agg.js&quot; title=&quot;repro_no_agg.js attached to SERVER-38852&quot;&gt;repro_no_agg.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; which demonstrates that an unordered bulk insert can also leave an active insert in the cluster since it also uses the AsyncRequestSender, though this is less surprising and I think not a bug. It is not possible to leave an active operation without being interrupted. Triggering an interruption due to an error like DuplicateKey is unique to the aggregation system.&lt;/p&gt;</description>
                <environment></environment>
        <key id="665876">SERVER-38852</key>
            <summary>Failing $merge can leave zombie writes in the cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-query-execution">Backlog - Query Execution</assignee>
                                    <reporter username="charlie.swanson@mongodb.com">Charlie Swanson</reporter>
                        <labels>
                            <label>query-44-grooming</label>
                    </labels>
                <created>Fri, 4 Jan 2019 22:09:12 +0000</created>
                <updated>Fri, 8 Sep 2023 15:20:56 +0000</updated>
                                                                            <component>Aggregation Framework</component>
                    <component>Write Ops</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2410997" author="charlie.swanson" created="Fri, 6 Sep 2019 17:27:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt; I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43198&quot; title=&quot;Zombie writes from failing $merge should not be able to re-create a collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43198&quot;&gt;SERVER-43198&lt;/a&gt; to track the work that is actually feasible. I&apos;m not sure what we should do with this ticket but it does look like all the linked BFs are instances of a &lt;tt&gt;remove({})&lt;/tt&gt; followed by lingering inserts/upserts rather than a drop and implicit recreate which &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43198&quot; title=&quot;Zombie writes from failing $merge should not be able to re-create a collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43198&quot;&gt;SERVER-43198&lt;/a&gt; would help with. We haven&apos;t seen any of these failures recently though so it&apos;s worth discussion whether we should mark them trivial, investigate why they stopped happening, or something else.&lt;/p&gt;</comment>
                            <comment id="2409286" author="charlie.swanson" created="Thu, 5 Sep 2019 17:12:04 +0000"  >&lt;p&gt;Charlie to do some JIRA paperwork tomorrow to make it clear that we&apos;re interested in fixing zombie writes re-creating collections, but are not pursuing a fix for the more generic problem of leftover work from a cancelled operation.&lt;/p&gt;</comment>
                            <comment id="2385118" author="charlie.swanson" created="Fri, 23 Aug 2019 20:59:27 +0000"  >&lt;p&gt;Didn&apos;t get as much time to work on this today since I was mostly preoccupied by BF-14452. I did try to run the reproduction script but it no longer works. I&apos;d like to keep looking into that next time.&lt;/p&gt;</comment>
                            <comment id="2382517" author="charlie.swanson" created="Thu, 22 Aug 2019 13:56:02 +0000"  >&lt;p&gt;I did get to work on this last Friday. I have a patch that I think will address the problem and hopefully is something we can do for the 4.2 branch as well. It&apos;s a little unpolished at this point and I still haven&apos;t reproduced this problem so I have no evidence that it actually works, but I&apos;ve attached what I did so far in  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/228077/228077_zombie_killer.patch&quot; title=&quot;zombie_killer.patch attached to SERVER-38852&quot;&gt;zombie_killer.patch&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. &lt;/p&gt;

&lt;p&gt;After just attaching that I see there are reproduction scripts attached to this ticket (which I apparently made??) so I&apos;d like to start there and continue tomorrow.&lt;/p&gt;</comment>
                            <comment id="2363665" author="charlie.swanson" created="Thu, 8 Aug 2019 21:16:30 +0000"  >&lt;p&gt;After discussing in-person, we believe there&apos;s probably a bug where $out and $merge are setting allowImplicitCreate when they should not be. I&apos;m assigning to myself to investigate.&lt;/p&gt;</comment>
                            <comment id="2345940" author="jason.carey" created="Fri, 26 Jul 2019 17:20:18 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We were thinking of circumventing maxTimeMs and interrupts.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;That&apos;s going to be a non-starter.  It&apos;s enormously problematic to make uninterruptible opCtxs, especially one&apos;s waiting for unbounded amounts of time.  Even if we want to wait somewhere else farther up the stack (and return different errors to users), you have to unwind your stack (to allow for the release of locks that might be needed to make progress on stepDown).&lt;/p&gt;</comment>
                            <comment id="2345889" author="jacob.evans" created="Fri, 26 Jul 2019 17:01:54 +0000"  >&lt;p&gt;After discussing this we are now considering adding an option to aggregate (and potentially all operations in the long-term) to cause the client operation itself to wait for all children to die before returning control. This would be very useful to our tests since we&apos;d obviate the need to check currentOp to ensure the operation is really over. We also think it might be useful since it provides these optional guarantees to other clients if they&apos;re willing to wait. Let us know what you think.&lt;/p&gt;

&lt;p&gt;With that out of the way, let me answer open questions:&lt;br/&gt;
The top-level $merge does correspond to the user&apos;s command and we would like to wait.&lt;br/&gt;
We were thinking of circumventing maxTimeMs and interrupts.&lt;br/&gt;
Holding on to sockets seems like the simple implementation.&lt;br/&gt;
If a connection dies, returning an error through the parent operation seems appropriate and sufficient.&lt;br/&gt;
The ability to actively kill child ops would be an improvement but just waiting is all we have planned at the moment. The goal is to at least have the parent operation correctly reflect that it isn&apos;t completely done.&lt;br/&gt;
Although a timeframe would be sufficient to make the tests fail less, we are wary of any time-based approaches in testing since they usually lead to sporadic failures.&lt;/p&gt;</comment>
                            <comment id="2339240" author="jason.carey" created="Tue, 23 Jul 2019 16:41:29 +0000"  >&lt;p&gt;A couple of questions:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;How are you going to wait around for your children?  Is it entirely by holding onto sockets with outstanding requests?
	&lt;ul&gt;
		&lt;li&gt;Would it be a problem if one of those connections had a networking error (but kept persisting on the other side)&lt;/li&gt;
		&lt;li&gt;Do you intend to actively kill those ops?  Or just wait for them to complete?&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;How important is it that you know that all writes initiated by a command are complete before you return?  Would it suffice to have those writes cancelled within a time frame?
	&lt;ul&gt;
		&lt;li&gt;If so, you might be able to leverage the markKillOnClientDisconnect code to make these kinds of subsidiary operations automatically clean up on cancellation.  That would get you eventual, guaranteed, cleanup&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2338474" author="kaloian.manassiev" created="Tue, 23 Jul 2019 11:28:06 +0000"  >&lt;p&gt;The way I understand is that this is a manifestation of the generic problem about how to cancel a &quot;tree&quot; of distributed operations in the cluster. Your proposal to leave the &lt;tt&gt;$merge&lt;/tt&gt; parent active until all its children have completed seems like the architecturally correct way to achieve this - each parent is responsible for killing and waiting for clean-up of its children.&lt;/p&gt;

&lt;p&gt;A couple of considerations:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Does the top-level &lt;tt&gt;$merge&lt;/tt&gt; correspond to the user command against MongoS and is it possible, if the operation context holding it is killed, to give it out to an internal thread to complete the joining of the children instead of doing it on the user thread?&lt;/li&gt;
	&lt;li&gt;Since ClusterWriter is not cancelable cleanly (i.e., if you have a write batch executing, there is no &lt;tt&gt;killCursors&lt;/tt&gt;-equivalent through the cluster writer to clean its state on the remote node, only to stop waiting for it), is your plan to just keep waiting for it to return? Because this will not work if you are doing it on the client&apos;s operation context and it gets interrupted or its deadline expires.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2323789" author="jacob.evans" created="Fri, 12 Jul 2019 19:51:52 +0000"  >&lt;p&gt;The solution we&apos;ve settled on after discussion is to have a killed $merge persist in a dead state until all children are complete. Although this doesn&apos;t actually allow the command to be swiftly stopped, it leaves the parent $merge around as an indicator that the operation has not yet gone away. At least we would be more accurately reporting the current state of the cluster. This is also something our tests can rely on.&lt;/p&gt;

&lt;p&gt;We&apos;ve decided to solve this in the general case for other bulk operations if possible and if little extra work.&lt;/p&gt;

&lt;p&gt;In order to facilitate this we foresee the need to change the ClusterWriter and possibly the AsyncRequestHandler so that parent operations can track the lifetime of their children.&lt;/p&gt;</comment>
                            <comment id="2106190" author="charlie.swanson" created="Fri, 4 Jan 2019 22:10:40 +0000"  >&lt;p&gt;I&apos;m leaving this in Needs Triage because I&apos;m not sure whether we should work around this in our tests or add some customization to the AsyncRequestsSender to avoid leaving things around when interrupted for agg&apos;s purposes. As of this writing, I think we should probably fix the server.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="918697">SERVER-43198</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2436277">SERVER-80853</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="960278">SERVER-43851</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="205527" name="repro_agg_duplicate_key.js" size="1905" author="charlie.swanson@mongodb.com" created="Fri, 4 Jan 2019 22:03:06 +0000"/>
                            <attachment id="205526" name="repro_no_agg.js" size="2428" author="charlie.swanson@mongodb.com" created="Fri, 4 Jan 2019 22:03:17 +0000"/>
                            <attachment id="228077" name="zombie_killer.patch" size="15334" author="charlie.swanson@mongodb.com" created="Thu, 22 Aug 2019 13:54:58 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Mar 2019 21:28:20 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 22 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>jordi.serra-torrens@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 22 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-execution</customfieldvalue>
            <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>jacob.evans@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>mira.carey@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hui8tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2h27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;I&apos;ve first reproduced the error as described in the description in the attached  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/205527/205527_repro_agg_duplicate_key.js&quot; title=&quot;repro_agg_duplicate_key.js attached to SERVER-38852&quot;&gt;repro_agg_duplicate_key.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. I don&apos;t know why, but running this reproduction with&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;python buildscripts/resmoke.py --suites=sharding_continuous_config_stepdown repro_agg_duplicate_key.js&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;reproduces the failure more frequently than running with just &lt;tt&gt;--suites=sharding&lt;/tt&gt;. I think possibly I am imagining this.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huhv2n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>