<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:16:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-8012] multiple nodes in replica set crashed at the same time (E11000 duplicate key error index)</title>
                <link>https://jira.mongodb.org/browse/SERVER-8012</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We&apos;re running a 5-node replica set (including 1 hidden member).  This morning, 3 nodes (including the master) all crashed at the same time!&lt;/p&gt;

&lt;p&gt;The log file for the primary contains:&lt;/p&gt;

&lt;p&gt;Fri Dec 21 15:45:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 4&amp;#93;&lt;/span&gt; ERROR: writer worker caught exception: E11000 duplicate key error index: songza.song_bans.$station_id_1_song_id_1  dup key: { : 1382507, : 10983572 } on: { ts: Timestamp 1356103732000|9, h: -4292312148168796626, v: 2, op: &quot;i&quot;, ns: &quot;songza.song_bans&quot;, o: &lt;/p&gt;
{ _id: ObjectId(&apos;50d4803398896c3d577f54c4&apos;), ban_time: new Date(1356103731894), song_id: 10983572, user_id: 1079677, station_id: 1382507 }
&lt;p&gt; }&lt;br/&gt;
Fri Dec 21 15:45:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 4&amp;#93;&lt;/span&gt;   Fatal Assertion 16360&lt;br/&gt;
0xaf8c41 0xabe223 0x99b83e 0xacc4cd 0xb3ec79 0x7f671505e9ca 0x7f6714405cdd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaf8c41&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xabe223&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x13e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x99b83e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xacc4cd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb3ec79&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x69ca) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f671505e9ca&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f6714405cdd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Fri Dec 21 15:45:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 4&amp;#93;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;***aborting after fassert() failure&lt;/p&gt;


&lt;p&gt;Fri Dec 21 15:45:45 Got signal: 6 (Aborted).&lt;/p&gt;

&lt;p&gt;Fri Dec 21 15:45:45 Backtrace:&lt;br/&gt;
0xaf8c41 0x5586c9 0x7f6714352af0 0x7f6714352a75 0x7f67143565c0 0xabe25e 0x99b83e 0xacc4cd 0xb3ec79 0x7f671505e9ca 0x7f6714405cdd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaf8c41&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5586c9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(+0x33af0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f6714352af0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(gsignal+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f6714352a75&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(abort+0x180) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f67143565c0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xabe25e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x13e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x99b83e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xacc4cd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb3ec79&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x69ca) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f671505e9ca&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f6714405cdd&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Attempts to restart or repair this node have not been successful.  We were able to restart the two crashed secondaries, and one of them eventually stepped up to become the primary.&lt;/p&gt;

&lt;p&gt;I&apos;m attaching log files from the three nodes in question.  db1a is the hidden member.  db1c was the primary (and is the one which will not come back up).  db1d was (and continues to be) one of the secondaries.&lt;/p&gt;
</description>
                <environment>$ cat /etc/lsb-release &lt;br/&gt;
DISTRIB_ID=Ubuntu&lt;br/&gt;
DISTRIB_RELEASE=10.04&lt;br/&gt;
DISTRIB_CODENAME=lucid&lt;br/&gt;
DISTRIB_DESCRIPTION=&amp;quot;Ubuntu 10.04.4 LTS&amp;quot;&lt;br/&gt;
&lt;br/&gt;
$ uname -a&lt;br/&gt;
Linux db1a.songza.com 2.6.32-347-ec2 #52-Ubuntu SMP Fri Jul 27 14:38:36 UTC 2012 x86_64 GNU/Linux</environment>
        <key id="60148">SERVER-8012</key>
            <summary>multiple nodes in replica set crashed at the same time (E11000 duplicate key error index)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="andre.defrere@mongodb.com">Andre de Frere</assignee>
                                    <reporter username="roysmith">Roy Smith</reporter>
                        <labels>
                    </labels>
                <created>Fri, 21 Dec 2012 17:00:17 +0000</created>
                <updated>Wed, 10 Dec 2014 23:18:46 +0000</updated>
                            <resolved>Tue, 19 Aug 2014 18:41:53 +0000</resolved>
                                    <version>2.2.1</version>
                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="291505" author="andre.defrere" created="Sun, 17 Mar 2013 22:23:24 +0000"  >&lt;p&gt;Hi Roy,&lt;/p&gt;

&lt;p&gt;Given that, I&apos;ll set this ticket to closed and we can continue working in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9004&quot; title=&quot;mongodb crashed with E11000 duplicate key error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9004&quot;&gt;&lt;del&gt;SERVER-9004&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andr&#233;&lt;/p&gt;</comment>
                            <comment id="291158" author="roysmith" created="Sun, 17 Mar 2013 04:00:59 +0000"  >&lt;p&gt;Hmmm, just noticed this is waiting for input from me.  We were able to track down and delete the duplicate data, and upgraded to 2.2.2 on all our servers.  Everything we running fine for a couple of months, but we just had another crash this evening which looks like a similar problem.  See &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9004&quot; title=&quot;mongodb crashed with E11000 duplicate key error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9004&quot;&gt;&lt;del&gt;SERVER-9004&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="287914" author="andre.defrere" created="Wed, 13 Mar 2013 07:13:58 +0000"  >&lt;p&gt;Hi Roy,&lt;/p&gt;

&lt;p&gt;Checking in with you on this subject.  Do you have any responses to the previously asked questions?&lt;/p&gt;</comment>
                            <comment id="259078" author="andre.defrere" created="Wed, 6 Feb 2013 08:58:35 +0000"  >&lt;p&gt;Hi Roy,&lt;/p&gt;

&lt;p&gt;There could be a few things happening here.  First, the backup you restored could have had the corruption already in the oplog.  While the 2.2.2 release should prevent further instances of the corruption from appearing, it will not be able to repair corruption currently existing in the Oplog.  Secondly., it could be possible that there is actually duplicate keys in the index that you are trying to insert in to.  Have you checked the data for duplicate key values?&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andr&#233;&lt;/p&gt;</comment>
                            <comment id="234118" author="roysmith" created="Tue, 8 Jan 2013 13:57:05 +0000"  >&lt;p&gt;I don&apos;t think this problem has been solved by 2.2.2.&lt;/p&gt;

&lt;p&gt;We&apos;ve installed 2.2.2 everywhere.  Have not had any further crashes in production, but I just restored a dump from production into a development instance and got a similar crash.  The command I ran was:&lt;/p&gt;

&lt;p&gt;$ mongorestore mongo-db1a.songza.com-2013-01-06 --oplogReplay&lt;/p&gt;

&lt;p&gt;the last few lines of output are:&lt;/p&gt;


&lt;p&gt;Tue Jan  8 11:55:53     Creating index: { key: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, ns: &quot;songza.situation_user_data&quot;, name: &quot;&lt;em&gt;id&lt;/em&gt;&quot; }&lt;br/&gt;
Tue Jan  8 11:55:54     Creating index: { key: &lt;/p&gt;
{ user_id: 1 }
&lt;p&gt;, ns: &quot;songza.situation_user_data&quot;, name: &quot;user_id_1&quot; }&lt;br/&gt;
Tue Jan  8 11:58:28 mongo-db1a.songza.com-2013-01-06/songza/song_bans.bson&lt;br/&gt;
Tue Jan  8 11:58:28     going into namespace &lt;span class=&quot;error&quot;&gt;&amp;#91;songza.song_bans&amp;#93;&lt;/span&gt;&lt;br/&gt;
Tue Jan  8 11:58:31             10635400/37529218       28%     (bytes)&lt;br/&gt;
Tue Jan  8 11:58:34             17416800/37529218       46%     (bytes)&lt;br/&gt;
Tue Jan  8 11:58:37             24190360/37529218       64%     (bytes)&lt;br/&gt;
Tue Jan  8 11:58:40             30520760/37529218       81%     (bytes)&lt;br/&gt;
Tue Jan  8 11:58:43             37228360/37529218       99%     (bytes)&lt;br/&gt;
457669 objects found&lt;br/&gt;
Tue Jan  8 11:58:43     Creating index: { key: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, ns: &quot;songza.song_bans&quot;, name: &quot;&lt;em&gt;id&lt;/em&gt;&quot; }&lt;br/&gt;
Tue Jan  8 11:58:48     Creating index: { key: &lt;/p&gt;
{ station_id: 1, song_id: 1 }
&lt;p&gt;, unique: true, ns: &quot;songza.song_bans&quot;, name: &quot;station_id_1_song_id_1&quot;, background: true }&lt;br/&gt;
Tue Jan  8 11:58:53 ERROR: Error creating index songza.song_bans: 11000 err: &quot;E11000 duplicate key error index: songza.song_bans.$station_id_1_song_id_1  dup key: { : 1382507, : 10983572 }&quot;&lt;br/&gt;
Aborted (core dumped)&lt;br/&gt;
roy@dev3:/mnt/songza$ &lt;/p&gt;

&lt;p&gt;This is the same collection (song_bans) and the same error&lt;/p&gt;

&lt;p&gt;dup key: { : 1382507, : 10983572 } &lt;/p&gt;

&lt;p&gt;on which mongo crashed earlier.  There is apparently some piece of bad data in this collection which is not being handled properly, and 2.2.2 didn&apos;t fix that.&lt;/p&gt;</comment>
                            <comment id="225413" author="roysmith" created="Thu, 27 Dec 2012 17:14:44 +0000"  >&lt;p&gt;OK, we&apos;ll try upgrading to 2.2.2.&lt;/p&gt;

&lt;p&gt;We haven not made any attempts to replicate the issue.&lt;/p&gt;

&lt;p&gt;We have already recovered the secondary by resyncing.&lt;/p&gt;</comment>
                            <comment id="225029" author="andre.defrere" created="Thu, 27 Dec 2012 05:03:22 +0000"  >&lt;p&gt;Hi Roy,&lt;/p&gt;

&lt;p&gt;It appears you have an even number of voting nodes in your replica set.  This is not a recommended strategy, as discussed in the &lt;a href=&quot;http://docs.mongodb.org/manual/administration/replication-architectures/#sets-with-four-or-more-members&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;documentation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The stack trace that appears in the logs, and in the description of this ticket could be caused by the idempotency issues that were resolved in 2.2.2.  You can see more discussion on this subject at &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7551&quot; title=&quot;_id Unique key violation during initial sync&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-7551&quot;&gt;&lt;del&gt;SERVER-7551&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7186&quot; title=&quot;Batched oplog application increases frequency of idempotency violations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-7186&quot;&gt;&lt;del&gt;SERVER-7186&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6671&quot; title=&quot;oplog is not strictly idempotent when unique index is present&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6671&quot;&gt;&lt;del&gt;SERVER-6671&lt;/del&gt;&lt;/a&gt;.  Version 2.2.2 should prevent any new issues for idempotency in the OpLog, but you will still need to manually recover the failed secondaries (by fully resyncing, or by doing a file system copy of the db path).  Do you have a test environment where you can try 2.2.2?  Can you cause this problem to happen again at will?&lt;/p&gt;</comment>
                            <comment id="221462" author="roysmith" created="Fri, 21 Dec 2012 17:17:53 +0000"  >&lt;p&gt;I turns out I described the situation incorrectly.  I originally thought db1c was the primary.  That&apos;s not correct.  It was db1e.   Apparently, when multiple secondaries crashed, db1e demoted itself to a secondary:&lt;/p&gt;

&lt;p&gt;roy@dev1:~$ mongo db1e.songza.com&lt;br/&gt;
MongoDB shell version: 2.2.1&lt;br/&gt;
connecting to: db1e.songza.com/test&lt;br/&gt;
songza-new:SECONDARY&amp;gt;&lt;/p&gt;

&lt;p&gt;and then came back to being the primary once the other two secondaries came back up.  We also have an arbiter in the mix:&lt;/p&gt;

&lt;p&gt;songza-new:PRIMARY&amp;gt; rs.conf()&lt;br/&gt;
{&lt;br/&gt;
	&quot;_id&quot; : &quot;songza-new&quot;,&lt;br/&gt;
	&quot;version&quot; : 70,&lt;br/&gt;
	&quot;members&quot; : [&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 3,&lt;br/&gt;
			&quot;host&quot; : &quot;db1a.songza.com:27017&quot;,&lt;br/&gt;
			&quot;priority&quot; : 0,&lt;br/&gt;
			&quot;hidden&quot; : true,&lt;br/&gt;
			&quot;tags&quot; : &lt;/p&gt;
{
				&quot;role&quot; : &quot;reporting&quot;
			}
&lt;p&gt;		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 4,&lt;br/&gt;
			&quot;host&quot; : &quot;db1b.songza.com:27017&quot;,&lt;br/&gt;
			&quot;tags&quot; : &lt;/p&gt;
{
				&quot;feed_test&quot; : &quot;A&quot;,
				&quot;role&quot; : &quot;production&quot;
			}
&lt;p&gt;		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 6,&lt;br/&gt;
			&quot;host&quot; : &quot;db1e.songza.com:27017&quot;,&lt;br/&gt;
			&quot;priority&quot; : 5,&lt;br/&gt;
			&quot;tags&quot; : &lt;/p&gt;
{
				&quot;role&quot; : &quot;production&quot;
			}
&lt;p&gt;		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 7,&lt;br/&gt;
			&quot;host&quot; : &quot;db1c.songza.com:27017&quot;,&lt;br/&gt;
			&quot;tags&quot; : &lt;/p&gt;
{
				&quot;feed_test&quot; : &quot;B&quot;,
				&quot;role&quot; : &quot;production&quot;
			}
&lt;p&gt;		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 8,&lt;br/&gt;
			&quot;host&quot; : &quot;db1d.songza.com:27017&quot;,&lt;br/&gt;
			&quot;priority&quot; : 0,&lt;br/&gt;
			&quot;hidden&quot; : true,&lt;br/&gt;
			&quot;tags&quot; : &lt;/p&gt;
{
				&quot;role&quot; : &quot;production&quot;
			}
&lt;p&gt;		},&lt;/p&gt;
		{
			&quot;_id&quot; : 9,
			&quot;host&quot; : &quot;db-arbiter.songza.com:27017&quot;,
			&quot;arbiterOnly&quot; : true
		}
&lt;p&gt;	]&lt;br/&gt;
}&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="22777" name="db1a-mongodb.log" size="68282774" author="roysmith" created="Fri, 21 Dec 2012 17:00:17 +0000"/>
                            <attachment id="22778" name="db1c-mongodb.log" size="1861983" author="roysmith" created="Fri, 21 Dec 2012 17:00:17 +0000"/>
                            <attachment id="22779" name="db1d-mongodb.log" size="1632201" author="roysmith" created="Fri, 21 Dec 2012 17:00:17 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 27 Dec 2012 05:03:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 48 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 48 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andre.defrere@mongodb.com</customfieldvalue>
            <customfieldvalue>roysmith</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrndnr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrl7qf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>37594</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;I certainly hope not&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0m93:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>