<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:58:02 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-21666] Small stall during concurrent insert/update</title>
                <link>https://jira.mongodb.org/browse/SERVER-21666</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In some test case, Non periodic small stall is observed.&lt;/p&gt;

&lt;p&gt;First time, I have thought wiredtiger checkpoint cause this small stall.&lt;br/&gt;
But, it&apos;s nothing to do with checkpoint (After some test and code eval, I found that wiredtiger table lock could cause this stall).&lt;br/&gt;
And this small stall makes a lot of slow query(over 150ms).&lt;br/&gt;
During this stall period, __wt_spin_lock(pthread_mutex_lock) took over 4~8 milliseconds.&lt;/p&gt;

&lt;p&gt;__wt_schema_open_table :: elapsed 21 us&lt;br/&gt;
__wt_spin_lock         :: elapsed:8394 us (pthread_mutex_lock     :: session.open_cursor -&amp;gt; table creation(0))&lt;br/&gt;
__wt_schema_add_table  :: elapsed 8436 us (need table lock)&lt;br/&gt;
__wt_schema_get_table  :: elapsed 8437 us&lt;br/&gt;
__wt_curtable_open     :: elapsed 8439 us&lt;br/&gt;
__wt_open_cursor       :: elapsed 8456 us&lt;/p&gt;


&lt;p&gt;My test case is really simple, Just make 1000 client thread and doing update(upsert:true) and total documents is about 400000. &lt;br/&gt;
And server have processed about 6~7000 update/sec. System load was under 1.0&lt;/p&gt;

&lt;p&gt;12 core cpu, 32GB memory, 2.6.32-573.7.1.el6.centos.plus.x86_64&lt;br/&gt;
Looks like disk is nothing to do with this issue (I&apos;ve tested NVME, SAS+RAID, SATA)&lt;/p&gt;

&lt;p&gt;Current implementation, table lock is implemented with WT_SPINLOCK.&lt;br/&gt;
What do you think about changing it as WT_RWLOCK ?&lt;/p&gt;</description>
                <environment></environment>
        <key id="241080">SERVER-21666</key>
            <summary>Small stall during concurrent insert/update</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="sunguck.lee@gmail.com">&#50500;&#45208; &#54616;&#47532;</reporter>
                        <labels>
                    </labels>
                <created>Wed, 25 Nov 2015 10:11:15 +0000</created>
                <updated>Wed, 8 Feb 2023 15:40:58 +0000</updated>
                            <resolved>Fri, 5 Feb 2016 00:41:09 +0000</resolved>
                                    <version>3.0.1</version>
                    <version>3.0.2</version>
                    <version>3.0.3</version>
                    <version>3.0.4</version>
                    <version>3.0.5</version>
                    <version>3.0.6</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1164498" author="ramon.fernandez" created="Fri, 5 Feb 2016 00:41:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, we haven&apos;t heard back from you for some time, so we&apos;re closing this ticket. If this is still an issue for you please provide the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21666?focusedCommentId=1130603&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1130603&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;information requested above&lt;/a&gt; so we can investigate further.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1130603" author="david.hows" created="Mon, 11 Jan 2016 00:09:15 +0000"  >&lt;p&gt;Hi Matt,&lt;/p&gt;

&lt;p&gt;Sorry for the delay in getting back to you over the holiday period.&lt;/p&gt;

&lt;p&gt;Would you be able to run the following generic MongoDB diagnostics script, it collects a number of MongoDB releated pieces of information and system diagnostics. I&apos;d like to confirm what filesystem the data storage is hosted on, but having all the other details available would be valuable. &lt;/p&gt;

&lt;p&gt;To execute this script please run the following and email me the resulting output (please don&apos;t attach it to the ticket).&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;wget https://raw.githubusercontent.com/mongodb/support-tools/master/mdiag/mdiag.sh&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;sudo bash mdiag.sh SERVER-21666&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Additionally, would you be able to run the following small script to gather system statistics during another run of your script and then attach the resulting &lt;tt&gt;ss.log&lt;/tt&gt; file to the ticket along with the server logfile.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:true}))); sleep(1000)}&quot; &amp;gt;ss.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This should give me an insight as to what actions are being undertaken by the server when the delays are seen.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="1114650" author="sunguck.lee@gmail.com" created="Tue, 15 Dec 2015 09:07:59 +0000"  >&lt;p&gt;Hi.&lt;/p&gt;

&lt;p&gt;I attached slow query graph.&lt;br/&gt;
You will see the spike of red line (slow query counter) in the graph.&lt;/p&gt;

&lt;p&gt;And this is sample slow queries. (I changed some part of query)&lt;/p&gt;

&lt;p&gt;2015-12-15T17:17:22.510+0900 I COMMAND  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15072&amp;#93;&lt;/span&gt; command DB.$cmd command: update { update: &quot;COLL&quot;, ordered: true, updates: [ { q: &lt;/p&gt;
{ _id: 160228 }
&lt;p&gt;, u: { $set: &lt;/p&gt;
{ id: 140738, ... }
&lt;p&gt;, $currentDate: &lt;/p&gt;
{ updated_at: true }
&lt;p&gt; }, upsert: true } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 locks:{ Global: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Database: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Collection: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; } } 135ms&lt;br/&gt;
2015-12-15T17:17:22.511+0900 I COMMAND  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn14547&amp;#93;&lt;/span&gt; command DB.$cmd command: update { update: &quot;COLL&quot;, ordered: true, updates: [ { q: &lt;/p&gt;
{ _id: 183208 }
&lt;p&gt;, u: { $set: &lt;/p&gt;
{ id: 133677, ...}
&lt;p&gt;, $currentDate: &lt;/p&gt;
{ updated_at: true }
&lt;p&gt; }, upsert: true } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 locks:{ Global: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Database: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Collection: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; } } 154ms&lt;br/&gt;
2015-12-15T17:17:22.510+0900 I COMMAND  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15119&amp;#93;&lt;/span&gt; command DB.$cmd command: update { update: &quot;COLL&quot;, ordered: true, updates: [ { q: &lt;/p&gt;
{ _id: 228814 }
&lt;p&gt;, u: { $set: &lt;/p&gt;
{ id: 4016, ... }
&lt;p&gt;, $currentDate: &lt;/p&gt;
{ updated_at: true }
&lt;p&gt; }, upsert: true } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 locks:{ Global: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Database: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; }, Collection: { acquireCount: &lt;/p&gt;
{ w: 3 }
&lt;p&gt; } } 159ms&lt;/p&gt;</comment>
                            <comment id="1114649" author="sunguck.lee@gmail.com" created="Tue, 15 Dec 2015 09:06:06 +0000"  >&lt;p&gt;Slow query graph&lt;/p&gt;</comment>
                            <comment id="1114582" author="sunguck.lee@gmail.com" created="Tue, 15 Dec 2015 05:55:24 +0000"  >&lt;p&gt;Hi David Hows.&lt;/p&gt;

&lt;p&gt;Slow query limit was 100ms in my test mongodb. Did you also set 100ms threshold for slow query logging ?&lt;br/&gt;
And your above comment, looks like some slow processing &quot;2015-12-15 12:36:40 ~ 2015-12-15 12:36:41&quot; is showed.&lt;br/&gt;
I think there have been a lot of slow query at that time (If slow query threshold is 100ms).&lt;/p&gt;

&lt;p&gt;&quot;Small stall&quot;, I used this word as 50~100ms level processing stop , Not 1~5 seconds level stop.&lt;br/&gt;
When this issue happen on our service, system load is increased 20 (according to linux uptime command) and cpu usage is increasing for a short time.&lt;/p&gt;

&lt;p&gt;I will attach mongostat and log file which include slow query logs.&lt;/p&gt;
</comment>
                            <comment id="1114522" author="david.hows" created="Tue, 15 Dec 2015 03:40:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, I have run your workload for an hour, and only seen a few long operations within the MongoDB logfiles.&lt;/p&gt;

&lt;p&gt;Would you be able to provide me with the exact details of how you categorise a &quot;small stall&quot;? What specific indicators do you use? What what outputs are you using to see the stalls? &lt;/p&gt;

&lt;p&gt;I&apos;d like to ensure I am looking for the same issue that you have stated.&lt;/p&gt;</comment>
                            <comment id="1114437" author="david.hows" created="Tue, 15 Dec 2015 01:52:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, is the following indicative of the &quot;small stall&apos;s&quot; you are seeing?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-12-15 12:36:39            6561          6561             0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-12-15 12:36:40            2507          2507             0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-12-15 12:36:41            2328          2328             0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-12-15 12:36:42            6710          6710             0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1114351" author="alexander.gorrod" created="Tue, 15 Dec 2015 00:09:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; Could you get the reproducer for this running, and turn it into a js test script?&lt;/p&gt;</comment>
                            <comment id="1113576" author="keith.bostic" created="Mon, 14 Dec 2015 15:33:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, I have not yet run your test program &amp;#8211; I&apos;m working on that, and will be in touch as soon as I have more information.&lt;/p&gt;</comment>
                            <comment id="1111791" author="sunguck.lee@gmail.com" created="Fri, 11 Dec 2015 06:54:49 +0000"  >&lt;p&gt;Hi Keith.&lt;/p&gt;

&lt;p&gt;No, It happens to me continuously during test.&lt;br/&gt;
I don&apos;t know the detailed WiredTiger internal story, but I saw the creation a bunch of WiredTiger session internally during test.&lt;br/&gt;
And this time, WiredTiger&apos;s table lock contention was getting high.&lt;/p&gt;

&lt;p&gt;Have you ran the test program I attached ?&lt;/p&gt;</comment>
                            <comment id="1110602" author="keith.bostic" created="Thu, 10 Dec 2015 13:51:27 +0000"  >&lt;p&gt;Hi, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, thanks for your report.&lt;/p&gt;

&lt;p&gt;It looks to me like the stalls you&apos;re seeing are on the open-cursor path, that is, they happen when the collections are first opened. Is that the case, do the stalls disappear once the collections have been opened?&lt;/p&gt;</comment>
                            <comment id="1098125" author="sunguck.lee@gmail.com" created="Wed, 25 Nov 2015 10:16:50 +0000"  >&lt;p&gt;Mongo config &amp;amp; table stats&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="99777" name="SmallStall.tar.gz" size="5731" author="matt.lee" created="Wed, 25 Nov 2015 10:11:15 +0000"/>
                            <attachment id="99778" name="mongo_config_table_stats.txt" size="5650" author="matt.lee" created="Wed, 25 Nov 2015 10:16:50 +0000"/>
                            <attachment id="102230" name="slow-queries.png" size="70294" author="matt.lee" created="Tue, 15 Dec 2015 09:06:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 4 Dec 2015 21:20:23 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 1 week, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 1 week, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>keith.bostic@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>sunguck.lee@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrknm7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsembb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Run attatched java test program&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsfeun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>