<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:29:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32254] mongod crashes at higher application loads (multiple mongod version)</title>
                <link>https://jira.mongodb.org/browse/SERVER-32254</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We are experiencing crashes in both Mongo v.3.2.16 and v3.4.5 when operating at a higher application load.&lt;/p&gt;

&lt;p&gt;We believe the crashes are similar or the same, but mongod logging is slightly different across the specified versions.&lt;/p&gt;

&lt;p&gt;I can specify any additional information you may require beyond what is in the log files.&lt;/p&gt;

&lt;p&gt;Logs attached.&lt;/p&gt;</description>
                <environment>Ubuntu 14.04.5 LTS&lt;br/&gt;
XFS filesystem for data, local, and journal&lt;br/&gt;
VMWare ESXi 6.0.0, build 5224934&lt;br/&gt;
48GB ram&lt;br/&gt;
6 cores</environment>
        <key id="470579">SERVER-32254</key>
            <summary>mongod crashes at higher application loads (multiple mongod version)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="mark.agarunov">Mark Agarunov</assignee>
                                    <reporter username="jesse.beard@captiveaire.com">Jesse Beard</reporter>
                        <labels>
                    </labels>
                <created>Mon, 11 Dec 2017 15:02:52 +0000</created>
                <updated>Wed, 21 Mar 2018 04:32:56 +0000</updated>
                            <resolved>Mon, 12 Feb 2018 18:53:18 +0000</resolved>
                                    <version>3.2.16</version>
                    <version>3.4.5</version>
                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1803205" author="mark.agarunov" created="Mon, 12 Feb 2018 18:53:18 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jesse.beard%40captiveaire.com&quot; class=&quot;user-hover&quot; rel=&quot;jesse.beard@captiveaire.com&quot;&gt;jesse.beard@captiveaire.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the additional information. My apologies for the delay in response, unfortuantely we have not been able to diagnose a root cause for the errors you are seeing. The errors in the most recent files indicate data corruption at the disk or filesystem level however as Bruce mentioned, the fact that the errors seem unrelated to each other points to a hardware or underlying issue outside of mongodb. If this is still an issue for you, we can take another look if there have been additional crashes logged, but as I don&apos;t see anything to indicate a bug in MongoDB I&apos;ve closed this ticket for the time being. &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1752993" author="jesse.beard@captiveaire.com" created="Fri, 15 Dec 2017 14:10:21 +0000"  >&lt;p&gt;Bruce,&lt;/p&gt;

&lt;p&gt;We have experienced the crash again and I have uploaded full logs and diagnostics.data as requested.  The logs represent server restart after initial data sync to the crash event.  Logs stop at the crash.  Instance ran for ~48 hours before crashing.&lt;/p&gt;

&lt;p&gt;Uploaded file:  &lt;tt&gt;captiveaire-crash-data-logs-2017-12-14_&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32254&quot; title=&quot;mongod crashes at higher application loads (multiple mongod version)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32254&quot;&gt;&lt;del&gt;SERVER-32254&lt;/del&gt;&lt;/a&gt;.zip&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;The only environmental difference is that we did upgrade the replica set to 3.4.10.&lt;/p&gt;

&lt;p&gt;Please let me know what else I can provide for you.&lt;/p&gt;</comment>
                            <comment id="1750003" author="bruce.lucas@10gen.com" created="Tue, 12 Dec 2017 18:47:43 +0000"  >&lt;p&gt;Thanks for mentioning those tickets. I don&apos;t see any relationship at the moment, but I will keep them in mind.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Do you know of any specific items to look at for VMWare ESXi beyond what is listed in the mongo best practices for production that may cause issues such as what I have posted?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I don&apos;t have any specific recommendations. I would look for errors or other unusual messages correlated in time with the crashes. If you would like to upload the guest syslog and the host VMWare system log files I can take take a look as well.&lt;/p&gt;</comment>
                            <comment id="1748885" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 20:38:35 +0000"  >&lt;p&gt;Bruce,&lt;/p&gt;

&lt;p&gt;Thank you for the response.  I will upload additional logs and diagnostic data for you.  The reason I am posting logs from two different versions of Mongo is that we thought the issue was isolated to the 3.4.x version range, so we slowly started backing down the replica set thinking the issue would subside once we reached what used to be a stable version for us which was 3.2.16.  However, 3.2.16 is also crashing as indicated in the logs, but with a different crash log vs. what is logged by 3.4.5.  I assumed maybe the difference was due to logging differences between the versions, but you are saying it is not.&lt;/p&gt;

&lt;p&gt;Do you see possible similarities to the below?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31121&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-31121&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17713&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-17713&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I do realize the linked issues are dealing with zlib compression.  Our replica set is not running any network compression, obviously not on 3.2.16, and is using the default of &quot;none&quot; when running 3.4.5 on the same replica set.&lt;/p&gt;

&lt;p&gt;Do you know of any specific items to look at for VMWare ESXi beyond what is listed in the mongo best practices for production that may cause issues such as what I have posted?&lt;/p&gt;</comment>
                            <comment id="1748821" author="bruce.lucas@10gen.com" created="Mon, 11 Dec 2017 19:49:04 +0000"  >&lt;p&gt;Hi Jesse,&lt;/p&gt;

&lt;p&gt;The three failures in the logs you attached have very different signatures, and none of them match known problems:&lt;/p&gt;

&lt;p&gt;In mongo1-backtrace-v3.4.5-2017-12-05.log: WT panic due to encountering bad data&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-12-05T03:50:35.095Z E STORAGE  [conn81395] WiredTiger error (0) [1512445835:92730][5812:0x7f9764a42700], file:local/collection/12-387512412807676176.wt, WT_CURSOR.insert: encountered an illegal file format or internal value&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-12-05T03:50:35.100Z E STORAGE  [conn81395] WiredTiger error (-31804) [1512445835:100374][5812:0x7f9764a42700], file:local/collection/12-387512412807676176.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after fassert() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;In mongo1-backtrace-v3.2.16-2017-12-05.log: seg fault in __wt_page_in_func&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-12-08T22:34:17.019Z F -        [conn33489] Got signal: 11 (Segmentation fault).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(__wt_page_in_func+0x1B7) [0x1a0a5a7]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(__wt_row_search+0x68F) [0x1a2ce7f]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(__wt_btcur_insert+0xB33) [0x19fa423]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;In mongo2-backtrace-v3.2.16-2017-12-11.log: seg fault in QueryPlanner::cacheDataFromTaggedTree&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-12-11T14:44:16.793Z F -        [conn42405] Got signal: 11 (Segmentation fault).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN5mongo12QueryPlanner23cacheDataFromTaggedTreeEPKNS_15MatchExpressionERKSt6vectorINS_10IndexEntryESaIS5_EEPPNS_18PlanCacheIndexTreeE+0x42) [0xe5f1c2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN5mongo12QueryPlanner23cacheDataFromTaggedTreeEPKNS_15MatchExpressionERKSt6vectorINS_10IndexEntryESaIS5_EEPPNS_18PlanCacheIndexTreeE+0x34D) [0xe5f4cd]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN5mongo12QueryPlanner4planERKNS_14CanonicalQueryERKNS_18QueryPlannerParamsEPSt6vectorIPNS_13QuerySolutionESaIS9_EE+0x19DF) [0xe60fff]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(+0xA13C20) [0xe13c20]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm+0x74) [0xe15124]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EENS_12PlanExecutor11YieldPolicyE+0x7B) [0xe15cbb]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Given that you are seeing multiple different types of failures in two different versions of mongod, I would first suspect a hardware issue, or an issue with the virtualization software. Can you check system logs on both the host and guest operating systems for errors?&lt;/p&gt;

&lt;p&gt;Also, if you can upload all available mongod log files and the archived content of the $dbpath/diagnostic.data directory to &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/0a98362c-0f4f-40a8-b664-e42791991ef6.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this secure upload portal&lt;/a&gt; we can take a look to see if we an spot some commonality between the failures.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1748548" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 16:59:47 +0000"  >&lt;p&gt;This is a 3 data node replica set.&lt;/p&gt;</comment>
                            <comment id="1748405" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 15:26:52 +0000"  >&lt;p&gt;Mongo VMs have host affinity in VMWare and have not been vMontioned during the time of these crashes.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="172984" name="mongo1-backtrace-v3.2.16-2017-12-05.log" size="213461" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 15:02:01 +0000"/>
                            <attachment id="172983" name="mongo1-backtrace-v3.4.5-2017-12-05.log" size="23898" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 15:02:01 +0000"/>
                            <attachment id="172985" name="mongo2-backtrace-v3.2.16-2017-12-11.log" size="140377" author="jesse.beard@captiveaire.com" created="Mon, 11 Dec 2017 15:09:39 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 11 Dec 2017 19:49:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>jesse.beard@captiveaire.com</customfieldvalue>
            <customfieldvalue>mark.agarunov</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htlz7r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htdl5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htllbz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>