<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:31:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53628] Corrupt wt file, checksum validation levels and integrity check</title>
                <link>https://jira.mongodb.org/browse/SERVER-53628</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I am facing some issues getting mongoDB running stable again after a disk failure and I&apos;am hoping to get some conclusive answers by posting this here.&lt;/p&gt;

&lt;p&gt;We are running the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;MongoDB 3.4.2 (GridFS) with WiredTiger engine 2.9.1&lt;/li&gt;
	&lt;li&gt;This is a single instance approximately 7 TB disk size (I know this is not optimal, but due to lack of resources, we can only deploy 1 instance currently)&lt;/li&gt;
	&lt;li&gt;Database is intensively used (CRD operation, no update operations)&lt;/li&gt;
	&lt;li&gt;Collections can become quite large, so collections are sharded to avoid big files&lt;/li&gt;
	&lt;li&gt;We do not configure the compression ourselves, so by&#160;default, WiredTiger uses Snappy block&#160;compression&#160;for all collections and prefix&#160;compression&#160;for all indexes&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The following happened:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;At some point in time, we faced a disk failure, resulting in disk level data corruption and eventually an unclean mongoDB shutdown&lt;/li&gt;
	&lt;li&gt;After resolving disk I/O issue, we tried to start mongoDB again (recover the data from last clean checkpoint) but this failed due to&#160;read checksum errors. Example error message at startup:&#160;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;WiredTiger error (0) [1608775397:914574][2001:0x7f2365280d40], file:store/index-22521-1925627743013728369.wt,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;WT_SESSION.checkpoint: read checksum error for 4096B block at offset 1343488: block header checksum of 4198219623 doesn&apos;t match expected checksum of 832517012&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Analyzing the &lt;em&gt;*.wt&lt;/em&gt; file revealed corrupt indexes. Deleting and rebuilding them enabled us to start mongoDB again but unfortunately it exited later when inserting data or reading data. Example error message:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;WiredTiger error (0) [1608843541:169607][1978:0x7f137e0f7700], file:store/index-22520-1925627743013728369.wt, WT_CURSOR.insert: read checksum error for 12288B block at offset 446464: block header checksum of 1107817407 doesn&apos;t match expected checksum of 3384971618&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;This suggests more data files are corrupted. Running a repair action on a 7 TB database would take too long, so we decided to restore a backup from a day before (before the disk IO issue was detected).&lt;/li&gt;
	&lt;li&gt;Backups are created daily by copying the underlying data files from a running mongoDB instance. Note that journaling is enabled on the same logical volume.&lt;/li&gt;
	&lt;li&gt;Backup files were restored successfully and mongoDB running fine again.&lt;/li&gt;
	&lt;li&gt;Unfortunately, after 3 days, mongoDB exited again due to a checksum error. This suggests we have restored corrupted data from our backup. Example error message:&#160;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2021-01-02T04:28:59.249+0800 E STORAGE [conn45] WiredTiger error (0) [1609532939:249830][1952:0x7fe688738700], file:store/index-14545-1925627743013728369.wt, WT_CURSOR.search_near: read checksum error for 12288B block at offset 2908160: block header checksum of 0 doesn&apos;t match expected checksum of 4186336817&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;We were able to resolve this particular corruption (deleted) and mongoDB is running again&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Of course, this is not acceptable; this might be a ticking time bomb&#160;as other data blocks might be corrupted and mongoDB would exit whenever they are accessed.&lt;/p&gt;

&lt;p&gt;So a couple of questions I hope to find an answer to:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;To my understanding WiredTiger does some initial integrity checks at Mongo startup. Which checks are those? I suspect collection and index checksums only, is this statement correct? If not, which integrity checks are performed at startup?&lt;/li&gt;
	&lt;li&gt;A block consists of a page header structure, a block header structure and a chunk of data. Snappy compression is the default, meaning that the checksum is not stored in the page header and checksum validation is taken care of by snappy. Is this statement correct?&lt;/li&gt;
	&lt;li&gt;Is there a way we can do block level integrity checks (aka snappy) to ensure data integrity for the whole database. I understand this would be an intrusive operation, but we could do this in a phased approach (e.g per collection, shard, ...) Is any useful documentation, references available concerning these kind of issues? I can imagine we are not the first once suffering from this.&lt;/li&gt;
	&lt;li&gt;I would be extremely grateful for any other suggestion how to perform a complete data integrity check on a running MongoDB. Repair actions would be secondary, the primary goal is to identify possible integrity issues if any.&#160;&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="1583028">SERVER-53628</key>
            <summary>Corrupt wt file, checksum validation levels and integrity check</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="edwin.zhou@mongodb.com">Edwin Zhou</assignee>
                                    <reporter username="tom.decsi@itinq.com">Tom Decsi</reporter>
                        <labels>
                    </labels>
                <created>Thu, 7 Jan 2021 10:01:04 +0000</created>
                <updated>Wed, 22 Jun 2022 15:55:01 +0000</updated>
                            <resolved>Wed, 10 Feb 2021 18:05:01 +0000</resolved>
                                                                    <component>GridFS</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="4335718" author="JIRAUSER1269689" created="Fri, 4 Feb 2022 13:41:39 +0000"  >&lt;p&gt;Hi Brajmohan,&lt;/p&gt;

&lt;p&gt;You can run&#160;db.collection.validate script to identify the corrupt indexes. We eventually dropped all corrupt collections as we could not afford downtime or performance impact. Data loss of in our case older data was accepted. Mongo running fine afterwards ... Probably not the solution you were hoping for ...&lt;/p&gt;

&lt;p&gt;Rgds&lt;/p&gt;

&lt;p&gt;Tom&lt;/p&gt;</comment>
                            <comment id="4335517" author="JIRAUSER1269853" created="Fri, 4 Feb 2022 11:20:13 +0000"  >&lt;p&gt;Hi Tom Decsi,&lt;/p&gt;

&lt;p&gt;We are facing the same issue. How you analyzing the&#160;&lt;em&gt;*.wt&lt;/em&gt;&#160;file to check corrupt indexes names. How you made instance up for Deleting and rebuilding them. We are unable to start the mongodb service.&lt;/p&gt;

&lt;p&gt;Many Thanks&lt;/p&gt;

&lt;p&gt;Braj Mohan&lt;/p&gt;</comment>
                            <comment id="3609805" author="JIRAUSER1257066" created="Wed, 10 Feb 2021 18:02:22 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tom.decsi%40itinq.com&quot; class=&quot;user-hover&quot; rel=&quot;tom.decsi@itinq.com&quot;&gt;tom.decsi@itinq.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Unfortunately we have no implementation available to run --repair on only select collections. &lt;/p&gt;

&lt;p&gt;To avoid a problem like this in the future, it is our strong recommendation to:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Use a &lt;a href=&quot;https://docs.mongodb.com/manual/replication/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;replica set&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Use the &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;most recent version&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Keep up to date &lt;a href=&quot;https://docs.mongodb.com/manual/core/backups/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;backups of your databases&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Have a process in place to restore backups.&lt;/li&gt;
	&lt;li&gt;Follow all &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;production notes&lt;/a&gt;, especially those for &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/#disk-and-storage-systems&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;underlying storage systems&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Schedule and perform regular checks of the integrity of your filesystems and disks.&lt;/li&gt;
	&lt;li&gt;Never manipulate the underlying database files in any way.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Best regards,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3608513" author="JIRAUSER1269689" created="Wed, 10 Feb 2021 07:55:48 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;Thanks for your reply. Yes, we are able to run the db.collection.validate script and several collections were identified as being corrupted. Pls note that this is not preventing Mongo from starting up. Mongo starts fine.&lt;/p&gt;

&lt;p&gt;But whenever data is accessed in on of those collections, Mongo will exit due to WiredTiger panic error, as we have seen before.&lt;/p&gt;

&lt;p&gt;{{&lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongod --repair&lt;/a&gt;&#160;}}is an option, but this would take a whole week according to our estimates. Do you know if we can execute the repair only on corrupted collections instead of the whole database? Or any other options we may consider (besides just dropping the corrupted ones)?&lt;/p&gt;</comment>
                            <comment id="3604531" author="JIRAUSER1257066" created="Mon, 8 Feb 2021 16:01:51 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tom.decsi%40itinq.com&quot; class=&quot;user-hover&quot; rel=&quot;tom.decsi@itinq.com&quot;&gt;tom.decsi@itinq.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We&apos;d love to hear back from you about your disk corruption!&lt;/p&gt;

&lt;p&gt;Were you able to try running &lt;a href=&quot;https://docs.mongodb.com/v3.4/reference/method/db.collection.validate/index.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;db.collection.validate&lt;/a&gt; on the affected collections? After validating collections, I recommend&#160;trying &lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;mongod --repair&lt;/tt&gt;&lt;/a&gt;. This may remove some documents, but it should eliminate any corruption that prevents mongod from starting up.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3576819" author="JIRAUSER1257066" created="Thu, 21 Jan 2021 22:26:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tom.decsi%40itinq.com&quot; class=&quot;user-hover&quot; rel=&quot;tom.decsi@itinq.com&quot;&gt;tom.decsi@itinq.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;MongoDB 3.4 &lt;a href=&quot;https://www.mongodb.com/support-policy&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;reached end of life in January of 2020&lt;/a&gt;. But we can provide limited guidance on this issue. As you&apos;ve identified, this appears to be a disk corruption.&lt;/p&gt;

&lt;p&gt;First, make a complete copy of the database&apos;s $dbpath directory to safeguard so that you can work off of the current $dbpath.&lt;/p&gt;

&lt;p&gt;The best way to look for corruption is to run &lt;a href=&quot;https://docs.mongodb.com/v3.4/reference/method/db.collection.validate/index.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;db.collection.validate&lt;/a&gt; on the affected collections. Index corruption can be solved by reindexing, which you&apos;ve mentioned you&apos;ve done in your steps. After validating collections, I recommend&#160;trying &lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;mongod --repair&lt;/tt&gt;&lt;/a&gt;. This may remove some documents, but it should eliminate any corruption that prevents mongod from starting up.&lt;/p&gt;

&lt;p&gt;Best,&lt;/p&gt;

&lt;p&gt;Edwin&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 21 Jan 2021 22:26:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>brajmohan.sharma@gmail.com</customfieldvalue>
            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tom.decsi@gmail.com</customfieldvalue>
            <customfieldvalue>tom.decsi@itinq.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyobhj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hya33z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hynxqv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>