<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:20:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29230] Journal files accumulating on ReplicaSet Secondary</title>
                <link>https://jira.mongodb.org/browse/SERVER-29230</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;we have a replica set composed by a primary, a secondary and an arbiter. we deployed it with automation-agent from mongodb cloud manager.&lt;br/&gt;
everything is ok on the primary but we noticed that the free disk space on the secondary in constantly decreasing, checking the filesystem we discovered that the journal directory is the culprit. &lt;br/&gt;
we performed a full resync of the secondary, but after 24h the journal directory occupies 380Gb (we have 485Gb of data).  see attachment of the free disk space graph over 24h (primary on the left, secondary on the right).&lt;/p&gt;

&lt;p&gt;currently there are 3926 journal files in the journal directory. (see attachment)&lt;br/&gt;
the primary is not accumulating journal files and behaves as expected.&lt;/p&gt;</description>
                <environment></environment>
        <key id="383890">SERVER-29230</key>
            <summary>Journal files accumulating on ReplicaSet Secondary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="sue.loverso@mongodb.com">Susan LoVerso</assignee>
                                    <reporter username="alberto.ornaghi@gmail.com">Alberto Ornaghi</reporter>
                        <labels>
                    </labels>
                <created>Tue, 16 May 2017 12:50:10 +0000</created>
                <updated>Mon, 26 Feb 2018 17:25:54 +0000</updated>
                            <resolved>Mon, 26 Feb 2018 17:25:54 +0000</resolved>
                                    <version>3.4.4</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1816177" author="thomas.schubert" created="Mon, 26 Feb 2018 17:25:40 +0000"  >&lt;p&gt;reopening to change resolution to clarify that this work was completed in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3327&quot; title=&quot;Checkpoints can hang if time runs backward&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3327&quot;&gt;&lt;del&gt;WT-3327&lt;/del&gt;&lt;/a&gt; and the issue been resolved in MongoDB 3.2.17 and 3.4.6.&lt;/p&gt;</comment>
                            <comment id="1576223" author="alberto.ornaghi@gmail.com" created="Sun, 21 May 2017 16:41:45 +0000"  >&lt;p&gt;just for reference (if needed by someone else in the future) doing this:&lt;/p&gt;

&lt;p&gt;&quot;echo 2dd1ce17-079e-403c-b352-a1921ee207ee &amp;gt; /sys/bus/vmbus/drivers/hv_util/unbind&quot;&lt;/p&gt;

&lt;p&gt;will resolve the &quot;Time has been changed&quot; problem.  &lt;br/&gt;
hoping it will not trigger the WT bug anymore.&lt;/p&gt;</comment>
                            <comment id="1576148" author="alberto.ornaghi@gmail.com" created="Sun, 21 May 2017 06:20:11 +0000"  >&lt;p&gt;it took 4.5 hours to recover, but now it seems ok (210 Mb of journal). it&apos;s rebuilding the index in foreground so i have to wait some more time...&lt;br/&gt;
after the secondary is up i will do it on the primary.&lt;/p&gt;

&lt;p&gt;the timesyncd daemon was already stopped. but the &quot;Time has been changed&quot; message appears even in that situation.&lt;br/&gt;
it seems s problem with Azure Hyper-V:&lt;br/&gt;
&lt;a href=&quot;https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635?comments=all&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635?comments=all&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;unfortunately i only have to wait for a kernel patch for this...  the perfect storm. &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; &lt;/p&gt;</comment>
                            <comment id="1576119" author="sue.loverso" created="Sun, 21 May 2017 00:39:14 +0000"  >&lt;p&gt;There is a WiredTiger verbose option that will give progress for recovery.  You can turn it on by adding this:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;--wiredTigerEngineConfigString=&apos;,verbose=(recovery_progress)&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;However, the checkpoint after recovery is where your issue is and the potential hang.  Can you turn off the time daemon or change its setting so that it updates much less frequently, particularly during this recovery process?&lt;/p&gt;</comment>
                            <comment id="1576101" author="alberto.ornaghi@gmail.com" created="Sat, 20 May 2017 21:09:28 +0000"  >&lt;p&gt;last week i tried it and it seemed stuck on recovering the journal files... 200 Gb of journal to recover is a lot...&lt;/p&gt;

&lt;p&gt;how can i check the recovery process progress?&lt;br/&gt;
will it help to configure the mongod.conf this way:&lt;/p&gt;

&lt;p&gt;systemLog:&lt;br/&gt;
  verbosity: 0&lt;br/&gt;
    component:&lt;br/&gt;
      storage:&lt;br/&gt;
        verbosity: 1&lt;br/&gt;
        journal:&lt;br/&gt;
          verbosity: 3&lt;/p&gt;</comment>
                            <comment id="1576099" author="sue.loverso" created="Sat, 20 May 2017 20:33:28 +0000"  >&lt;p&gt;For the moment, if you restart the site it will replay the journal and then it will perform a checkpoint and that will allow it to remove the old journal files.  The situation you have is that the code assumes time goes forward and the system is stuck in checkpoint in a very long sleep because it computed a time difference when time went backward.  Assuming the checkpoint that happens after the restart and replay of the journal completes then it will be able to remove the files.&lt;/p&gt;

&lt;p&gt;I suggest doing this one site at a time.&lt;/p&gt;</comment>
                            <comment id="1576028" author="alberto.ornaghi@gmail.com" created="Sat, 20 May 2017 12:28:24 +0000"  >&lt;p&gt;I created an index on the primary and then it replicated to the secondary.&lt;br/&gt;
i&apos;ve noticed that the new index is 4 Gb but the total index size grew from 105 Gb to 144 Gb. very strage, i&apos;ve checked the journal and now i have the journal stuck on both the primary and the secondary (234 Gb on primary and 193 on secondary).  how can i resolve this critical situation? they will fill up the free disk space in few days.&lt;/p&gt;

&lt;p&gt;is it safe to kill on of them and let them redo the journal?  is there some sort of debug output to see in the log when it re-do the journal on startup to check if it&apos;s stucked or not?&lt;br/&gt;
will the bug prevent the journal to be re-done on startup?&lt;br/&gt;
please help, thank you.&lt;/p&gt;</comment>
                            <comment id="1574905" author="sue.loverso" created="Thu, 18 May 2017 20:33:52 +0000"  >&lt;p&gt;Work in WiredTiger relating to time structures and time going backward will be done in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3327&quot; title=&quot;Checkpoints can hang if time runs backward&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3327&quot;&gt;&lt;del&gt;WT-3327&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1573341" author="alberto.ornaghi@gmail.com" created="Wed, 17 May 2017 14:43:54 +0000"  >&lt;p&gt;yes, its full of those messages:&lt;/p&gt;

&lt;p&gt;May 17 14:37:05 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;4467&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:05 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: snapd.refresh.timer: Adding 3h 53min 17.369714s random time.&lt;br/&gt;
May 17 14:37:10 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;4467&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:10 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:10 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: apt-daily.timer: Adding 8h 13min 56.523873s random time.&lt;br/&gt;
May 17 14:37:10 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: snapd.refresh.timer: Adding 4h 5min 54.630702s random time.&lt;br/&gt;
May 17 14:37:15 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;4467&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:15 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:15 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: apt-daily.timer: Adding 4h 40min 22.691008s random time.&lt;br/&gt;
May 17 14:37:15 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: snapd.refresh.timer: Adding 5h 46min 46.616894s random time.&lt;br/&gt;
May 17 14:37:20 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;4467&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:20 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: Time has been changed&lt;br/&gt;
May 17 14:37:20 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: apt-daily.timer: Adding 1h 19min 2.758074s random time.&lt;br/&gt;
May 17 14:37:20 db-rs0-2 systemd&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;: snapd.refresh.timer: Adding 5h 1min 24.560710s random time.&lt;/p&gt;

&lt;p&gt;unfortunately the comment in the stack overflow article you linked (the one about azure) is not reassuring...&lt;br/&gt;
I will try to resolve this issue searching for specific problem of ubuntu + azure.&lt;/p&gt;

&lt;p&gt;thanks again&lt;/p&gt;</comment>
                            <comment id="1573317" author="sue.loverso" created="Wed, 17 May 2017 14:28:03 +0000"  >&lt;p&gt;If you check &lt;tt&gt;/var/log/syslog&lt;/tt&gt; you may see messages that say something like &quot;Time has been changed&quot; (or if you want to upload that file I will look at it here).  You might also find this page helpful: &lt;a href=&quot;https://askubuntu.com/questions/888493/var-log-syslog-systemd1-time-has-been-changed-message-every-5-seconds&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://askubuntu.com/questions/888493/var-log-syslog-systemd1-time-has-been-changed-message-every-5-seconds&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;ETA: If you see those messages, resolving those time change messages should resolve the other problems you&apos;re seeing.  Please let me know.&lt;/p&gt;</comment>
                            <comment id="1573198" author="sue.loverso" created="Wed, 17 May 2017 13:09:17 +0000"  >&lt;p&gt;If you notice this happening again, please try to gather stacks (one suggested method is pmp, poor man&apos;s profiler, from poormansprofiler.org).  Something inside WiredTiger using time is causing a checkpoint to get stuck and that needs to get fixed.  That is the reason the journal files are accumulating.  I will be looking at code to avoid time going backward and also find the hang.  Thank you for your cooperation!&lt;/p&gt;</comment>
                            <comment id="1573168" author="alberto.ornaghi@gmail.com" created="Wed, 17 May 2017 12:32:55 +0000"  >&lt;p&gt;I&apos;ve restarted the secondary from another snapshot from the primary. deleted the diagnostic.data dir.  the resync is done and the journal files are not accumulating right now, but I will monitor the situation and let you know if it happens again.  till now it happened 2 times over 3 resync. pretty bad rate.&lt;/p&gt;

&lt;p&gt;do you have any recommendation on the timesyncd configuration?&lt;/p&gt;</comment>
                            <comment id="1573038" author="alberto.ornaghi@gmail.com" created="Wed, 17 May 2017 07:52:10 +0000"  >&lt;p&gt;we are running on azure VM, they are Ubuntu 16.04, time is synchronised with the default timesyncd (no change to the configuration).&lt;br/&gt;
same configuration is on primary and secondary.&lt;br/&gt;
the issue happened after the first import, so I deleted the data dir of the secondary and copied over a snapshot of the primary, but it happened again.&lt;br/&gt;
what do you suggest now? should I resync again from another snapshot from the primary ? do I have to delete the diagnostic.data dir after the copy before restarting the secondary?&lt;/p&gt;

&lt;p&gt;thank you&lt;/p&gt;</comment>
                            <comment id="1572901" author="sue.loverso" created="Wed, 17 May 2017 00:41:51 +0000"  >&lt;p&gt;Can you describe the system you&apos;re running on?  Is it a virtual machine?  Or an AWS instance or some other cloud instance?  Are you running any time daemon process like ntpd or similar?  There are values that appear in the diagnostic data that indicate the system clock may have been adjusted backward, skewing several values, but also causing a checkpoint to get stuck.  That checkpoint getting stuck is the reason that journal files cannot be removed.&lt;/p&gt;

&lt;p&gt;If you can get it into this situation again, can you attempt to gather stacks so that we can see where the checkpoint thread is stuck.  I will be attempting to reproduce it here as well.  &lt;/p&gt;</comment>
                            <comment id="1572705" author="alberto.ornaghi@gmail.com" created="Tue, 16 May 2017 19:58:21 +0000"  >&lt;p&gt;the secondary server stopped working since it filled all the free space.&lt;/p&gt;

&lt;p&gt;the last server log (in mongodb.log) was: &lt;br/&gt;
2017-05-16T16:49:15.053+0000 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 10&amp;#93;&lt;/span&gt; WiredTiger error (28) &lt;span class=&quot;error&quot;&gt;&amp;#91;1494953355:45764&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;31637:0x7fd39ab78700&amp;#93;&lt;/span&gt;, WT_SESSION.commit_transaction: /data/rs0_1/journal/WiredTigerLog.0000012522: han&lt;/p&gt;

&lt;p&gt;i was tailing it live, but now its a zero byte files. so i cannot send it &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1572697" author="alberto.ornaghi@gmail.com" created="Tue, 16 May 2017 19:53:41 +0000"  >&lt;p&gt;uploaded to the same portal.&lt;br/&gt;
if you need other info, just ask.&lt;/p&gt;

&lt;p&gt;note that yesterday we performed a snapshot on the primary and attached it to the secondary that resynched from that point.&lt;br/&gt;
the problem was present even before so i tried the way of resyncing from a primary snapshot.&lt;/p&gt;</comment>
                            <comment id="1572468" author="sue.loverso" created="Tue, 16 May 2017 17:10:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alberto.ornaghi%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;alberto.ornaghi@gmail.com&quot;&gt;alberto.ornaghi@gmail.com&lt;/a&gt; can you also upload the WiredTiger.wt and WiredTiger.turtle files from your database directory?&lt;/p&gt;</comment>
                            <comment id="1572176" author="alberto.ornaghi@gmail.com" created="Tue, 16 May 2017 13:48:32 +0000"  >&lt;p&gt;I&apos;ve uploaded the diagnostic data.  &lt;/p&gt;

&lt;p&gt;Thank you&lt;/p&gt;</comment>
                            <comment id="1572129" author="thomas.schubert" created="Tue, 16 May 2017 13:17:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alberto.ornaghi%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;alberto.ornaghi@gmail.com&quot;&gt;alberto.ornaghi@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please upload an archive of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory from the affected node&apos;s dbpath?&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/4c262531-ae6e-4d27-a354-99515fbf9096.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you to use. Files uploaded to this portal are only visible to MongoDB employees investigating this issue and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="406664">DOCS-10562</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="384187">WT-3327</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="385338">SERVER-29308</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="156135" name="Screen Shot 2017-05-16 at 14.17.43.png" size="115140" author="alberto.ornaghi@gmail.com" created="Tue, 16 May 2017 12:44:43 +0000"/>
                            <attachment id="156134" name="journal_dir.txt" size="302338" author="alberto.ornaghi@gmail.com" created="Tue, 16 May 2017 12:45:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 16 May 2017 13:17:06 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 50 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 50 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alberto.ornaghi@gmail.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>sue.loverso@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht7hk7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hszmxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht4387:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>