<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:40:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35558] Drop in performance for the insert command after restarting the MongoDB database</title>
                <link>https://jira.mongodb.org/browse/SERVER-35558</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Good afternoon,&lt;/p&gt;

&lt;p&gt;Our team has experienced a substantial drop in performance for the bulk insert command (when inserting 10,000 documents) after restarting the MongoDB database (3.2.20). Based on a brief analysis, there seem to be two issues: (1) The particular collection uses random scattered keys (2) The index tree is only partially loaded in the CPU memory.&lt;/p&gt;

&lt;p&gt;The first issue can be easily fixed by prepending timestamp to the scattered keys. For resolving the second issue, we are looking for feedback from MongoDB experts. Specifically, there are three questions:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Is it &#8220;a bug or feature&#8221; of the MongoDB database to keep only a part of the index tree in CPU memory? We are using a multi-core server with around 200 GB memory. According to the collection statistics, however, the &#8216;datum_id_1&#8217; cache contains only 1,912,845,502.0 bytes while the total index size is 9,059,713,024.0 as demonstrated by the following script:&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;a href=&quot;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.6.10.18.ipynb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.6.10.18.ipynb&lt;/a&gt;&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Is there any mechanism to push and keep an index tree in the CPU memory?&lt;/li&gt;
	&lt;li&gt;Are the data and index caches managed by one or two different LRU procedures?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thank you in advance,&lt;/p&gt;

&lt;p&gt;Nikolay&lt;/p&gt;

&lt;p&gt;This question has been submitted to &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://groups.google.com/group/mongodb-user&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;and &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://stackoverflow.com/questions/tagged/mongodb&lt;/a&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="558067">SERVER-35558</key>
            <summary>Drop in performance for the insert command after restarting the MongoDB database</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="nmalitsky">Nikolay Malitsky</reporter>
                        <labels>
                    </labels>
                <created>Tue, 12 Jun 2018 16:10:30 +0000</created>
                <updated>Fri, 27 Oct 2023 13:53:44 +0000</updated>
                            <resolved>Sun, 17 Jun 2018 06:22:56 +0000</resolved>
                                                                    <component>Index Maintenance</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1922821" author="dmitry.agranat" created="Sun, 17 Jun 2018 06:22:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nmalitsky&quot; class=&quot;user-hover&quot; rel=&quot;nmalitsky&quot;&gt;nmalitsky&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for your report and questions. As we did not find any evidence that the reported issue is related to MongoDB, I will go ahead and close this ticket.&lt;/p&gt;

&lt;p&gt;Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt;&lt;br/&gt;
or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;.&lt;br/&gt;
A question like this involving more discussion would be best posted on the mongodb-user group.&lt;/p&gt;

&lt;p&gt;If this issue is reproducible on a storage system as mentioned &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35558?focusedCommentId=1920993&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1920993&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, feel free to reopen this ticket.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="1921745" author="nmalitsky" created="Fri, 15 Jun 2018 14:33:04 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Our IT team is analyzing the file system. I expect that that slow&lt;br/&gt;
throughput rate is determined by the seek time of spinning disks&lt;br/&gt;
(0.004-0.009 s) dominating reading small index pages (16,384 bytes/100 MB/s&lt;br/&gt;
= 0.00016). I will let you know once our IT team has the disk&lt;br/&gt;
specification.&lt;/p&gt;

&lt;p&gt;Most importantly, your analysis confirmed my previous concern associated&lt;br/&gt;
with partial index allocation. Could you answer the following questions?&lt;/p&gt;

&lt;p&gt;1. Is there any mechanism to push and keep an index tree in the CPU memory?&lt;/p&gt;

&lt;p&gt;2. Are the data and index caches managed by one or two different LRU&lt;br/&gt;
procedures?&lt;/p&gt;

&lt;p&gt;3. Is prepending the scattered random keys with timestamp (line 7-8 in&lt;br/&gt;
GitHub script) the correct/best solution for achieving the best insert&lt;br/&gt;
performance for our case (partially loaded index tree and slow i/o access).&lt;br/&gt;
This solution is derived from the MongoDB brief notes:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/ensure-indexes-fit-ram/#indexes-that-hold-only-recent-values-in-ram&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/tutorial/ensure-indexes-fit-ram/#indexes-that-hold-only-recent-values-in-ram&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;4. What btree and cache information is provided in &#8216;wiredTiger&#8217; entry&lt;br/&gt;
(towards the end) of the collStats command (line 10 in GitHub script below)&lt;br/&gt;
in addition to &#8216;indexDetails&#8217; (towards the beginning of this command):&lt;/p&gt;


&lt;p&gt;&lt;a href=&quot;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;These questions are important for updating our MongoDB environment and we&lt;br/&gt;
thank you in advance for taking the time to answer our questions.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;/p&gt;

&lt;p&gt;Nikolay&lt;/p&gt;


&lt;p&gt;On Thu, Jun 14, 2018 at 2:39 PM, Dmitry Agranat (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1920993" author="dmitry.agranat" created="Thu, 14 Jun 2018 18:38:47 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nmalitsky&quot; class=&quot;user-hover&quot; rel=&quot;nmalitsky&quot;&gt;nmalitsky&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;If you can reproduce this issue on the system which has more than just a few MB/s read &amp;amp; write throughput capacity, we&apos;ll be able to look into this. For a reference, these are our &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/index.html#disk-and-storage-systems&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;disk&lt;/a&gt; and &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/index.html#kernel-and-file-systems&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file-system&lt;/a&gt; recommendations.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A rate of ~ 2.8 MB/s looks very slow indeed. Would this rate include some&lt;br/&gt;
MongoDB overhead?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I was not able to find any indication that the reads from disk were related to MongoDB.&lt;/p&gt;

&lt;p&gt;If this issue is reproducible on a storage system according to our recommendations (and ideally on the latest MongoDB version), please provide the following information:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;compressed &lt;tt&gt;mongod&lt;/tt&gt; log and archived content of &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;file-system type (EXT4 or XFS) and disk throughput specifications&lt;/li&gt;
	&lt;li&gt;any supporting profiling information like you did in your previous comments&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="1920843" author="nmalitsky" created="Thu, 14 Jun 2018 16:51:04 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Thank you for the analysis.&lt;/p&gt;

&lt;p&gt;A rate of ~ 2.8 MB/s looks very slow indeed. Would this rate include some&lt;br/&gt;
MongoDB overhead?&lt;/p&gt;

&lt;p&gt;Here is the output for the &#8220;df &#8211;l&#8221; command:&lt;br/&gt;
Filesystem        1K-blocks         Used    Available Use% Mounted on&lt;/p&gt;

&lt;p&gt;/dev/sda7         913578428    355946304    511201916  42% /&lt;br/&gt;
udev                  10240            0        10240   0% /dev&lt;br/&gt;
tmpfs              52914392         9832     52904560   1% /run&lt;br/&gt;
tmpfs             132285972            0    132285972   0% /dev/shm&lt;br/&gt;
tmpfs                  5120            0         5120   0% /run/lock&lt;br/&gt;
tmpfs             132285972            0    132285972   0% /sys/fs/cgroup&lt;br/&gt;
/dev/sda6         230563988      4562220    214266680   3% /var&lt;br/&gt;
/dev/sda1            235021        48886       174001  22% /boot&lt;br/&gt;
The MongoDB data files are located in a local directory.&lt;/p&gt;

&lt;p&gt;From our side, profiling clearly identified a performance issue (CPU vs&lt;br/&gt;
Wall Time) located in&lt;/p&gt;

&lt;p&gt;the __page_read method that reads a page from the file:&lt;/p&gt;



&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/bt_read.c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/bt_read.c&lt;/a&gt;&lt;br/&gt;
Regarding MongoDB 3.6.5 version. At this time, we are evaluating the&lt;br/&gt;
MongoDB sharded cluster using the 3.6 version.&lt;br/&gt;
The sharded server is connected with the production system, but we can&lt;br/&gt;
install the 3.6.5 version on&lt;br/&gt;
the research &amp;amp; developer server and repeat the same test.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Nikolay&lt;/p&gt;

&lt;p&gt;On Thu, Jun 14, 2018 at 5:46 AM, Dmitry Agranat (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1920453" author="dmitry.agranat" created="Thu, 14 Jun 2018 09:43:26 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nmalitsky&quot; class=&quot;user-hover&quot; rel=&quot;nmalitsky&quot;&gt;nmalitsky&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading the requested information. It was very useful.&lt;/p&gt;

&lt;p&gt;I suspect the issue is related to the i/o subsystem.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/189453/189453_io_SERVER_35558.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;At &lt;b&gt;A&lt;/b&gt;, around 14:49:30 (UTC), we are starting to read from disk with a rate of ~2.8MB/s which results in 100% disk utilization.&lt;/li&gt;
	&lt;li&gt;There is a stall between &lt;b&gt;B&lt;/b&gt; and &lt;b&gt;C&lt;/b&gt;.&lt;/li&gt;
	&lt;li&gt;Right before the stall at &lt;b&gt;B&lt;/b&gt;, there is a checkpoint trying to write about 8MB/s.&lt;/li&gt;
	&lt;li&gt;Bulk insert runs between &lt;b&gt;A&lt;/b&gt; and &lt;b&gt;D&lt;/b&gt;.&lt;/li&gt;
	&lt;li&gt;At &lt;b&gt;E&lt;/b&gt;, when there are no read operations, 10K insert operations is observed&lt;/li&gt;
	&lt;li&gt;There is another short stall at &lt;b&gt;F&lt;/b&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;To proceed with this investigation, could you please provide more details about the &lt;tt&gt;sda&lt;/tt&gt; disk and what filesystem type is used? Also, is there any specific reason you are not running with latest MongoDB 3.6.5 version?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="1919578" author="nmalitsky" created="Wed, 13 Jun 2018 15:27:04 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Thank you for your prompt response.&lt;/p&gt;

&lt;p&gt;At this time, our production system is running a fresh database (with a few&lt;br/&gt;
documents of the same collection)&lt;br/&gt;
and we are evaluating this issue on the research &amp;amp; development multi-core&lt;br/&gt;
server.&lt;/p&gt;

&lt;p&gt;The performance issue is reproducible. To address your request I ran the&lt;br/&gt;
following two steps:&lt;/p&gt;

&lt;p&gt;1. Start the MongoDB database (at 10:33 am EDT, June 13, 2018). This&lt;br/&gt;
database was compiled from the github source code (git clone -b v3.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo.git&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo.git&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;malitsky@gpu-001:~/mongo/bench.v3.2$&lt;br/&gt;
/home/malitsky/mongo/bench.v3.2/opt/mongodb/bin/mongod --dbpath&lt;br/&gt;
/home/malitsky/mongo/bench.v3.2/data/mongodb.5.27.18 --logpath&lt;br/&gt;
/home/malitsky/mongo/bench.v3.2/var/log/mongodb/mongodb.log --port 27018&lt;br/&gt;
--bind_ip 127.0.0.1,gpu-001&lt;/p&gt;

&lt;p&gt;2018-06-13T10:32:51.849-0400 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;main&amp;#93;&lt;/span&gt; log file&lt;br/&gt;
&quot;/home/malitsky/mongo/bench.v3.2/var/log/mongodb/mongodb.log&quot; exists; moved&lt;br/&gt;
to&lt;br/&gt;
&quot;/home/malitsky/mongo/bench.v3.2/var/log/mongodb/mongodb.log.2018-06-13T14-32-51&quot;.&lt;/p&gt;

&lt;p&gt;2. Run the python script for inserting 10,000 documents (at 10:49 am EDT,&lt;br/&gt;
June 13, 2018):&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The corresponding diagnostics.data.tar.gz and log.tar.gz files  are&lt;br/&gt;
uploaded to your server.&lt;/p&gt;

&lt;p&gt;Based on preliminary profiling, the issue (CPU vs Wall time) is&lt;br/&gt;
encapsulated in the __wt_row_search method:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/row_srch.c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/row_srch.c&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Thanks again,&lt;/p&gt;

&lt;p&gt;Nikolay&lt;/p&gt;



&lt;p&gt;On Tue, Jun 12, 2018 at 4:39 PM, Dmitry Agranat (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1918904" author="dmitry.agranat" created="Tue, 12 Jun 2018 20:38:10 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nmalitsky&quot; class=&quot;user-hover&quot; rel=&quot;nmalitsky&quot;&gt;nmalitsky&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;So we can investigate it can you please upload to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/3d17c073-fb78-4aa2-8e84-cff17ff98746.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure private portal&lt;/a&gt;, for the affected instance:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;archived content of $dbpath/diagnostic.data&lt;/li&gt;
	&lt;li&gt;complete mongod log files ideally covering time from upgrade until the present&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also can you please give me a timeline of the events that you describe as accurately as you can (including timezone). This will ensure that we&apos;re looking at the right time period in the data.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="189453" name="io_SERVER_35558.png" size="268934" author="dmitry.agranat@mongodb.com" created="Thu, 14 Jun 2018 09:43:17 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 Jun 2018 20:38:10 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 34 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 34 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>nmalitsky</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0e1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htr9tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu00bb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>