<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:32:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53857] Successive segfaults of several cluster members</title>
                <link>https://jira.mongodb.org/browse/SERVER-53857</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Our primary (called &quot;archi&quot; in the logs, IP ending in .29) crashed with a segfault at night, during a low traffic time.&lt;/p&gt;

&lt;p&gt;One secondary (called &quot;sonic&quot; in the logs, IP ending in .31) became primary to take over, and immediately crashed too, with a different stack trace.&lt;/p&gt;

&lt;p&gt;Finally, after a manual restart of servers to get the cluster with enough voting members (our voting members were slightly misconfigured at that point), another secondary (called &quot;loquy&quot;, IP ending in .34) tried to become primary and crashed too (stack trace identical to the first secondary). After a last restart of all of them, they recovered.&lt;/p&gt;

&lt;p&gt;I have attached all 3 log excerpts and stack traces. The &quot;diagnostic.data&quot; files for the day represent ~160MB total, I wasn&apos;t sure it was good form to plop that in a ticket. They are available here: &lt;a href=&quot;https://database.lichess.org/mongo-crash/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://database.lichess.org/mongo-crash/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The primary is running 4.4.2, the secondaries are on 4.4.3 (I was waiting for a maintenance window to upgrade primary and didn&apos;t dare to&#160; do it mid-incident).&lt;/p&gt;

&lt;p&gt;The only recent admin operation on that cluster was setting the minimum opLog window to 25h (via CLI), and restarting a few secondaries (not affected by those crashes) with the matching config file setting.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1592961">SERVER-53857</key>
            <summary>Successive segfaults of several cluster members</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-servicearch">Backlog - Service Architecture</assignee>
                                    <reporter username="lucas@lichess.org">Lucas Bonnet</reporter>
                        <labels>
                    </labels>
                <created>Sun, 17 Jan 2021 09:50:29 +0000</created>
                <updated>Tue, 6 Dec 2022 01:38:30 +0000</updated>
                            <resolved>Tue, 25 May 2021 16:13:55 +0000</resolved>
                                    <version>4.4.2</version>
                    <version>4.4.3</version>
                                                    <component>Stability</component>
                                        <votes>2</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="3838285" author="kyle.suarez" created="Tue, 25 May 2021 16:13:55 +0000"  >&lt;p&gt;Per &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=billy.donahue&quot; class=&quot;user-hover&quot; rel=&quot;billy.donahue&quot;&gt;billy.donahue&lt;/a&gt;, I&apos;m closing this issue as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53566&quot; title=&quot;Investigate and reproduce &amp;quot;opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&amp;quot; invariant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53566&quot;&gt;&lt;del&gt;SERVER-53566&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucas%40lichess.org&quot; class=&quot;user-hover&quot; rel=&quot;lucas@lichess.org&quot;&gt;lucas@lichess.org&lt;/a&gt;, sorry for the delay in getting back to you. Please see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53566&quot; title=&quot;Investigate and reproduce &amp;quot;opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&amp;quot; invariant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53566&quot;&gt;&lt;del&gt;SERVER-53566&lt;/del&gt;&lt;/a&gt; for a summary of where the fix has been released.&lt;/p&gt;</comment>
                            <comment id="3837821" author="kyle.suarez" created="Tue, 25 May 2021 15:12:13 +0000"  >&lt;p&gt;Service Arch, could you please take a look at this to confirm if it&apos;s the same issue as in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53566&quot; title=&quot;Investigate and reproduce &amp;quot;opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&amp;quot; invariant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53566&quot;&gt;&lt;del&gt;SERVER-53566&lt;/del&gt;&lt;/a&gt; as Billy suggests?&lt;/p&gt;</comment>
                            <comment id="3696751" author="billy.donahue" created="Thu, 1 Apr 2021 16:54:11 +0000"  >&lt;p&gt;It&apos;s really &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53566&quot; title=&quot;Investigate and reproduce &amp;quot;opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&amp;quot; invariant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53566&quot;&gt;&lt;del&gt;SERVER-53566&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3616621" author="dmitry.agranat" created="Tue, 16 Feb 2021 09:38:00 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucas%40lichess.org&quot; class=&quot;user-hover&quot; rel=&quot;lucas@lichess.org&quot;&gt;lucas@lichess.org&lt;/a&gt;, sorry to hear you&apos;ve experienced yet another issue:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-02-16T06:49:28.076+00:00&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23079,   &quot;ctx&quot;:&quot;TopologyVersionObserver&quot;,&quot;msg&quot;:&quot;Invariant failure&quot;,&quot;attr&quot;:{&quot;expr&quot;:&quot;opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot;,&quot;file&quot;:&quot;src/mongo/db/client.cpp&quot;,&quot;line&quot;:126}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This new issue is being tracked by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49468&quot; title=&quot;Invalidate previous OperationContext when a new OperationContext is created&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49468&quot;&gt;&lt;del&gt;SERVER-49468&lt;/del&gt;&lt;/a&gt;. it shows that a backport to 4.4 was requested but not yet implemented.&lt;/p&gt;</comment>
                            <comment id="3616617" author="JIRAUSER1258313" created="Tue, 16 Feb 2021 09:29:30 +0000"  >&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;this cluster crash happened a second time, at peak time for us, but this time it happened in a loop (each time a server recovered, it crashed again a few seconds later) until we disabled the feature relying on those 999 BSON nested documents. We have since rewrote this part of the code and moved the &quot;study&quot; collection to a distinct mongodb server with a different data storage format, no longer relying on this undocumented feature.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;However, during a routine maintenance to finally upgrade our PRI server from 4.4.2 to 4.4.3, we experienced another cluster crash, with the following timeline (UTC):&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;2021-02-16T06:49:28.064 : stepDown manually sent to archi, our PRI&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:28.071 : archi crashes&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:28.132 : loquy notices the change of PRI&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:28.145 : loquy becomes PRI as planned&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:28.156 : loquy crashes&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:39.004 : sonic notices there is no PRI, triggers an election&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:39.006 : sonic becomes PRI&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:49:39.017 : sonic crashes&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:12.904 : loquy manually restarted (right before boole crashes, but too late to rejoin the cluster in time)&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:22.704 : boole notices there is no PRI, triggers an election&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:22.709 : boole becomes PRI&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:22.719 : boole crashes&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:47.788 : sonic manually restarted&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:53.969 : sonic recovers&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:55.336 : loquy recovers (becomes PRI)&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:55.414 : higgs crashes (despite never becoming PRI)&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:50:55.586 : boole manually restarted&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:51:01.585 : boole recovers&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:51:15.123 : archi manually restarted after upgrade to 4.4.3&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:51:33.377 : archi recovers (becomes PRI), loquy becomes SEC&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:51:50.564 : higgs manually restarted&lt;/li&gt;
	&lt;li&gt;2021-02-16T06:51:53.781 : higgs recovers&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I attached new stack traces excerpts (trace-2-* files) but they look different, is that worthy of a new issue or is this related to our first crash? Also please tell me if you need more info, logs, or diagnostics data.&lt;/p&gt;</comment>
                            <comment id="3582477" author="dmitry.agranat" created="Tue, 26 Jan 2021 14:23:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucas%40lichess.org&quot; class=&quot;user-hover&quot; rel=&quot;lucas@lichess.org&quot;&gt;lucas@lichess.org&lt;/a&gt;, thank you for the report, I was able to reproduce this issue. We&apos;re assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen.&lt;/p&gt;</comment>
                            <comment id="3569797" author="JIRAUSER1258313" created="Mon, 18 Jan 2021 16:24:42 +0000"  >&lt;p&gt;Hello, thanks for looking into it.&lt;/p&gt;

&lt;p&gt;We actually have &lt;tt&gt;maxBSONDepth: 999&lt;/tt&gt; in our config files because we rely on deeply nested structures for one collection, I have attached our config file for one server, they&apos;re all identical.&lt;/p&gt;</comment>
                            <comment id="3569544" author="dmitry.agranat" created="Mon, 18 Jan 2021 12:25:37 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucas%40lichess.org&quot; class=&quot;user-hover&quot; rel=&quot;lucas@lichess.org&quot;&gt;lucas@lichess.org&lt;/a&gt;, based on &lt;a href=&quot;https://docs.mongodb.com/manual/reference/limits/#Nested-Depth-for-BSON-Documents&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;our documentation&lt;/a&gt;, MongoDB supports no more than 100 levels of nesting for BSON documents.&lt;/p&gt;

&lt;p&gt;By counting the lines with the backtrace, I saw exactly 100 of &lt;tt&gt;BSONObjBuilder&lt;/tt&gt; frames.&lt;/p&gt;

&lt;p&gt;Does this observation align with your workload?&lt;/p&gt;</comment>
                            <comment id="3569153" author="JIRAUSER1258313" created="Sun, 17 Jan 2021 09:55:09 +0000"  >&lt;p&gt;Forgot to state the obvious:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Ubuntu 18.04 with &amp;lt;1yr uptime&lt;/li&gt;
	&lt;li&gt;Upstream mongodb packages&lt;/li&gt;
	&lt;li&gt;NVMe storage and generous hardware configuration (96 or 128GB RAM, 16 cores)&lt;/li&gt;
	&lt;li&gt;1.5TB used by mongodb data files on disk on primary&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1580342">SERVER-53566</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1407903">SERVER-49468</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1580342">SERVER-53566</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="295804" name="archi.log.bz2" size="14571" author="lucas@lichess.org" created="Sun, 17 Jan 2021 09:40:26 +0000"/>
                            <attachment id="295803" name="loquy.log.bz2" size="66514" author="lucas@lichess.org" created="Sun, 17 Jan 2021 09:40:34 +0000"/>
                            <attachment id="295869" name="mongod.conf" size="379" author="lucas@lichess.org" created="Mon, 18 Jan 2021 16:24:07 +0000"/>
                            <attachment id="295802" name="sonic.log.bz2" size="13653" author="lucas@lichess.org" created="Sun, 17 Jan 2021 09:40:37 +0000"/>
                            <attachment id="300179" name="trace-2-archi.txt" size="14704" author="lucas@lichess.org" created="Tue, 16 Feb 2021 09:28:18 +0000"/>
                            <attachment id="300178" name="trace-2-higgs.txt" size="33997" author="lucas@lichess.org" created="Tue, 16 Feb 2021 09:28:17 +0000"/>
                            <attachment id="300177" name="trace-2-loquy.txt" size="35806" author="lucas@lichess.org" created="Tue, 16 Feb 2021 09:28:17 +0000"/>
                            <attachment id="295801" name="trace-archi.txt" size="47436" author="lucas@lichess.org" created="Sun, 17 Jan 2021 09:40:41 +0000"/>
                            <attachment id="295800" name="trace-sonic.txt" size="8574" author="lucas@lichess.org" created="Sun, 17 Jan 2021 09:40:43 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25132"><![CDATA[Service Arch]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 18 Jan 2021 12:25:37 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 37 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 37 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-servicearch</customfieldvalue>
            <customfieldvalue>billy.donahue@mongodb.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>kyle.suarez@mongodb.com</customfieldvalue>
            <customfieldvalue>lucas@lichess.org</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyq0e7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hybpbb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4462">Query Execution 2021-02-22</customfieldvalue>
    <customfieldvalue id="4464">Query Execution 2021-03-08</customfieldvalue>
    <customfieldvalue id="4466">Query Execution 2021-03-22</customfieldvalue>
    <customfieldvalue id="4468">Query Execution 2021-04-05</customfieldvalue>
    <customfieldvalue id="4470">Query Execution 2021-04-19</customfieldvalue>
    <customfieldvalue id="4472">Query Execution 2021-05-03</customfieldvalue>
    <customfieldvalue id="4697">Query Execution 2021-05-17</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hypmnj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>