<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:31:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53667] High rate of journal flushes on secondary in 4.4</title>
                <link>https://jira.mongodb.org/browse/SERVER-53667</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;A simple workload with a single thread doing updates as fast as possible sees the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;4.2
	&lt;ul&gt;
		&lt;li&gt;primary
		&lt;ul&gt;
			&lt;li&gt;updates: 1200/s&lt;/li&gt;
			&lt;li&gt;log flushes: 300/s&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
		&lt;li&gt;secondary
		&lt;ul&gt;
			&lt;li&gt;batches: 280/s&lt;/li&gt;
			&lt;li&gt;log flushes: 300/s&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;4.4
	&lt;ul&gt;
		&lt;li&gt;primary
		&lt;ul&gt;
			&lt;li&gt;updates: 1250/s&lt;/li&gt;
			&lt;li&gt;log flushes: 10/s; much improved in 4.4 due to PM-1274&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
		&lt;li&gt;seccondary
		&lt;ul&gt;
			&lt;li&gt;batches: 1200/s&lt;/li&gt;
			&lt;li&gt;log flushes: 2400/s; much worse in 4.4 than 4.2, and much higher than primary&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This creates a much higher i/o requirement on the secondary in 4.4, and much higher on the secondary than the primary.&lt;/p&gt;

&lt;p&gt;Two possible issues:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Do we need to flush more often on the secondary than the primary?&lt;/li&gt;
	&lt;li&gt;Do we need to do 2 flushes per batch?&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="1584556">SERVER-53667</key>
            <summary>High rate of journal flushes on secondary in 4.4</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>perf-escapes</label>
                    </labels>
                <created>Fri, 8 Jan 2021 18:48:04 +0000</created>
                <updated>Mon, 19 Dec 2022 19:17:21 +0000</updated>
                            <resolved>Mon, 13 Jun 2022 14:58:51 +0000</resolved>
                                                                                        <votes>1</votes>
                                    <watches>44</watches>
                                                                                                                <comments>
                            <comment id="4611366" author="matthew.russotto" created="Mon, 13 Jun 2022 15:25:32 +0000"  >&lt;p&gt;The issue of doing two journal flushes per batch was fixed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54938&quot; title=&quot;Only flush journal once per batch on secondary oplog application&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54938&quot;&gt;&lt;del&gt;SERVER-54938&lt;/del&gt;&lt;/a&gt; (backported to 4.4.5).  The reduced batch size on 4.4 is due to changes designed to improve read concern majority latency.  We have added a tunable parameter in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-65723&quot; title=&quot;Add tunable parameter to improve batching on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-65723&quot;&gt;&lt;del&gt;SERVER-65723&lt;/del&gt;&lt;/a&gt; to allow increased batching at the cost of higher read concern majority latency.&lt;/p&gt;</comment>
                            <comment id="3652127" author="lingzhi.deng" created="Mon, 8 Mar 2021 15:47:27 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54938&quot; title=&quot;Only flush journal once per batch on secondary oplog application&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54938&quot;&gt;&lt;del&gt;SERVER-54938&lt;/del&gt;&lt;/a&gt; answered and fixed the second issue in the description (i.e. Do we need to do 2 flushes per batch?) and will be backported to 4.4. We still need more investigation in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54939&quot; title=&quot;Investigate secondary batching behavior in v4.4&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54939&quot;&gt;&lt;del&gt;SERVER-54939&lt;/del&gt;&lt;/a&gt; to answer the first question. Once we have an answer to that and if we have a fix, we will definitely fix that in 4.4+.&lt;/p&gt;</comment>
                            <comment id="3652081" author="eric.lafontaine@bell.ca" created="Mon, 8 Mar 2021 15:35:15 +0000"  >&lt;p&gt;Hi All,&lt;/p&gt;

&lt;p&gt;Just wondering if this will be fixed in Mongo 4.x or only in Mongo 5.x as per the bug resolution ticket.&#160; The difference is huge on our side if the fix is backward incompatible.&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;

&lt;p&gt;Eric Lafontaine&lt;/p&gt;</comment>
                            <comment id="3648928" author="bruce.lucas@10gen.com" created="Fri, 5 Mar 2021 13:09:52 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason.chan&quot; class=&quot;user-hover&quot; rel=&quot;jason.chan&quot;&gt;jason.chan&lt;/a&gt;, it&apos;s not completely clear to me whether the increased number of batches is the real issue or whether changing the number of batches is the solution, so I&apos;d like to keep this ticket open until we&apos;ve investigated that question.&lt;/p&gt;</comment>
                            <comment id="3645926" author="jason.chan" created="Thu, 4 Mar 2021 00:56:16 +0000"  >&lt;p&gt;Filed&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54938&quot; title=&quot;Only flush journal once per batch on secondary oplog application&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54938&quot;&gt;&lt;del&gt;SERVER-54938&lt;/del&gt;&lt;/a&gt;&#160;to remove the call to &lt;tt&gt;triggerJournalFlush&lt;/tt&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54939&quot; title=&quot;Investigate secondary batching behavior in v4.4&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54939&quot;&gt;&lt;del&gt;SERVER-54939&lt;/del&gt;&lt;/a&gt; to investigate the increased number of batches between 4.2 and 4.4.&lt;/p&gt;</comment>
                            <comment id="3644925" author="bruce.lucas@10gen.com" created="Wed, 3 Mar 2021 17:50:16 +0000"  >&lt;p&gt;Here&apos;s the simple workload that was used to generate the numbers above.&lt;/p&gt;

&lt;p&gt;Setup:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;function insert() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    db.c.drop()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (var i = 0; i &amp;lt; 1000*1000; i+=1000) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        many = []&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (var j = 0; j &amp;lt; 1000; j++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            many.push({_id: (i+j), x: 0})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        db.c.insertMany(many)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Workload:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;function update() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    while (true) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (var i = 0; i &amp;lt; 1000*1000; i++) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            db.c.update({_id: 1}, {$inc: {x: 1}}})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3585079" author="dianna.hohensee" created="Wed, 27 Jan 2021 16:05:57 +0000"  >&lt;p&gt;I&apos;m passing this back to replication to determine how to reduce (or parallelize) journal flush calls.&lt;/p&gt;</comment>
                            <comment id="3555476" author="bruce.lucas@10gen.com" created="Fri, 8 Jan 2021 18:58:26 +0000"  >&lt;p&gt;I assigned this to replication based on &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=evin.roesle&quot; class=&quot;user-hover&quot; rel=&quot;evin.roesle&quot;&gt;evin.roesle&lt;/a&gt;&apos;s suggestion, but PM-1274 was done by the execution team, so not sure who should look into this.&lt;/p&gt;</comment>
                            <comment id="3555474" author="bruce.lucas@10gen.com" created="Fri, 8 Jan 2021 18:57:33 +0000"  >&lt;p&gt;I suspect this is a side effect of PM-1274: in 4.2 we journal before replicating, so batch rate and therefore flush rate on secondary is determined by flush rate on primary. But in 4.4 we replicate before journaling, so the batch rate is much higher, often equal to operation rate.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1638748">SERVER-54938</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1638751">SERVER-54939</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="1857011">DOCS-14755</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1638751">SERVER-54939</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1867214">SERVER-59776</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="303673">SERVER-25249</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1638748">SERVER-54938</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2027032">SERVER-65723</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="302923" name="Screen Shot 2021-03-03 at 2.03.12 PM.png" size="37174" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 22:35:23 +0000"/>
                            <attachment id="302911" name="Screen Shot 2021-03-03 at 2.03.34 PM.png" size="30846" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 22:04:09 +0000"/>
                            <attachment id="302924" name="Screen Shot 2021-03-03 at 2.35.57 PM.png" size="35222" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 22:36:22 +0000"/>
                            <attachment id="302925" name="Screen Shot 2021-03-03 at 2.36.01 PM.png" size="33368" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 22:36:36 +0000"/>
                            <attachment id="310579" name="Screenshot 2021-04-20 at 10.14.07-1.png" size="97772" author="frank.derwin@mongodb.com" created="Tue, 20 Apr 2021 09:52:01 +0000"/>
                            <attachment id="310578" name="Screenshot 2021-04-20 at 10.14.07.png" size="97772" author="frank.derwin@mongodb.com" created="Tue, 20 Apr 2021 09:49:43 +0000"/>
                            <attachment id="302909" name="primary-asis.png" size="4898" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 21:57:32 +0000"/>
                            <attachment id="302910" name="secondary_as_is.png" size="5122" author="jason.chan@mongodb.com" created="Wed, 3 Mar 2021 21:58:21 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000rzvmxQAA, 5002K00000scFC8QAM, 5002K00000tTXUIQA4, 5002K00000uGoz1QAC, 5002K00000uzg6EQAQ, 5002K00000uzfw9QAA, 5002K00000xj6OpQAI, 5002K00000xDlvMQAS, 5002K000011BWP1QAO, 5006R00001l880VQAQ, 5006R00001ommH7QAI]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 8 Jan 2021 19:46:39 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 34 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-54938'>SERVER-54938</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-54939'>SERVER-54939</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 34 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>eric.lafontaine@bell.ca</customfieldvalue>
            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyokv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr42en:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4366">Execution Team 2021-01-25</customfieldvalue>
    <customfieldvalue id="4756">Repl 2021-03-08</customfieldvalue>
    <customfieldvalue id="4762">Repl 2021-03-22</customfieldvalue>
    <customfieldvalue id="4798">Repl 2021-04-05</customfieldvalue>
    <customfieldvalue id="5709">Repl 2022-03-21</customfieldvalue>
    <customfieldvalue id="5710">Repl 2022-04-04</customfieldvalue>
    <customfieldvalue id="5711">Repl 2022-04-18</customfieldvalue>
    <customfieldvalue id="5995">Repl 2022-05-02</customfieldvalue>
    <customfieldvalue id="6170">Repl 2022-05-16</customfieldvalue>
    <customfieldvalue id="6248">Repl 2022-05-30</customfieldvalue>
    <customfieldvalue id="6249">Repl 2022-06-13</customfieldvalue>
    <customfieldvalue id="6250">Repl 2022-06-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyo74f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>