<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:01:41 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-4288] Allow configuration of MAX_CONNECTING on the connection pool after 4.3.x changes</title>
                <link>https://jira.mongodb.org/browse/JAVA-4288</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;Hi&lt;/p&gt;

&lt;p&gt;After migrating from 4.2.1 to 4.3.1 with the connection pool changes made in &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-3927&quot; title=&quot;Rate limit new connection creations (maxConnecting)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-3927&quot;&gt;&lt;del&gt;JAVA-3927&lt;/del&gt;&lt;/a&gt; we find that only allowing 2 new connections at a time is too conservative and would like that value to be configurable.&lt;/p&gt;

&lt;p&gt;Our setup used to be (we have 60 worker threads in our app):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.minSize(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;60&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxSize(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;120&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxConnectionLifeTime(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;60&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, TimeUnit.SECONDS)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxWaitTime(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, TimeUnit.MILLISECONDS)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Due to external factors we need to limit our connection lifetime to 60 seconds, and we have high throughput on our connections, and we need to limit p99 latency spikes which is why we&apos;ve instructed the driver to time out rather than wait the default 2 seconds.&lt;/p&gt;


&lt;p&gt;After upgrading to 4.3.1 we started encountering a LOT of timeouts and changed the settings to:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.minSize(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;90&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxSize(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;120&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxConnectionLifeTime(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;60&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, TimeUnit.SECONDS)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maxWaitTime(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;50&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, TimeUnit.MILLISECONDS)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;.maintenanceFrequency(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, TimeUnit.MILLISECONDS)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;(changed max wait to 50ms, increased min-size and lowered maintenance freq)&lt;/p&gt;

&lt;p&gt;This takes care of MOST of our time-outs, but we still get the occassional one. It has increased our p99/p100 latency by 50ms, and we&apos;re essentially wasting 30 threads per host connecting to the DB.&lt;/p&gt;

&lt;p&gt;It&apos;d be most appreciated if we could configure MAX_CONNECTING instead of having it as a static final package-private constant.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1865741">JAVA-4288</key>
            <summary>Allow configuration of MAX_CONNECTING on the connection pool after 4.3.x changes</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="valentin.kovalenko@mongodb.com">Valentin Kavalenka</assignee>
                                    <reporter username="bridner.marc@gmail.com">Marc Bridner</reporter>
                        <labels>
                            <label>external-user</label>
                    </labels>
                <created>Wed, 1 Sep 2021 21:32:46 +0000</created>
                <updated>Wed, 4 May 2022 13:11:59 +0000</updated>
                            <resolved>Fri, 15 Oct 2021 23:51:35 +0000</resolved>
                                                                    <component>Connection Management</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="4127715" author="JIRAUSER1258163" created="Fri, 15 Oct 2021 23:50:40 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bridner.marc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;bridner.marc@gmail.com&quot;&gt;bridner.marc@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for all the information. The performance bug (&lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-4316&quot; title=&quot;The background thread should be able to hand over a new connection to a thread that is blocked by the max connecting limit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-4316&quot;&gt;&lt;del&gt;JAVA-4316&lt;/del&gt;&lt;/a&gt;) I identified while thinking about your report is now fixed in 4.3.4 and will be released with the next patch release.&lt;/p&gt;

&lt;p&gt;We got a request similar to yours for a different driver (&lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-3885&quot; title=&quot;[Unconfigurable] connection rate limiter in 2.13.x breaks existing applications&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CSHARP-3885&quot;&gt;&lt;del&gt;CSHARP-3885&lt;/del&gt;&lt;/a&gt;), but unlike in your scenario, it does not seem that there is any chance the scenario there can be mitigated with any approach other than either allowing to manually control the max connecting limit, or making it adaptive. The former approach has a chance to be agreed upon, specified and implemented in all drivers that implement the max connecting limit. I am closing this ticket and suggesting you to follow the progress at first via &lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1943&quot; title=&quot;Make maxConnecting configurable&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1943&quot;&gt;DRIVERS-1943&lt;/a&gt; and then via the JAVA ticket, which will be split from &lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1943&quot; title=&quot;Make maxConnecting configurable&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1943&quot;&gt;DRIVERS-1943&lt;/a&gt; when it is decided that the change should be implemented in the drivers.&lt;/p&gt;</comment>
                            <comment id="4090538" author="JIRAUSER1262189" created="Tue, 28 Sep 2021 21:22:25 +0000"  >&lt;p&gt;Hi&lt;/p&gt;

&lt;p&gt;Yes, we do storm the server somewhat, but ideally the storm should spread out over all 60 seconds, we don&apos;t refresh ALL connections in the same instant. We never ran into any starvation issues before the max connecting limit. We only have 6 hosts though, if we had 600 hosts it&apos;d be an entirely different story.&lt;/p&gt;

&lt;p&gt;More eager population of the connection pool would be ideal, if there&apos;s a guarantee that the minimum is actually maintained (except for network issues). Re-creating a connection on termination check-in would work too, if that was done on a thread other than the ones serving requests. Avoiding any connection management on request threads is important in order to have stable and predictable latencies.&lt;/p&gt;

&lt;p&gt;Our work-load are all .find()&apos;s that complete in p50 &amp;lt;3ms p99 &amp;lt;5 ms (at 12000 TPS), so I assume the duration of connections being checked out is p99 &amp;lt;5ms as well. I unfortunately do not have any reliable metrics on how long connection setup takes. Each of our servers run 60 worker threads, and during load tests they&apos;re 95% occupied. I don&apos;t know have any more detailed distribution metrics for concurrent check-outs.&lt;/p&gt;

&lt;p&gt;The 60 second limit is artificial due to internal company network structure, the IP we connect to is of an AWS Network Load Balancer with 1 host (the primary database instance) that has a hard limit of 350 seconds for one connection. We elected to do limit our connection to 60 seconds, it&apos;ll result in the same problem regardless of what value we pick. Having a load balancer with 1 host doesn&apos;t make much sense, but we have to use AWS VPC Endpoints to connect to this NLB from our own VPC due to company InfoSec policies as well as topology choices - we have no choice in this.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="4087908" author="JIRAUSER1258163" created="Mon, 27 Sep 2021 21:44:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bridner.marc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;bridner.marc@gmail.com&quot;&gt;bridner.marc@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for reporting the issue you encountered. Allowing to configure the max connecting limit, essentially allows to disable the limit and change the driver behavior back to when it could storm servers with new connection requests without being throttled. We would prefer to find other ways of improving latencies in situations like yours. You mitigated the problem to an extent by changing the driver configuration. One of the additional ideas is expressed in &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-4316&quot; title=&quot;The background thread should be able to hand over a new connection to a thread that is blocked by the max connecting limit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-4316&quot;&gt;&lt;del&gt;JAVA-4316&lt;/del&gt;&lt;/a&gt;, another could be to populate the connection pool based on &lt;tt&gt;minSize&lt;/tt&gt; more eagerly: either in anticipation that a connection will soon exceed its &lt;tt&gt;maxConnectionLifeTime&lt;/tt&gt; (this can be done if &lt;tt&gt;maxSize&lt;/tt&gt; - &lt;a href=&quot;https://github.com/mongodb/specifications/blob/cfe5b5b2450948f1cfae07dcc780602b8b738a22/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#connection-pool&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;totalConnectionCount&lt;/tt&gt;&lt;/a&gt; &amp;gt; 0), or right away when the pool terminates a connection on check in.&lt;/p&gt;

&lt;p&gt;Since it is not obvious what idea / a combination of ideas of mitigating p99 latency may help and to what extent, we would like to know more about your scenario of using the driver (note that this does not imply a commitment to implement any such idea). The most useful things to know are probably&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The distribution of the durations of time during which connections remain checked out (these durations do not include the checking out time). If this is impossible to provide, maybe you at least could tell us what kind of operations are executed via the Java driver.&lt;/li&gt;
	&lt;li&gt;The distribution of the durations of time it takes to establish a new connection when checking out.&lt;/li&gt;
	&lt;li&gt;The distribution of the number of concurrent checkouts per pool.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Of course, we are not asking for the actual probability distribution, but rather some information (different percentiles, mean/median/mode, etc.) that sheds light on the timings and the level of concurrency in your scenario.&lt;/p&gt;

&lt;p&gt;Another question I have is about limiting the connection lifetime to 60 seconds. What causes this? Effectively, your application regularly storms the servers with requests to establish new connections, which is a surprising approach per se, and even more so taking into the account your sensitivity to latencies.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1890889">DRIVERS-1943</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1884794">JAVA-4316</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1919386">JAVA-4390</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10257" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Documentation Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11861"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzk5yf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>