<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:21:51 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9934] Slow failovers after step down due to sleeping rsBackgroundSync</title>
                <link>https://jira.mongodb.org/browse/SERVER-9934</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;With a healthy replset, failovers via rs.stepDown() take 10+s,&lt;br/&gt;
even when the election completes in 1-2s.&lt;/p&gt;

&lt;p&gt;After an rs.steDown(), a healthy moderately-loaded replset on a&lt;br/&gt;
good network will normally complete an election for an new&lt;br/&gt;
primary normally within 1-2s. However, before the new primary can&lt;br/&gt;
step up, it has to stop the background sync thread.&lt;/p&gt;

&lt;p&gt;Suppose (as is usually/often the case) that the new primary was&lt;br/&gt;
syncing from the stepped-down node. When that node stepped down,&lt;br/&gt;
it closed all connections, including the connection the new&lt;br/&gt;
primary was using to read the oplog. This triggered a DBException&lt;br/&gt;
inside of _producerThread, triggering this code:&lt;/p&gt;

&lt;p&gt;    sethbmsg(str::stream() &amp;lt;&amp;lt; &quot;db exception in producer: &quot; &amp;lt;&amp;lt; e.toString());&lt;br/&gt;
    sleepsecs(10);&lt;/p&gt;

&lt;p&gt;When the new primary goes to stop the rsBackgroundSync thread,&lt;br/&gt;
the thread is caught in the middle of that sleep, and so we end&lt;br/&gt;
up literally waiting on a sleep(10) before the cluster has a new&lt;br/&gt;
primary.&lt;/p&gt;</description>
                <environment>Tested on Linux amd64, Ubuntu (10.04 Lucid and 13.04 Raring)</environment>
        <key id="79004">SERVER-9934</key>
            <summary>Slow failovers after step down due to sleeping rsBackgroundSync</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="milkie@mongodb.com">Eric Milkie</assignee>
                                    <reporter username="nelhage">Nelson Elhage</reporter>
                        <labels>
                    </labels>
                <created>Thu, 13 Jun 2013 22:23:46 +0000</created>
                <updated>Mon, 11 Jul 2016 17:40:39 +0000</updated>
                            <resolved>Sat, 29 Jun 2013 13:04:22 +0000</resolved>
                                    <version>2.4.4</version>
                                    <fixVersion>2.4.6</fixVersion>
                    <fixVersion>2.5.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>1</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="411329" author="onyxmaster" created="Mon, 26 Aug 2013 12:51:30 +0000"  >&lt;p&gt;The &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9464&quot; title=&quot;On election, relax priority restriction when another member is fresher&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9464&quot;&gt;&lt;del&gt;SERVER-9464&lt;/del&gt;&lt;/a&gt; wasn&apos;t actually fixed by this, see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-10621&quot; title=&quot;Replication failover hangs indefinitely when priorities conflict with replication&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-10621&quot;&gt;&lt;del&gt;SERVER-10621&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="394516" author="auto" created="Fri, 2 Aug 2013 15:31:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9934&quot; title=&quot;Slow failovers after step down due to sleeping rsBackgroundSync&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9934&quot;&gt;&lt;del&gt;SERVER-9934&lt;/del&gt;&lt;/a&gt; remove double getmore when pulling new ops&lt;/p&gt;

&lt;p&gt;This reduces the potential delay in ending the bgsync thread from 10 seconds&lt;br/&gt;
to 5 seconds.  This is important because assumingPrimaryness waits on this loop&lt;br/&gt;
before changing state to PRIMARY after an election.&lt;br/&gt;
To reduce the delay lower than 5 seconds requires further redesign of the code.&lt;br/&gt;
Branch: v2.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b89e706faca937c07cc578c37c962532fc7b751b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b89e706faca937c07cc578c37c962532fc7b751b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="391746" author="milkie" created="Tue, 30 Jul 2013 12:50:00 +0000"  >&lt;p&gt;Hi Nelson.  Yes, the Backport field is what we use to indicate that.  When it&apos;s complete we will change the field to &quot;Done&quot; (or &quot;Cannot&quot;).&lt;/p&gt;</comment>
                            <comment id="391544" author="nelhage" created="Tue, 30 Jul 2013 03:36:53 +0000"  >&lt;p&gt;Is this planned for backport to 2.4? It&apos;d be really nice to have this fix in our production environment &amp;#8211; slow failovers are pretty painful for us.&lt;/p&gt;</comment>
                            <comment id="387306" author="mattd@10gen.com" created="Tue, 23 Jul 2013 21:13:50 +0000"  >&lt;p&gt;I cherry-picked this commit on top of 2.4.5 and it made the server-9464 repro (test.sh) behave properly (get to a state with a primary).&lt;/p&gt;</comment>
                            <comment id="370721" author="milkie" created="Sat, 29 Jun 2013 13:04:22 +0000"  >&lt;p&gt;Further code changes are required to reduce the delay from 5s.&lt;/p&gt;</comment>
                            <comment id="362612" author="auto" created="Tue, 18 Jun 2013 14:35:11 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9934&quot; title=&quot;Slow failovers after step down due to sleeping rsBackgroundSync&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9934&quot;&gt;&lt;del&gt;SERVER-9934&lt;/del&gt;&lt;/a&gt; remove double getmore when pulling new ops&lt;/p&gt;

&lt;p&gt;This reduces the potential delay in ending the bgsync thread from 10 seconds&lt;br/&gt;
to 5 seconds.  This is important because assumingPrimaryness waits on this loop&lt;br/&gt;
before changing state to PRIMARY after an election.&lt;br/&gt;
To reduce the delay lower than 5 seconds requires further redesign of the code.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/acbae789795b2f628712ada73b2d89684b97da9d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/acbae789795b2f628712ada73b2d89684b97da9d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="360592" author="nelhage" created="Fri, 14 Jun 2013 20:19:37 +0000"  >&lt;p&gt;Ah yeah, that would do it. I was testing on a hacked mongod that replaced the sleep with a timed-wait on a condvar &amp;#8211; that version fails-over quickly in the no-ops case, but hits the double-wait if there&apos;s any work to do. The --force-sync test case is probably worthwhile anyways, since that&apos;s the more interesting case.&lt;/p&gt;</comment>
                            <comment id="360543" author="milkie" created="Fri, 14 Jun 2013 19:29:43 +0000"  >&lt;p&gt;I hit the double wait just with your original test, in fact.  I was quite intrigued when I had removed the 10 second sleep and yet observed no change in the test&apos;s behavior, so I dug deeper.  Thanks very much for writing up a complete test, by the way.&lt;/p&gt;</comment>
                            <comment id="360540" author="nelhage" created="Fri, 14 Jun 2013 19:26:30 +0000"  >&lt;p&gt;Not sure if you have your own test tools, but I added an option to &lt;a href=&quot;https://gist.github.com/nelhage/5777797&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/nelhage/5777797&lt;/a&gt; to generate a bunch of writes immediately before the failover, which will force the new-primary to finish syncing before stepping up, thereby exhibiting the double-wait bug.&lt;/p&gt;</comment>
                            <comment id="360535" author="milkie" created="Fri, 14 Jun 2013 19:21:33 +0000"  >&lt;p&gt;Right, I was about to update this with what I&apos;ve discovered, which was exactly that.  I&apos;m going to fix the double wait.  To fix the single wait, I have some ideas on how to twist it to treat a getmore wait as being already done &amp;#8211; if _appliedBuffer is already true in stopReplicationAndFlushBuffer(), we can make it guaranteed that it will not go false without transitioning to pause mode, in which case we don&apos;t need to wait for it.&lt;/p&gt;</comment>
                            <comment id="360523" author="nelhage" created="Fri, 14 Jun 2013 19:10:45 +0000"  >&lt;p&gt;(Let me know if this should be a separate issue)&lt;/p&gt;

&lt;p&gt;In the case where we &lt;b&gt;do&lt;/b&gt; have ops to replicate, after replication is complete, we end up waiting for two GETMORE await timeouts. produce() does:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                if (!r.moreInCurrentBatch()) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                    ...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        r.more();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                    ...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                if (!r.more())&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                    break;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;In the case where there are no more ops to read, each of those .more() calls will end up issuing a GETMORE, which means we&apos;ll wait 2 AWAIT timeouts (roughly 8-10s) before we return, also delaying the new primary from stepping up.&lt;/p&gt;

&lt;p&gt;The current architecture makes it very difficult to avoid waiting for at least one AWAIT timeout, but there&apos;s no need to wait for two.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="79280">SERVER-9944</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="73107">SERVER-9464</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="82461">SERVER-10225</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 14 Jun 2013 19:21:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 25 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 25 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>onyxmaster</customfieldvalue>
            <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>matt.dannenberg</customfieldvalue>
            <customfieldvalue>nelhage</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmq73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrr553:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>72190</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Run this test script with -vv: &lt;a href=&quot;https://gist.github.com/nelhage/5777797&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/nelhage/5777797&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Observe the failovers are taking 10-12s, even though the cluster is healthy, unloaded, and running across the local interface.&lt;/p&gt;

&lt;p&gt;Look at the logs for a node that has become primary. Observe that after the election completes, the server logs &quot;replSet waiting for replication to finish before becoming primary&quot;, and does not actually step up for up to 10 seconds later.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10166" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Tests Written</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10153"><![CDATA[Unneeded]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hribfj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>