<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:21:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9756] network partition where minority was primary and stayed primary</title>
                <link>https://jira.mongodb.org/browse/SERVER-9756</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;With a 3 node replicaset we had a presumed network partition cause an odd issue.&lt;/p&gt;

&lt;p&gt;given n1, n2, n3 with n1 being the PRIMARY we saw the following in our logs:&lt;/p&gt;

&lt;p&gt;Tue May 21 9:53:49.372 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-one:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 9:53:49.372 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 9:53:49.372 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:53:50.425 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: node-three:27017&lt;br/&gt;
Tue May 21 9:53:50.442 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset setting oplog notifier to node-three:27017&lt;br/&gt;
Tue May 21 9:53:50.460 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsGhostSync&amp;#93;&lt;/span&gt; handshake between 1 and node-one:27017&lt;br/&gt;
Tue May 21 9:53:55.378 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:01.390 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:07.394 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:13.400 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:19.407 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:25.416 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:31.421 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-two:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 9:54:37.427 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, we are not freshest&lt;br/&gt;
Tue May 21 9:54:38.027 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-one:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 9:54:38.027 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 9:54:38.219 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info electSelf 2&lt;br/&gt;
Tue May 21 9:54:39.186 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet PRIMARY&lt;br/&gt;
Tue May 21 9:54:39.326 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state PRIMARY&lt;br/&gt;
Tue May 21 9:54:39.718 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state PRIMARY&lt;br/&gt;
...&lt;br/&gt;
(these happened during the entire time every few seconds)&lt;br/&gt;
Tue May 21 14:43:04.286 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; DBClientCursor::init call() failed&lt;br/&gt;
Tue May 21 14:43:04.286 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-one:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 14:43:04.286 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; DBClientCursor::init call() failed&lt;br/&gt;
Tue May 21 14:43:04.286 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-one:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 14:44:57.996 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-three:27017 thinks that we are down&lt;br/&gt;
Tue May 21 14:44:59.919 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-two:27017 thinks that we are down&lt;br/&gt;
...&lt;br/&gt;
Tue May 21 11:46:24.143 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state PRIMARY&lt;br/&gt;
Tue May 21 11:46:24.143 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is up&lt;br/&gt;
Tue May 21 11:46:24.143 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info two primaries (transiently)&lt;br/&gt;
Tue May 21 11:46:24.528 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state PRIMARY&lt;br/&gt;
Tue May 21 11:46:24.528 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is up&lt;br/&gt;
Tue May 21 11:46:24.754 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet closing client sockets after relinquishing primary&lt;br/&gt;
Tue May 21 11:46:24.754 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet relinquishing primary state&lt;br/&gt;
Tue May 21 11:46:24.754 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
Tue May 21 11:46:24.759 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet db exception in producer: 10278 dbclient error communicating with server: node-three:27017&lt;br/&gt;
Tue May 21 11:46:24.782 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset tracking exception: exception: 10278 dbclient error communicating with server: node-three:27017&lt;br/&gt;
Tue May 21 11:46:25.726 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: node-one:27017&lt;br/&gt;
Tue May 21 11:46:25.729 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet our last op time fetched: May 21 14:46:24:a&lt;br/&gt;
Tue May 21 11:46:25.729 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replset source&apos;s GTE: May 21 14:46:25:1&lt;br/&gt;
Tue May 21 11:46:25.744 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet ROLLBACK&lt;br/&gt;
Tue May 21 11:46:25.744 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 0&lt;br/&gt;
Tue May 21 11:46:25.744 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 1&lt;br/&gt;
Tue May 21 11:46:25.744 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 2 FindCommonPoint&lt;br/&gt;
Tue May 21 11:46:25.745 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback diff in end of log times: -1 seconds&lt;br/&gt;
Tue May 21 11:46:25.745 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback our last optime:   May 21 14:46:24:a&lt;br/&gt;
Tue May 21 11:46:25.745 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback their last optime: May 21 14:46:25:2&lt;br/&gt;
Tue May 21 11:46:26.135 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:46:26.171 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:46:28.545 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet replSet rollback 3 fixup&lt;br/&gt;
Tue May 21 11:46:28.545 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback findcommonpoint scanned : 775735&lt;br/&gt;
Tue May 21 11:46:28.545 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback found matching events at May 21 12:54:38:2&lt;br/&gt;
Tue May 21 11:46:34.759 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: node-one:27017&lt;br/&gt;
Tue May 21 11:46:34.766 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet our last op time fetched: May 21 14:46:24:a&lt;br/&gt;
Tue May 21 11:46:34.766 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replset source&apos;s GTE: May 21 14:46:25:1&lt;br/&gt;
Tue May 21 11:46:34.767 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet ROLLBACK&lt;br/&gt;
Tue May 21 11:46:34.767 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 0&lt;br/&gt;
Tue May 21 11:46:34.767 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 1&lt;br/&gt;
Tue May 21 11:46:34.767 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 2 FindCommonPoint&lt;br/&gt;
Tue May 21 11:46:34.770 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback our last optime:   May 21 14:46:24:a&lt;br/&gt;
Tue May 21 11:46:34.771 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback diff in end of log times: -10 seconds&lt;br/&gt;
Tue May 21 11:46:34.771 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback their last optime: May 21 14:46:34:4&lt;br/&gt;
Tue May 21 11:46:37.401 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet replSet rollback 3 fixup&lt;br/&gt;
Tue May 21 11:46:37.401 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback findcommonpoint scanned : 775974&lt;br/&gt;
Tue May 21 11:46:37.401 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback found matching events at May 21 12:54:38:2&lt;br/&gt;
Tue May 21 11:46:45.952 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-two:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 11:46:46.953 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-two:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 11:46:46.954 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 11:46:47.177 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-two:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 11:46:47.177 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 11:48:10.964 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-two:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 11:49:27.780 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 3.5&lt;br/&gt;
Tue May 21 11:49:27.787 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4 n:372470&lt;br/&gt;
Tue May 21 11:49:27.801 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet minvalid=May 21 14:49:27 519bc1b7:10&lt;br/&gt;
Tue May 21 11:49:27.802 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.1 coll resync lucid_prod.UserActivity&lt;br/&gt;
Tue May 21 11:49:27.894 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; build index lucid_prod.UserActivity &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Tue May 21 11:49:27.895 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0 secs&lt;br/&gt;
Tue May 21 11:49:27.908 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet minvalid=May 21 14:49:27 519bc1b7:12&lt;br/&gt;
Tue May 21 11:49:27.908 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.2&lt;br/&gt;
Tue May 21 11:49:27.911 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.3&lt;br/&gt;
Tue May 21 11:49:27.911 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.6&lt;br/&gt;
Tue May 21 11:49:27.911 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.7&lt;br/&gt;
Tue May 21 11:49:35.355 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 3.5&lt;br/&gt;
Tue May 21 11:49:35.363 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet minvalid=May 21 14:49:35 519bc1bf:9&lt;br/&gt;
Tue May 21 11:49:35.363 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4 n:372470&lt;br/&gt;
Tue May 21 11:49:35.363 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.1 coll resync lucid_prod.UserActivity&lt;br/&gt;
Tue May 21 11:49:35.367 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 519bc1bf:9&lt;br/&gt;
Tue May 21 11:49:35.459 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:49:35.459 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is up&lt;br/&gt;
Tue May 21 11:49:35.459 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is up&lt;br/&gt;
Tue May 21 11:49:35.467 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; build index lucid_prod.UserActivity &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Tue May 21 11:49:35.468 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0.001 secs&lt;br/&gt;
Tue May 21 11:49:35.477 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet minvalid=May 21 14:49:35 519bc1bf:c&lt;br/&gt;
Tue May 21 11:49:35.477 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.2&lt;br/&gt;
Tue May 21 11:49:35.479 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.3&lt;br/&gt;
Tue May 21 11:49:35.479 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.6&lt;br/&gt;
Tue May 21 11:49:35.479 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.7&lt;br/&gt;
Tue May 21 11:49:38.134 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; old journal file will be removed: /mnt/apps/mongo/db/journal/j._9&lt;br/&gt;
Tue May 21 11:49:40.222 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-three:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 11:49:41.222 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-three:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 11:49:41.222 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-two:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; can&apos;t see a majority of the set, relinquishing primary&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet closing client sockets after relinquishing primary&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet relinquishing primary state&lt;br/&gt;
Tue May 21 11:49:47.460 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
Tue May 21 11:49:48.372 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info node-three:27017 is down (or slow to respond):&lt;br/&gt;
Tue May 21 11:49:53.224 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet can&apos;t see a majority, will not try to elect self&lt;br/&gt;
Tue May 21 11:49:54.504 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; Socket recv() errno:104 Connection reset by peer 10.10.1.234:27017&lt;br/&gt;
Tue May 21 11:49:54.504 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; SocketException: remote: 10.10.1.234:27017 error: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;10.10.1.234:27017&amp;#93;&lt;/span&gt;&lt;br/&gt;
Tue May 21 11:49:54.530 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet warning rollback error querying for existence of lucid_prod.UserMessageQueue at the primary, ignoring&lt;br/&gt;
Tue May 21 11:49:54.558 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet warning rollback error querying for existence of lucid_prod.VerificationToken at the primary, ignoring&lt;br/&gt;
Tue May 21 11:49:54.644 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 5 d:596790 u:70830&lt;br/&gt;
Tue May 21 11:49:59.576 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet warning rollback error querying for existence of lucid_prod.UserMessageQueue at the primary, ignoring&lt;br/&gt;
Tue May 21 11:49:59.576 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; Socket recv() errno:104 Connection reset by peer 10.10.1.234:27017&lt;br/&gt;
Tue May 21 11:49:59.576 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; SocketException: remote: 10.10.1.234:27017 error: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;10.10.1.234:27017&amp;#93;&lt;/span&gt;&lt;br/&gt;
Tue May 21 11:49:59.584 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet warning rollback error querying for existence of lucid_prod.VerificationToken at the primary, ignoring&lt;br/&gt;
Tue May 21 11:49:59.586 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 5 d:596790 u:70830&lt;br/&gt;
Tue May 21 11:50:29.230 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replset info node-three:27017 heartbeat failed, retrying&lt;br/&gt;
Tue May 21 11:50:48.995 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 6&lt;br/&gt;
Tue May 21 11:50:51.204 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 6&lt;br/&gt;
Tue May 21 11:50:52.301 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 7&lt;br/&gt;
Tue May 21 11:50:54.340 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 7&lt;br/&gt;
Tue May 21 11:51:01.111 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback done&lt;br/&gt;
Tue May 21 11:51:01.157 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet RECOVERING&lt;br/&gt;
Tue May 21 11:51:01.330 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 519bc1b7:12&lt;br/&gt;
Tue May 21 11:51:01.338 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: node-one:27017&lt;br/&gt;
Tue May 21 11:51:01.339 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:51:01.434 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 519bc1b7:12&lt;br/&gt;
Tue May 21 11:51:01.437 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:51:01.437 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is up&lt;br/&gt;
Tue May 21 11:51:01.437 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-one:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 11:51:01.508 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset tracking exception: exception: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;SEND_ERROR&amp;#93;&lt;/span&gt; for 10.10.1.234:27017&lt;br/&gt;
Tue May 21 11:51:01.508 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; Socket say send() errno:9 Bad file descriptor 10.10.1.234:27017&lt;br/&gt;
Tue May 21 11:51:01.529 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
Tue May 21 11:51:01.895 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback done&lt;br/&gt;
Tue May 21 11:51:01.940 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet RECOVERING&lt;br/&gt;
Tue May 21 11:51:02.002 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 519bc1bf:c&lt;br/&gt;
Tue May 21 11:51:02.003 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state DOWN&lt;br/&gt;
Tue May 21 11:51:02.017 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: node-one:27017&lt;br/&gt;
Tue May 21 11:51:02.033 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state ROLLBACK&lt;br/&gt;
Tue May 21 11:51:02.033 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is up&lt;br/&gt;
Tue May 21 11:51:02.035 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-three:27017 would veto with &apos;node-one:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 11:51:02.327 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 519bc1bf:c&lt;br/&gt;
Tue May 21 11:51:02.335 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset setting oplog notifier to node-one:27017&lt;br/&gt;
Tue May 21 11:51:02.508 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt;   caught exception (socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;SEND_ERROR&amp;#93;&lt;/span&gt; for 10.10.1.234:27017) in destructor (~PiggyBackData)&lt;br/&gt;
Tue May 21 11:51:02.508 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset setting oplog notifier to node-one:27017&lt;br/&gt;
Tue May 21 11:51:02.508 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; Socket flush send() errno:9 Bad file descriptor 10.10.1.234:27017&lt;br/&gt;
Tue May 21 11:51:03.334 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state SECONDARY&lt;br/&gt;
Tue May 21 11:51:03.340 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state RECOVERING&lt;br/&gt;
Tue May 21 11:51:03.437 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state SECONDARY&lt;br/&gt;
Tue May 21 11:51:03.438 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, node-two:27017 would veto with &apos;node-one:27017 is trying to elect itself but node-one:27017 is already primary and more up-to-date&apos;&lt;br/&gt;
Tue May 21 11:51:04.003 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state SECONDARY&lt;br/&gt;
Tue May 21 11:51:04.017 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is now in state SECONDARY&lt;br/&gt;
Tue May 21 11:51:04.017 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-three:27017 is up&lt;br/&gt;
Tue May 21 11:51:04.033 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state RECOVERING&lt;br/&gt;
Tue May 21 11:51:04.034 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info electSelf 0&lt;br/&gt;
Tue May 21 11:51:04.317 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet PRIMARY&lt;br/&gt;
Tue May 21 11:51:05.334 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state PRIMARY&lt;br/&gt;
Tue May 21 11:51:06.004 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-one:27017 is now in state PRIMARY&lt;br/&gt;
Tue May 21 11:51:11.141 node-two &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
Tue May 21 11:51:11.342 node-three &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state SECONDARY&lt;br/&gt;
Tue May 21 11:51:12.035 node-one &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member node-two:27017 is now in state SECONDARY&lt;/p&gt;


&lt;p&gt;1. Why would it roll back the majority (N2 and N3) instead of N1?&lt;br/&gt;
2. Why wouldn&apos;t N1 step down if it knew the other 2 thought it was down?&lt;br/&gt;
3. The bson files that were output only had newly created documents. What happened to any updates that happened in the rolled-back period? Are they just gone forever?&lt;/p&gt;</description>
                <environment>centos 6.2</environment>
        <key id="76252">SERVER-9756</key>
            <summary>network partition where minority was primary and stayed primary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="matt.dannenberg">Matt Dannenberg</assignee>
                                    <reporter username="scottbessler">Scott Bessler</reporter>
                        <labels>
                    </labels>
                <created>Wed, 22 May 2013 20:35:14 +0000</created>
                <updated>Wed, 10 Dec 2014 23:18:59 +0000</updated>
                            <resolved>Mon, 15 Jul 2013 15:15:15 +0000</resolved>
                                    <version>2.4.3</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="355390" author="mattd@10gen.com" created="Fri, 7 Jun 2013 16:13:34 +0000"  >&lt;p&gt;The tickets are indeed similar.&lt;/p&gt;

&lt;p&gt;1. When electing a primary, only the freshness (not the quantity) of data is considered. It is possible that N1 received more new data while N2 and N3 were failing over and since N1&apos;s new data is newer than N2 and N3&apos;s new data, N1 remained primary. &lt;/p&gt;

&lt;p&gt;2. That&apos;s been fixed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;3. That&apos;s been fixed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9283&quot; title=&quot;lastHeartbeatRecv flaps between invalid and valid dates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9283&quot;&gt;&lt;del&gt;SERVER-9283&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="347280" author="scottbessler" created="Tue, 28 May 2013 20:40:05 +0000"  >&lt;p&gt;this looks a lot like a manifestation of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9730&quot; title=&quot;Asymmetrical network partition can cause the election of two PRIMARY nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9730&quot;&gt;&lt;del&gt;SERVER-9730&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="71606">SERVER-9339</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="75931">SERVER-9730</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="71139">SERVER-9283</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 7 Jun 2013 16:13:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 36 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 36 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>matt.dannenberg</customfieldvalue>
            <customfieldvalue>scottbessler</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmsaf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrqq73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>69752</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrt4jr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>