<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:15:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-47966] Operation interrupted error adding second csrs member on 4.4.0</title>
                <link>https://jira.mongodb.org/browse/SERVER-47966</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I&apos;m getting &quot;operation was interrupted&quot; errors trying to add a second csrs member to a sharded cluster on 4.4.&lt;/p&gt;

&lt;p&gt;I started up a sharded cluster on 4.4.0-rc3 with 1 csrs member. I waited for the first csrs member to be primary with &lt;tt&gt;ismaster:true&lt;/tt&gt;. Then, I started up a second csrs member and tried to add it to the csrs, but the reconfig fails after exactly 10 seconds with &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;(InterruptedDueToReplStateChange) Reconfig finished but failed to propagate to a majority :: caused by :: Current config with {version: 2, term: 1} has not yet propagated to a majority of nodes :: caused by :: operation was interrupted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;br/&gt;
Fuller log snippet from us:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[.info] [cm/action/dbcmd.go:runRsReconfigCmd:237] &amp;lt;shFT_cs1&amp;gt; [13:44:59.824] Running ReplSetReconfig&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[.error] [cm/action/dbcmd.go:runRsReconfigCmd:256] &amp;lt;shFT_cs1&amp;gt; [13:45:09.932] Error running command:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	cmd=[{replSetReconfig map[_id:csrs configsvr:true members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:louisamac:9004 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:louisamac:9019 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1 settings:map[catchUpTakeoverDelayMillis:30000 catchUpTimeoutMillis:-1 chainingAllowed:true electionTimeoutMillis:10000 getLastErrorDefaults:map[w:1 wtimeout:0] getLastErrorModes:map[] heartbeatIntervalMillis:2000 heartbeatTimeoutSecs:10 replicaSetId:ObjectID(&quot;5eb1a5c5950801a13ee31207&quot;)] version:2]} {maxTimeMS 40000}]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	connParams=louisamac:9004 (local=false) : (InterruptedDueToReplStateChange) Reconfig finished but failed to propagate to a majority :: caused by :: Current config with {version: 2, term: 1} has not yet propagated to a majority of nodes :: caused by :: operation was interrupted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;


&lt;p&gt;Entirely possible we&apos;re doing something wrong here on our end, but I&apos;m not sure what step we&apos;re missing. &lt;/p&gt;


&lt;p&gt;Attached the first and second CSRS member logs. Let me know if you need anything else. &lt;/p&gt;</description>
                <environment></environment>
        <key id="1340756">SERVER-47966</key>
            <summary>Operation interrupted error adding second csrs member on 4.4.0</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="pavithra.vetriselvan@mongodb.com">Pavithra Vetriselvan</assignee>
                                    <reporter username="louisa.berger@mongodb.com">Louisa Berger</reporter>
                        <labels>
                    </labels>
                <created>Tue, 5 May 2020 18:05:57 +0000</created>
                <updated>Fri, 27 Oct 2023 13:52:51 +0000</updated>
                            <resolved>Wed, 6 May 2020 22:24:43 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="3072714" author="louisa.berger@10gen.com" created="Wed, 6 May 2020 20:34:16 +0000"  >&lt;p&gt;The rest is being covered in HELP-15470, feel free to close this one out, and thank you for your help!&lt;/p&gt;</comment>
                            <comment id="3070228" author="pavithra.vetriselvan" created="Tue, 5 May 2020 19:47:05 +0000"  >&lt;p&gt;No problem! And sure, feel free to post updated logs if you need insight from us!&lt;/p&gt;</comment>
                            <comment id="3070214" author="louisa.berger@10gen.com" created="Tue, 5 May 2020 19:39:58 +0000"  >&lt;p&gt;Thank you Pavi! The config svr flag was the problem here. We&apos;re now looking into a second problem that might be related. Is it okay if I leave this ticket open until we iron that out? &lt;/p&gt;</comment>
                            <comment id="3070084" author="pavithra.vetriselvan" created="Tue, 5 May 2020 18:44:16 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louisa.berger&quot; class=&quot;user-hover&quot; rel=&quot;louisa.berger&quot;&gt;louisa.berger&lt;/a&gt; It looks like the root cause of this has to do with the new node rejecting the config from the primary because of a &lt;tt&gt;BadValue&lt;/tt&gt; error. This comes from our config checks, so I don&apos;t think this is a bug with the safe reconfig protocol. I think this has to do with how we&apos;re setting up the new node based on the following log:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Nodes being used for config servers must be started with the --configsvr flag&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I&apos;m not too familiar with starting config servers, but could you try starting the node with that flag? I see that the new config already has a &lt;tt&gt;&quot;configcvr&quot;: true&lt;/tt&gt; field and &lt;tt&gt;&quot;writeConcernMajorityJournalDefault&quot;:true&lt;/tt&gt;, which is good because both those fields are necessary. &lt;/p&gt;</comment>
                            <comment id="3070082" author="pavithra.vetriselvan" created="Tue, 5 May 2020 18:41:11 +0000"  >&lt;p&gt;It looks like the new node 9019 rejects the new config:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.864-04:00 I  REPL     21400   [conn21] &quot;Scheduling heartbeat to fetch a new config since we are not a member of our current config&quot;,&quot;attr&quot;:{&quot;senderHost&quot;:&quot;louisamac:9004&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.865-04:00 I  CONNPOOL 22576   [ReplNetwork] &quot;Connecting to {hostAndPort}&quot;,&quot;attr&quot;:{&quot;hostAndPort&quot;:&quot;louisamac:9004&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.911-04:00 W  REPL     21487   [ReplCoord-0] &quot;Not persisting new configuration in heartbeat response to disk because it is invalid&quot;,&quot;attr&quot;:{&quot;error&quot;:{&quot;code&quot;:2,&quot;codeName&quot;:&quot;BadValue&quot;,&quot;errmsg&quot;:&quot;Nodes being used for config servers must be started with the --configsvr flag&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.911-04:00 E  REPL     21490   [ReplCoord-0] &quot;Could not validate configuration received from remote node; Removing self until an acceptable configuration arrives&quot;,&quot;attr&quot;:{&quot;error&quot;:{&quot;code&quot;:2,&quot;codeName&quot;:&quot;BadValue&quot;,&quot;errmsg&quot;:&quot;Nodes being used for config servers must be started with the --configsvr flag&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.911-04:00 I  REPL     21392   [ReplCoord-0] &quot;New replica set config in use&quot;,&quot;attr&quot;:{&quot;config&quot;:{&quot;_id&quot;:&quot;csrs&quot;,&quot;version&quot;:2,&quot;term&quot;:1,&quot;configsvr&quot;:true,&quot;protocolVersion&quot;:1,&quot;writeConcernMajorityJournalDefault&quot;:true,&quot;members&quot;:[{&quot;_id&quot;:0,&quot;host&quot;:&quot;louisamac:9004&quot;,&quot;arbiterOnly&quot;:false,&quot;buildIndexes&quot;:true,&quot;hidden&quot;:false,&quot;priority&quot;:1.0,&quot;tags&quot;:{},&quot;slaveDelay&quot;:0,&quot;votes&quot;:1},{&quot;_id&quot;:1,&quot;host&quot;:&quot;louisamac:9019&quot;,&quot;arbiterOnly&quot;:false,&quot;buildIndexes&quot;:true,&quot;hidden&quot;:false,&quot;priority&quot;:1.0,&quot;tags&quot;:{},&quot;slaveDelay&quot;:0,&quot;votes&quot;:1}],&quot;settings&quot;:{&quot;chainingAllowed&quot;:true,&quot;heartbeatIntervalMillis&quot;:2000,&quot;heartbeatTimeoutSecs&quot;:10,&quot;electionTimeoutMillis&quot;:10000,&quot;catchUpTimeoutMillis&quot;:-1,&quot;catchUpTakeoverDelayMillis&quot;:30000,&quot;getLastErrorModes&quot;:{},&quot;getLastErrorDefaults&quot;:{&quot;w&quot;:1,&quot;wtimeout&quot;:0},&quot;replicaSetId&quot;:{&quot;$oid&quot;:&quot;5eb1a5c5950801a13ee31207&quot;}}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.924-04:00 I  REPL     21394   [ReplCoord-0] &quot;This node is not a member of the config&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.937-04:00 I  REPL     21358   [ReplCoord-0] &quot;Replica set state transition&quot;,&quot;attr&quot;:{&quot;newState&quot;:&quot;REMOVED&quot;,&quot;oldState&quot;:&quot;STARTUP&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Since node 9019 is removed, I think the original primary 9004 will eventually step down (after 10 seconds, which is the default election timeout). The reconfig command is interrupted because we kill the operation on step down:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.913-04:00 I  REPL     21809   [ReplCoord-2] &quot;Can&apos;t see a majority of the set, relinquishing primary&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.922-04:00 I  REPL     21475   [ReplCoord-2] &quot;Stepping down from primary in response to heartbeat&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.922-04:00 I  REPL     21343   [RstlKillOpThread] &quot;Starting to kill user operations&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.922-04:00 I  REPL     21344   [RstlKillOpThread] &quot;Stopped killing user operations&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.922-04:00 I  REPL     21340   [RstlKillOpThread] &quot;State transition ops metrics&quot;,&quot;attr&quot;:{&quot;metrics&quot;:{&quot;lastStateTransition&quot;:&quot;stepDown&quot;,&quot;userOpsKilled&quot;:2,&quot;userOpsRunning&quot;:0}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:09.923-04:00 I  COMMAND  51803   [conn104] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;local.system.replset&quot;,&quot;appName&quot;:&quot;MongoDB Automation Agent v10.15.0 (git: DEV)&quot;,&quot;command&quot;:{&quot;replSetReconfig&quot;:{&quot;_id&quot;:&quot;csrs&quot;,&quot;version&quot;:2,&quot;members&quot;:[{&quot;slaveDelay&quot;:0,&quot;votes&quot;:1,&quot;arbiterOnly&quot;:false,&quot;hidden&quot;:false,&quot;tags&quot;:{},&quot;buildIndexes&quot;:true,&quot;host&quot;:&quot;louisamac:9004&quot;,&quot;priority&quot;:1.0,&quot;_id&quot;:0},{&quot;hidden&quot;:false,&quot;priority&quot;:1.0,&quot;tags&quot;:{},&quot;votes&quot;:1,&quot;host&quot;:&quot;louisamac:9019&quot;,&quot;buildIndexes&quot;:true,&quot;arbiterOnly&quot;:false,&quot;slaveDelay&quot;:0,&quot;_id&quot;:1}],&quot;settings&quot;:{&quot;chainingAllowed&quot;:true,&quot;heartbeatTimeoutSecs&quot;:10,&quot;catchUpTimeoutMillis&quot;:-1,&quot;getLastErrorDefaults&quot;:{&quot;w&quot;:1,&quot;wtimeout&quot;:0},&quot;getLastErrorModes&quot;:{},&quot;electionTimeoutMillis&quot;:10000,&quot;replicaSetId&quot;:{&quot;$oid&quot;:&quot;5eb1a5c5950801a13ee31207&quot;},&quot;heartbeatIntervalMillis&quot;:2000,&quot;catchUpTakeoverDelayMillis&quot;:30000},&quot;configsvr&quot;:true,&quot;protocolVersion&quot;:1},&quot;maxTimeMS&quot;:40000,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;da7e2d9a-165d-4383-8033-10bc0b1b480e&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1588700699,&quot;i&quot;:1}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;admin&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primaryPreferred&quot;}},&quot;numYields&quot;:0,&quot;ok&quot;:0,&quot;errMsg&quot;:&quot;Reconfig finished but failed to propagate to a majority :: caused by :: Current config with {version: 2, term: 1} has not yet propagated to a majority of nodes :: caused by :: operation was interrupted&quot;,&quot;errName&quot;:11602,&quot;errCode&quot;:11602,&quot;reslen&quot;:604,&quot;locks&quot;:{&quot;ParallelBatchWriterMode&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:5}},&quot;ReplicationStateTransition&quot;:{&quot;acquireCount&quot;:{&quot;w&quot;:5}},&quot;Global&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:1,&quot;w&quot;:4}},&quot;Database&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:1,&quot;w&quot;:2,&quot;W&quot;:1}},&quot;Collection&quot;:{&quot;acquireCount&quot;:{&quot;w&quot;:2}},&quot;Mutex&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:3}},&quot;oplog&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:1}}},&quot;flowControl&quot;:{&quot;acquireCount&quot;:2,&quot;timeAcquiringMicros&quot;:3},&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:10097}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3070063" author="pavithra.vetriselvan" created="Tue, 5 May 2020 18:35:34 +0000"  >&lt;p&gt;It looks like the primary (9004) is able to persist the new config to disk and is waiting for a majority to replicate the config:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.906-04:00 I  REPL     21392   [conn104] &quot;New replica set config in use&quot;,&quot;attr&quot;:{&quot;config&quot;:{&quot;_id&quot;:&quot;csrs&quot;,&quot;version&quot;:2,&quot;term&quot;:1,&quot;configsvr&quot;:true,&quot;protocolVersion&quot;:1,&quot;writeConcernMajorityJournalDefault&quot;:true,&quot;members&quot;:[{&quot;_id&quot;:0,&quot;host&quot;:&quot;louisamac:9004&quot;,&quot;arbiterOnly&quot;:false,&quot;buildIndexes&quot;:true,&quot;hidden&quot;:false,&quot;priority&quot;:1.0,&quot;tags&quot;:{},&quot;slaveDelay&quot;:0,&quot;votes&quot;:1},{&quot;_id&quot;:1,&quot;host&quot;:&quot;louisamac:9019&quot;,&quot;arbiterOnly&quot;:false,&quot;buildIndexes&quot;:true,&quot;hidden&quot;:false,&quot;priority&quot;:1.0,&quot;tags&quot;:{},&quot;slaveDelay&quot;:0,&quot;votes&quot;:1}],&quot;settings&quot;:{&quot;chainingAllowed&quot;:true,&quot;heartbeatIntervalMillis&quot;:2000,&quot;heartbeatTimeoutSecs&quot;:10,&quot;electionTimeoutMillis&quot;:10000,&quot;catchUpTimeoutMillis&quot;:-1,&quot;catchUpTakeoverDelayMillis&quot;:30000,&quot;getLastErrorModes&quot;:{},&quot;getLastErrorDefaults&quot;:{&quot;w&quot;:1,&quot;wtimeout&quot;:0},&quot;replicaSetId&quot;:{&quot;$oid&quot;:&quot;5eb1a5c5950801a13ee31207&quot;}}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.909-04:00 I  REPL     21393   [conn104] &quot;Found self in config&quot;,&quot;attr&quot;:{&quot;hostAndPort&quot;:&quot;louisamac:9004&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:44:59.910-04:00 I  REPL     4508702 [conn104] &quot;Waiting for the current config to propagate to a majority of nodes.&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;A bit later, we see failed heartbeats to the new node we added (9019):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;| 2020-05-05T13:45:01.922-04:00 I  REPL_HB  23974   [ReplCoord-2] &quot;Heartbeat failed after max retries&quot;,&quot;attr&quot;:{&quot;target&quot;:&quot;louisamac:9019&quot;,&quot;maxHeartbeatRetries&quot;:2,&quot;error&quot;:{&quot;code&quot;:93,&quot;codeName&quot;:&quot;InvalidReplicaSetConfig&quot;,&quot;errmsg&quot;:&quot;Our replica set configuration is invalid or does not include us&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="258714" name="csrs_primary.log" size="7696464" author="louisa.berger@mongodb.com" created="Tue, 5 May 2020 17:59:54 +0000"/>
                            <attachment id="258713" name="new_csrs.log" size="2542352" author="louisa.berger@mongodb.com" created="Tue, 5 May 2020 17:59:54 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 5 May 2020 18:35:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 40 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 40 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>louisa.berger@mongodb.com</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxjohb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx70hz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3882">Repl 2020-05-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxjaqn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>