<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:57:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-2543] Mongo driver exception &quot;Replication is shutting down&quot; on mongo save during replicaset failover and election process</title>
                <link>https://jira.mongodb.org/browse/JAVA-2543</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;Main Question: Should I change application code to resolve exceptions due to primary step down and election of a new primary by opening the connection again or should I be able to handle this using the driver timeout settings and a retry of save on exception?&lt;/p&gt;

&lt;p&gt;Background&lt;/p&gt;

&lt;p&gt;I have a four node replicaset that we are putting through initial development / resilience tests. We are pre-production.. I&apos;m testing failover of the primary by stopping the mongod service. The election occurs and a new primary is elected. I have no errors in the mongod logs and the election takes place.&lt;/p&gt;

&lt;p&gt;However, I have a process that makes a connection to the replicaset over SSL, and when we are running a long running batch jobs performing batch mongo saves, when the service fails over, I get the following error on the client. The exception is caught and but the mongo save throws an exception again in the time before the new primary is elected.&lt;/p&gt;

&lt;p&gt;I have tried resetting values for electionTimeoutMilllis up and down, reducing the heartbeat etc..&lt;/p&gt;

&lt;p&gt;We&apos;re Currently on Mongo 3.4.5 database on Windows and the replica set uses protocol 1.&lt;/p&gt;

&lt;p&gt;The client connection uses the default values for timeout. We use &quot;majority&quot; to write.&lt;br/&gt;
I have also tried setting different values for maxConnectionIdleTime, connectTimeout,&lt;br/&gt;
serverSelectionTimeout but none of these change the exception message we are getting.&lt;/p&gt;

&lt;p&gt;For now I&apos;m simply using the following.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		MongoClient mongoClient =  &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;new&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; MongoClient(&lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;new&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; MongoClientURI(mongoUri));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		mongoClient.setWriteConcern(WriteConcern.MAJORITY);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The stacktrace and config are attaching in configuration.txt but the main error I get is:&lt;/p&gt;

&lt;p&gt;org.springframework.data.mongodb.UncategorizedMongoDbException: Query failed with error code 11600 and error message &apos;interrupted at shutdown&apos; on server xxxx; nested exception is com.mongodb.MongoQueryException: Query failed with error code 11600 and error message &apos;interrupted at shutdown&apos;&lt;/p&gt;

&lt;p&gt;I would be grateful for some ideas whether we could make a settings change which would enable the client code to detect the server problem and then wait for the election process to occur.&lt;/p&gt;

&lt;p&gt;I&apos;ve tried pinging the server after the first mongo save but this hasn&apos;t helped.&lt;/p&gt;

&lt;p&gt;Do we need to re-architect our application code with a more sophisticated approach than retry on initial exception?&lt;/p&gt;

&lt;p&gt;Thanks in advance.&lt;/p&gt;</description>
                <environment>Windows Server 2012 Release 2</environment>
        <key id="396491">JAVA-2543</key>
            <summary>Mongo driver exception &quot;Replication is shutting down&quot; on mongo save during replicaset failover and election process</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="gearoid68">Garrett Donnelly</reporter>
                        <labels>
                    </labels>
                <created>Wed, 21 Jun 2017 10:40:17 +0000</created>
                <updated>Fri, 27 Oct 2023 19:48:24 +0000</updated>
                            <resolved>Tue, 2 Jan 2018 20:35:13 +0000</resolved>
                                    <version>3.4.1</version>
                                                    <component>Cluster Management</component>
                                        <votes>2</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1747460" author="jeff.yemin" created="Fri, 8 Dec 2017 21:53:51 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gearoid68&quot; class=&quot;user-hover&quot; rel=&quot;gearoid68&quot;&gt;gearoid68&lt;/a&gt; I apologize that no one has responded to your initial question.  As it&apos;s been quite a while now, can I ask what your current status is on this issue and if you still need assistance?&lt;/p&gt;

&lt;p&gt;Also, note that the JAVA project is for specific questions about the Java driver, and as this is more of an architectural question about how to properly use MongoDB (albeit in a Java environment), it&apos;s best handled by asking in the &lt;a href=&quot;https://groups.google.com/forum/#!forum/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB user forum&lt;/a&gt;.  &lt;/p&gt;


&lt;p&gt;Regards,&lt;br/&gt;
Jeff&lt;/p&gt;</comment>
                            <comment id="1605206" author="gearoid68" created="Fri, 23 Jun 2017 14:31:16 +0000"  >&lt;p&gt;I have noticed the following in the mongod logs for the new primary after election. Is it possible &lt;/p&gt;

&lt;p&gt;2017-06-15T14:16:04.642+0100 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; My optime is most up-to-date, skipping catch-up and completing transition to primary.&lt;br/&gt;
2017-06-15T14:16:04.642+0100 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; Replication producer stopped after oplog fetcher finished returning a batch from our sync source.  Abandoning this batch of oplog entries and re-evaluating our sync source.&lt;/p&gt;

&lt;p&gt;Is it possible this is linked to the driver error code 91?&lt;/p&gt;

&lt;p&gt;org.springframework.dao.DataIntegrityViolationException: Write failed with error code 91 and error message &apos;Replication is being shut down&apos;; nested exception is com.mongodb.WriteConcernException: Write failed with error code 91 and error message &apos;Replication is being shut down&apos;&lt;/p&gt;

&lt;p&gt;My theory on this is that the error is due to the oplog being too small to facilitate replication in a batch scenario. Data loss is detected by the driver.&lt;/p&gt;

&lt;p&gt;Does anyone know if the missing oplog entries can cause the &quot;Replication is shutting down&quot; message?&lt;/p&gt;


&lt;p&gt;In more detail, the replication of a long running batch in Mongo perhaps needs a larger oplog size. This would be equivalent to sizing the roll-back segment appropriately for batch in Oracle.&lt;/p&gt;

&lt;p&gt;The following advice appears in the Mongo documentation and concerns the example I have of 2,000 medium size documents being saved in batch mode.&lt;/p&gt;

&lt;p&gt;&#8220;For example, you might need to change the oplog size if your applications perform large numbers of multi-updates or deletes in short periods of time&#8221;&lt;/p&gt;

&lt;p&gt;The oplog is a capped collection so what appears to be happening is that some of the oplog entries are being lost on stepdown of the primary. If we were to increase the size of the oplog this would possibly be mitigated.&lt;/p&gt;

&lt;p&gt;There is a rather complex operation to increase the oplog size which could be attempted.&lt;/p&gt;



</comment>
                    </comments>
                    <attachments>
                            <attachment id="159185" name="configuration.txt" size="16164" author="gearoid68" created="Wed, 21 Jun 2017 10:38:05 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrar2v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>