<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:55:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-1881] Java application CPU 100% when mongod replica set member is stopped</title>
                <link>https://jira.mongodb.org/browse/JAVA-1881</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;The non-default configuration options used by the application:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;readPreference: nearest&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;cluster:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  hosts: mongodb-oracle-a1-ci-set1, mongodb-oracle-b1-ci-set1, mongodb-oracle-c1-ci-set1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;codeRegistry: jackson (https://github.com/ylemoigne/mongo-jackson-codec)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We have a replica set with 3 members, identified by host names &lt;em&gt;mongodb-oracle-a1-ci-set1&lt;/em&gt;, &lt;em&gt;mongodb-oracle-b1-ci-set1&lt;/em&gt;, and &lt;em&gt;mongodb-oracle-c1-ci-set1&lt;/em&gt;. There is a Java application running on a fourth host. If I stop any one of the mongod daemons, primary or secondary, while the application is running, CPU for the application process goes to 100% and the application is unusable until restarted. I can replicate the issue consistently.&lt;/p&gt;

&lt;p&gt;Attached is the log output when replicating the issue.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The application is started with &lt;em&gt;mongodb-oracle-a1-ci-set1&lt;/em&gt; daemon stopped. Everything looks fine. The other two daemons are operating normally.&lt;/li&gt;
	&lt;li&gt;At about 20:24:23, I start mongod on &lt;em&gt;mongodb-oracle-a1-ci-set1&lt;/em&gt;. Everything looks fine. The new member joins the replica set and the log indicates that the application connects to it.&lt;/li&gt;
	&lt;li&gt;At about 20:25:05, I stop mongod on &lt;em&gt;mongodb-oracle-a1-ci-set1&lt;/em&gt;. The application process CPU goes to 100%. From the logs, it looks like the application still thinks its connected to the missing member.&lt;br/&gt;
4. At about 20:26:04, I start the mongod on &lt;em&gt;mongodb-oracle-a1-ci-set1&lt;/em&gt;. The application process CPU remains at 100%.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Note that no mongo queries are executed. The application is sitting idle when all this occurs. Once in a bad state, the application is unresponsive and must be restarted.&lt;/p&gt;</description>
                <environment>java version &amp;quot;1.7.0_79&amp;quot;&lt;br/&gt;
OpenJDK Runtime Environment (amzn-2.5.5.1.59.amzn1-x86_64 u79-b14)&lt;br/&gt;
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)&lt;br/&gt;
&lt;br/&gt;
Amazon EC2 instances running Amazon Linux.&lt;br/&gt;
&lt;br/&gt;
mongod 3.0.4&lt;br/&gt;
Async Java driver 3.0.2</environment>
        <key id="215373">JAVA-1881</key>
            <summary>Java application CPU 100% when mongod replica set member is stopped</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="cthomas">Cory Thomas</reporter>
                        <labels>
                    </labels>
                <created>Tue, 7 Jul 2015 20:47:10 +0000</created>
                <updated>Fri, 1 Apr 2016 20:59:28 +0000</updated>
                            <resolved>Fri, 10 Jul 2015 15:48:41 +0000</resolved>
                                    <version>3.0.0</version>
                                    <fixVersion>3.1.0</fixVersion>
                                    <component>Async</component>
                    <component>Connection Management</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="967894" author="jeff.yemin" created="Thu, 16 Jul 2015 16:01:11 +0000"  >&lt;p&gt;Fix is now available in the &lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/releases/tag/r3.0.3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;3.0.3 release&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="963417" author="cthomas" created="Fri, 10 Jul 2015 18:06:11 +0000"  >&lt;p&gt;Awesome! Thanks a bunch!&lt;/p&gt;</comment>
                            <comment id="963266" author="ross@10gen.com" created="Fri, 10 Jul 2015 15:48:41 +0000"  >&lt;p&gt;Fixed in master and on the 3.0.x branch.  The 3.0.3 release is currently scheduled for next week and a snapshot with this fix will be available shortly.&lt;/p&gt;

&lt;p&gt;Alternatively, tests with netty showed that it wasn&apos;t impacted by this bug and is a possible workaround until then.&lt;/p&gt;</comment>
                            <comment id="963264" author="xgen-internal-githook" created="Fri, 10 Jul 2015 15:46:04 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;rozza&apos;, u&apos;name&apos;: u&apos;Ross Lawley&apos;, u&apos;email&apos;: u&apos;ross.lawley@gmail.com&apos;}
&lt;p&gt;Message: Check for premature end of stream in Async&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-1881&quot; title=&quot;Java application CPU 100% when mongod replica set member is stopped&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-1881&quot;&gt;&lt;del&gt;JAVA-1881&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: 3.0.x&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/commit/75d8cda40a52ee42c348a407489b9160bb85b2b9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-java-driver/commit/75d8cda40a52ee42c348a407489b9160bb85b2b9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="963262" author="xgen-internal-githook" created="Fri, 10 Jul 2015 15:45:43 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;rozza&apos;, u&apos;name&apos;: u&apos;Ross Lawley&apos;, u&apos;email&apos;: u&apos;ross.lawley@gmail.com&apos;}
&lt;p&gt;Message: Check for premature end of stream in Async&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-1881&quot; title=&quot;Java application CPU 100% when mongod replica set member is stopped&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-1881&quot;&gt;&lt;del&gt;JAVA-1881&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/commit/e51a5d16a86f0eedb34a7cb48ab0f741f4b1b807&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-java-driver/commit/e51a5d16a86f0eedb34a7cb48ab0f741f4b1b807&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="963016" author="ross@10gen.com" created="Fri, 10 Jul 2015 09:59:31 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cthomas&quot; class=&quot;user-hover&quot; rel=&quot;cthomas&quot;&gt;cthomas&lt;/a&gt;, thats the issue unlike the sync version there isn&apos;t a guarded check for this.&lt;/p&gt;</comment>
                            <comment id="962814" author="cthomas" created="Thu, 9 Jul 2015 23:12:43 +0000"  >&lt;p&gt;I think I have found a clue. I connected a debugger up to my simple app running on an EC2 instance. The problem seems to be located in the &lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/blob/master/driver-core/src/main/com/mongodb/connection/AsynchronousSocketChannelStream.java#L214&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;AsynchronousSocketChannelStream completion handler&lt;/a&gt;. According to &lt;a href=&quot;http://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousSocketChannel.html#read(java.nio.ByteBuffer,%20long,%20java.util.concurrent.TimeUnit,%20A,%20java.nio.channels.CompletionHandler)&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;AsynchronousSocketChannel&lt;/a&gt; documentation, the result value is -1 if no bytes could be read because the channel has reached end of stream. In the debugger, I am seeing the result value as -1 when the driver is trying to execute &lt;tt&gt;ismaster&lt;/tt&gt; command. However, since the driver does not handle the result value, it just tries to read again. The read operation completes immediately because the channel is closed, so the CPU goes way up.&lt;/p&gt;</comment>
                            <comment id="962207" author="cthomas" created="Thu, 9 Jul 2015 15:26:58 +0000"  >&lt;p&gt;I can not replicate the issue with a 3 node replica set running locally (3 mongo daemons running on localhost) on my OSX machine. I can reliably replicate the issue on 4 EC2 linux instances. 3 machines running mongo daemons and one machine running the provided application.&lt;/p&gt;

&lt;p&gt;How I run it:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Build the shaded jar: &lt;tt&gt;mvn clean package&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;Upload the jar: &lt;tt&gt;scp target/mongo-driver-test-1.0-SNAPSHOT.jar host:~&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;Run the jar: &lt;tt&gt;ssh ec2-user@host java -jar /home/ec2-user/mongo-driver-test-1.0-SNAPSHOT.jar&lt;/tt&gt;&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="962058" author="ross@10gen.com" created="Thu, 9 Jul 2015 13:17:16 +0000"  >&lt;p&gt;Unfortunately, I&apos;ve been unable to reproduce with your example code.&lt;/p&gt;

&lt;p&gt;I&apos;ve run it in IntelliJ and as a standalone jar, both times when taking down members of the replicaset I&apos;ve seen the expected connection refused error messages.  I haven&apos;t seen CPU usage rise and I&apos;ve also been able to stop and start the same member of the replicaset multiple times without any visible impact on CPU.  &lt;/p&gt;

&lt;p&gt;Something is missing in my test case - how are you running the local app?&lt;/p&gt;</comment>
                            <comment id="961156" author="cthomas" created="Wed, 8 Jul 2015 15:56:07 +0000"  >&lt;p&gt;It appears I was mistaken when I said stopping the master or a secondary causes the issue. Stopping the master does not appear to cause a problem; only stopping a secondary. In fact, I can kill the master, start it again so it comes up as a secondary (driver handles that fine), and then immediately stop it again (driver goes crazy).&lt;/p&gt;

&lt;p&gt;I attached a very simple java app that I am now using to reproduce the problem. All it does is create a MongoClient object and then sleep.&lt;/p&gt;</comment>
                            <comment id="961019" author="ross@10gen.com" created="Wed, 8 Jul 2015 14:32:45 +0000"  >&lt;p&gt;Thanks, let me try and reproduce.&lt;/p&gt;</comment>
                            <comment id="961006" author="cthomas" created="Wed, 8 Jul 2015 14:26:53 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ross%4010gen.com&quot; class=&quot;user-hover&quot; rel=&quot;ross@10gen.com&quot;&gt;ross@10gen.com&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;It is a web server waiting for requests. It does not do any querying of mongo except when handling a request and there are no requests coming in.&lt;/li&gt;
	&lt;li&gt;I put the driver configuration options in the ticket description. All other options are left at their defaults. I can get the mongod logs if you think it would be helpful.&lt;/li&gt;
	&lt;li&gt;I stopped only one instance in the replica set with init: &lt;tt&gt;service mongod stop&lt;/tt&gt;. The init script does &lt;tt&gt;kill -TERM&lt;/tt&gt; (and a &lt;tt&gt;kill -KILL&lt;/tt&gt; if that doesn&apos;t work).&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="960747" author="ross@10gen.com" created="Wed, 8 Jul 2015 14:12:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cthomas&quot; class=&quot;user-hover&quot; rel=&quot;cthomas&quot;&gt;cthomas&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I have a couple of questions:&lt;/p&gt;

&lt;p&gt;1. Can you describe in more detail what the application is doing - when you say idling I&apos;m not sure what that means?  Is it in some event loop checking mongodb, on a web server awaiting requests.&lt;br/&gt;
2. How you have configured the driver? if possible could you include your connection settings and add the logs from mongod.&lt;br/&gt;
3. How are you stopping the mongod replicaset?  &lt;tt&gt;db.shutdownServer()&lt;/tt&gt;, via &lt;tt&gt;kill &amp;lt;pid&amp;gt;&lt;/tt&gt; or &lt;tt&gt;mongod --shutdown&lt;/tt&gt; etc..&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="80806" name="app.log" size="32555" author="cthomas" created="Tue, 7 Jul 2015 20:47:10 +0000"/>
                            <attachment id="80990" name="mongo-driver-test.tar.gz" size="1861" author="cthomas" created="Wed, 8 Jul 2015 15:56:07 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsb2h3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="522">Java Sprint 20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>