<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:59:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-3285] Urgent: Java Driver Logging : &quot;Cluster created with settings&quot; only to one ip</title>
                <link>https://jira.mongodb.org/browse/JAVA-3285</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;End client is Tomcat with webapp which uses Spring&lt;/p&gt;

&lt;p&gt;Recently we converted a single instance mongo with ip address 1.1.1.1 (example) to a replica set with ip addresses 1.1.1.1, 2.2.2.2, 3.3.3.3 (arbiter)&lt;/p&gt;

&lt;p&gt;Connection String :&#160;&lt;/p&gt;

&lt;p&gt;mongodb://username:password@1.1.1.1:27017,2.2.2.2:27017,3.3.3.3:27017/database?replicaSet=replicaset&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;After updating the connection string on application server to standard mongouri, the application is stalling after switchover from 1.1.1.1 to 2.2.2.2.&lt;/p&gt;

&lt;p&gt;We enabled detailed logging for mongo driver using below logging.properties&#160;org.mongodb.driver.level=ALL&#160;&lt;/p&gt;

&lt;p&gt;The observation is that once the war file is loaded and during the &quot;Cluster created with settings &quot; is only connecting to 1.1.1.1 no matter what we do.&#160;&lt;/p&gt;

&lt;p&gt;The most interesting (and concerning) aspect is that we tried removing 1.1.1.1 from all the property files, removed 1.1.1.1 from the replica set (2.2.2.2 was made the PRIMARY) and shutdown mongo on 1.1.1.1 to make sure there is no way the driver could know or understand the presence of 1.1.1.1 but still as soon as I start tomcat, i could see below in the tomcat logs.&lt;/p&gt;

&lt;p&gt;Another issue is that the application is getting stalled after switching over to 2.2.2.2 and we believe understanding and resolving this issue can bring light to the stalling issue.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Note : We have a node js app running which is connecting to this replicaset and it is able to connect to 2.2.2.2 seamlessly after a switchover from 1.1.1.1. It is only on tomcat we are facing this issue.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;May 08, 2019 12:18:33 AM org.apache.catalina.core.StandardEngine startInternal&lt;br/&gt;
INFO: Starting Servlet Engine: Apache Tomcat/7.0.76&lt;br/&gt;
May 08, 2019 12:18:33 AM org.apache.catalina.startup.HostConfig deployWAR&lt;br/&gt;
INFO: Deploying web application archive xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&lt;br/&gt;
May 08, 2019 12:18:34 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile&lt;br/&gt;
INFO: validateJarFile(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/lib/javax.servlet-api-3.0.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class&lt;br/&gt;
May 08, 2019 12:18:44 AM org.apache.catalina.startup.TldConfig execute&lt;br/&gt;
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.&lt;br/&gt;
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Cluster created with settings {hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;1.1.1.1:27017&amp;#93;&lt;/span&gt;, mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=&apos;30000 ms&apos;, maxWaitQueueSize=500}&lt;br/&gt;
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Updating cluster description to {type=UNKNOWN, servers=[&lt;/p&gt;
{address=1.1.1.1:27017, type=UNKNOWN, state=CONNECTING}
&lt;p&gt;]&lt;br/&gt;
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Closing connection connectionId{localValue:1}&lt;br/&gt;
May 08, 2019 12:18:51 AM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Exception in monitor thread while connecting to server 1.1.1.1:27017&lt;br/&gt;
com.mongodb.MongoSocketOpenException: Exception opening socket&lt;br/&gt;
 at com.mongodb.connection.SocketStream.open(SocketStream.java:63)&lt;br/&gt;
 at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)&lt;br/&gt;
 at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113)&lt;br/&gt;
 at java.lang.Thread.run(Thread.java:748)&lt;br/&gt;
Caused by: java.net.ConnectException: Connection refused (Connection refused)&lt;br/&gt;
 at java.net.PlainSocketImpl.socketConnect(Native Method)&lt;br/&gt;
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)&lt;br/&gt;
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)&lt;br/&gt;
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)&lt;br/&gt;
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)&lt;br/&gt;
 at java.net.Socket.connect(Socket.java:589)&lt;br/&gt;
 at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57)&lt;br/&gt;
 at com.mongodb.connection.SocketStream.open(SocketStream.java:58)&lt;br/&gt;
 ... 3 more&lt;/p&gt;</description>
                <environment>Mongo 3.4.20 : Replica set with three nodes one of which is an arbiter&lt;br/&gt;
Java driver 3.4.2&lt;br/&gt;
&lt;br/&gt;
Tomcat 7&lt;br/&gt;
Spring&lt;br/&gt;
</environment>
        <key id="760332">JAVA-3285</key>
            <summary>Urgent: Java Driver Logging : &quot;Cluster created with settings&quot; only to one ip</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="ubuntuuser01">Venkata Anil Kumar Gobburi</reporter>
                        <labels>
                    </labels>
                <created>Wed, 8 May 2019 01:22:09 +0000</created>
                <updated>Fri, 27 Oct 2023 13:21:04 +0000</updated>
                            <resolved>Thu, 9 May 2019 08:47:54 +0000</resolved>
                                    <version>3.4.2</version>
                                                    <component>Session Management</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="2239590" author="ross@10gen.com" created="Thu, 9 May 2019 08:47:33 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ubuntuuser01&quot; class=&quot;user-hover&quot; rel=&quot;ubuntuuser01&quot;&gt;ubuntuuser01&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Glad you got it sorted.  The logging will log all driver usage for the driver level as per the logging configuration.  So had you been logging to a file you would have seen the application logging there.  The case of using MongoDB for storing the logs will increase the logging - as you&apos;ve seen it will also log the logging of its own operations.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                            <comment id="2239099" author="ubuntuuser01" created="Wed, 8 May 2019 20:39:28 +0000"  >&lt;p&gt;Hello Ross,&lt;/p&gt;

&lt;p&gt;Thank you for quick response. We were able to resolve the issue. For the sake of everyone else who may face this issue now or in future I would like to post the root cause here.&lt;/p&gt;

&lt;p&gt;The root cause for our case was that there was log4j properties file which the application was using which triggered the &quot;Cluster created with settings&quot; message and the heartbeat connections there after during the tomcat startup. There were two log4j property files on two different folder locations and I was assuming the application was using one of the property file but turns out it was using the second property file located else where outside the folder we generally use to store all the config files. Apparently I have not scrubbed this file (since i didnt know its presence at the time) because of which logs were showing that connections were made to 1.1.1.1.&#160; All the while we were thinking its the application itself and its traffic which created those log messages which led us down the wrong track of investigation initially&lt;/p&gt;

&lt;p&gt;The reason this issue was difficult to trace by the team was that they made all the necessary changes in the code to connect using mongoUri above and the source of the issue was outside the source code. Interestingly these log4j settings were not used by the dev team locally but were present on the servers which made triaging even more difficult since local setups were working fine during switch over.&lt;/p&gt;

&lt;p&gt;Systematically narrowing down the scope of the root cause by performing different test cases helped identify the issue.&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;Before :&lt;/ins&gt;&lt;/p&gt;

&lt;p&gt;log4j.appender.MongoDB=org.log4mongo.MongoDbAppender&lt;br/&gt;
&lt;b&gt;log4j.appender.MongoDB.hostname=1.1.1.1&lt;/b&gt;&#160;&lt;br/&gt;
&lt;b&gt;log4j.appender.MongoDB.port=27017&lt;/b&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;After:&lt;/ins&gt; (I changed the hostname and the port parameters to include all the three nodes in the replica set)&lt;/p&gt;

&lt;p&gt;log4j.appender.MongoDB=org.log4mongo.MongoDbAppender&lt;br/&gt;
&lt;b&gt;log4j.appender.MongoDB.hostname=1.1.1.1 2.2.2.2 3.3.3.3&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;log4j.appender.MongoDB.port=27017 27017 27018&lt;/b&gt;&lt;/p&gt;



&lt;p&gt;After converting the log4j settings to replica set settings below is the new log.&lt;/p&gt;

&lt;p&gt;One can clearly see that the &quot;Cluster created with settings&quot; is picking up all the three ip&apos;s and ports from&#160;log4j.appender.MongoDB.hostname and&#160;log4j.appender.MongoDB.port during startup.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;May 08, 2019 6:36:28 PM org.apache.catalina.startup.TldConfig execute&lt;br/&gt;
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: &lt;b&gt;Cluster created with settings {hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;1.1.1.1:27017, 2.2.2.2:27017, 3.3.3.3:27018&amp;#93;&lt;/span&gt;, mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=&apos;30000 ms&apos;, maxWaitQueueSize=500}&lt;/b&gt;&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Adding discovered server 1.1.1.1:27017 to client view of cluster&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Adding discovered server 2.2.2.2:27017 to client view of cluster&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Adding discovered server 3.3.3.3:27018 to client view of cluster&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Updating cluster description to {type=UNKNOWN, servers=[&lt;/p&gt;
{address=1.1.1.1:27017, type=UNKNOWN, state=CONNECTING}
&lt;p&gt;, {address=2.2.2.2:27017, type=UNKNOWN, state=CONNECTING}, {address=3.3.3.3:27018, type=UNKNOWN, state=CONNECTING}]&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Opened connection &lt;span class=&quot;error&quot;&gt;&amp;#91;connectionId\{localValue:2, serverValue:2379}&amp;#93;&lt;/span&gt; to 3.3.3.3:27018&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Checking status of 3.3.3.3:27018&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Monitor thread successfully connected to server with description ServerDescription{address=3.3.3.3:27018, type=REPLICA_SET_ARBITER, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[3, 4, 20]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=26462773, setName=&apos;replicaset&apos;, canonicalAddress=3.3.3.3:27018, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;2.2.2.2:27017, 1.1.1.1:27017&amp;#93;&lt;/span&gt;, passives=[], arbiters=&lt;span class=&quot;error&quot;&gt;&amp;#91;3.3.3.3:27018&amp;#93;&lt;/span&gt;, primary=&apos;2.2.2.2:27017&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=15, lastWriteDate=Wed May 08 18:36:30 UTC 2019, lastUpdateTimeNanos=4709674695100178}&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Discovered cluster type of REPLICA_SET&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Updating cluster description to {type=REPLICA_SET, servers=[&lt;/p&gt;
{address=1.1.1.1:27017, type=UNKNOWN, state=CONNECTING}
&lt;p&gt;, {address=2.2.2.2:27017, type=UNKNOWN, state=CONNECTING}, {address=3.3.3.3:27018, type=REPLICA_SET_ARBITER, roundTripTime=26.5 ms, state=CONNECTED}]&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Opened connection &lt;span class=&quot;error&quot;&gt;&amp;#91;connectionId\{localValue:1, serverValue:244}&amp;#93;&lt;/span&gt; to 1.1.1.1:27017&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
FINE: Checking status of 1.1.1.1:27017&lt;br/&gt;
May 08, 2019 6:36:40 PM com.mongodb.diagnostics.logging.JULLogger log&lt;br/&gt;
INFO: Opened connection &lt;span class=&quot;error&quot;&gt;&amp;#91;connectionId\{localValue:3, serverValue:1626}&amp;#93;&lt;/span&gt; to 2.2.2.2:27017&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;b&gt;Question :&lt;/b&gt;&lt;/em&gt; Does&#160; logging from com.mongodb.diagnostics.logging.JULLogger represent only the log4j messages or do they represent real traffic as well? Is there a way to differentiate from the debug logs which connection is originating from the log4j and which is from the real application traffic? Want to understand if there is a general best practice in this area.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Thanks and Regards&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2238286" author="ross@10gen.com" created="Wed, 8 May 2019 10:13:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ubuntuuser01&quot; class=&quot;user-hover&quot; rel=&quot;ubuntuuser01&quot;&gt;ubuntuuser01&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the ticket.  All versions of the driver are tested across single, replica sets and sharded replicasets. So it is surprising to hear you are having an issue.  &lt;/p&gt;

&lt;p&gt;If you have removed 1.1.1.1 from your app and configuration and the driver is still trying to connect to it, it highlights there is some cached configuration / war files.  Please could you double check there are no cached  configurations / war files etc.. and add logging to check what uri is being picked up in your application.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hunlvj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>