[JAVA-2260] Thread blocked forever when operating on ReplicaSet Created: 25/Jul/16  Updated: 07/Feb/22  Resolved: 13/Sep/16

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: 3.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Frank Wen Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File jstack_tomcat.txt    
Issue Links:
Related
is related to JAVA-2434 Empty insert causes subsequent querie... Closed
Case:

 Description   

I use mongodb java driver 3.3.0. When the Java code operating on ReplicaSet, whatever query or update, many threads blocked and never return. Use jstack to observe the threads, it's like:

"catalina-exec-268" #354 daemon prio=5 os_prio=0 tid=0x00007fe0d8052000 nid=0x6553 waiting on condition [0x00007fe0934c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000007a36970b0> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:250)
    at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:105)
    at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:438)
    at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:112)
    at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
    at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
    at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176)
    at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216)
    at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:207)
    at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:113)
    at com.mongodb.operation.FindOperation$1.call(FindOperation.java:488)
    at com.mongodb.operation.FindOperation$1.call(FindOperation.java:483)
    at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:241)
    at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:214)
    at com.mongodb.operation.FindOperation.execute(FindOperation.java:483)
    at com.mongodb.operation.FindOperation.execute(FindOperation.java:80)
    at com.mongodb.Mongo.execute(Mongo.java:818)
    at com.mongodb.Mongo$2.execute(Mongo.java:805)
    at com.mongodb.DBCollection.findOne(DBCollection.java:777)
    at com.mongodb.DBCollection.findOne(DBCollection.java:747)
    at com.mongodb.DBCollection.findOne(DBCollection.java:694)
    at com.bugull.mongo.BuguDao.findOne(BuguDao.java:588)
    at com.bugull.farm.web.module.pad.PadApiController.place(PadApiController.java:344)
    at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:781)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:721)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.bugull.farm.web.filter.ApiFilter.doFilter(ApiFilter.java:42)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
    at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
  • locked <0x00000007bb13a0d0> (a org.apache.tomcat.util.net.NioChannel)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

Lots of theads are WAITING. Read the attachment for more details.
I change driver to 3.2.2, problems are still the same. Then I change driver to 2.14.3, everything works fine.
If connected to a single mongodb server, instead of replica set, all the driver works fine.
So I think it's a bug of the 3.x version.
I use JDK 8u71 and Tomcat 8.0.30. Mongodb server is 3.2.0.



 Comments   
Comment by Jeffrey Yemin [ 13/Sep/16 ]

As we haven't heard back from in some time in answer to a request for further information, I'm going to close this issue. If you have more information to share, I'll be happy to re-open it.

Comment by Jeffrey Yemin [ 25/Jul/16 ]

Hi Frank,

Thanks for reporting this issue. This is the first report of its kind, so can you answer a few questions to help us get to the bottom of it?

  • Are you able to reproduce this reliably, or is it something that happens intermittently? If the latter, are there any obvious triggers?
  • When did you first start seeing this issue, and did anything significant change prior to that point in the application or the environment?
  • You mentioned in the title that the application is connecting to a replica set. Are you able to reproduce this against a standalone as well?
  • Can you share server logs? If there is sensitive information we can arrange for you to upload the logs to a secure server.
  • If necessary, would you be able to try reproducing this with a patched driver that contains additional debug logs?

Thanks,
Jeff

Generated at Thu Feb 08 08:56:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.