I got an error 'Query failed with error code -5 and error message Cursor'. This error make our server load up to 100%.

The java driver throw a no cursor found exception like this:

org.springframework.dao.DataAccessResourceFailureException: Query failed with error code -5 and error message 'Cursor 3180947591717134009 not found on server 10.105.21
.185:27117' on server 10.105.21.185:27117; nested exception is com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 3180
947591717134009 not found on server 10.105.21.185:27117' on server 10.105.21.185:27117
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:90)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2756)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:2666)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:2386)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:2369)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:820)
at com.tmxbase.processserver.dal.dao.TranslationSegmentDAO.loadByDocumentId(TranslationSegmentDAO.java:198)
at com.tmxbase.processserver.dal.dao.TranslationSegmentDAO$$FastClassBySpringCGLIB$$ea3b06ee.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.tmxbase.processserver.dal.dao.TranslationSegmentDAO$$EnhancerBySpringCGLIB$$cd2b90ac.loadByDocumentId(<generated>)
at com.tmxbase.processserver.service.YiCATCountProcess.calculateRepeat(YiCATCountProcess.java:111)
at com.tmxbase.processserver.mq.consumer.QueenConsumer.receiveCalculateRepeat(QueenConsumer.java:281)
at sun.reflect.GeneratedMethodAccessor381.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

 

I got the cursor id in mongos.log.  like this:

 

2020-07-31T10:46:27.153+0800 I COMMAND [conn288126] command tmxbase.TranslationSegmentCollection_50 command: find { find: "TranslationSegmentCollection_50", filter:

{ documentId: "2020072309573638910900167092931_zh-CN" }

, sort: { seqNum: 1 }, $db: "tmxbase", $clusterTime: { clusterTime: Timestamp(1596163425, 4), signature:

{ hash:BinData(0, 2FC1FA509A43BC53338067B27B62EDA5A7E73EA7), keyId: 6828170725514280961 }

}, lsid: { id: UUID("337b7651-68a0-4771-b2a5-d6ea9a683820") } } nShards:2 cursorid:3180947591717134009 numYields:0 nreturned:101 reslen:242627 protocol:op_msg 161686ms
2020-07-31T10:54:12.537+0800 I COMMAND [conn288126] command tmxbase.TranslationSegmentCollection_50 command: getMore { getMore: 3180947591717134009, collection: "TranslationSegmentCollection_50", $db: "tmxbase", $clusterTime: { clusterTime: Timestamp(1596163587, 2), signature:

{ hash: BinData(0, 36E0558D9FFF999917224010F85A8EBBEF3DDC22), keyId: 6828170725514280961 }

}, lsid: { id: UUID("337b7651-68a0-4771-b2a5-d6ea9a683820") } } originatingCommand: { find: "TranslationSegmentCollection_50", filter:

{ documentId: "2020072309573638910900167092931_zh-CN" }

, sort: { seqNum: 1 }, $db: "tmxbase", $clusterTime: { clusterTime: Timestamp(1596163425, 4), signature:

{ hash: BinData(0, 2FC1FA509A43BC53338067B27B62EDA5A7E73EA7), keyId: 6828170725514280961 }

}, lsid: { id: UUID("337b7651-68a0-4771-b2a5-d6ea9a683820") } } nShards:2 cursorid:3180947591717134009 numYields:0 ok:0 errMsg:"cursor id 8226761856981 not found" errName:CursorNotFound errCode:43 reslen:248 protocol:op_msg 465374ms

 

To get the curor, the mongodb consume a lot time and failed.

 

I try to modify the mongodb driver's maxWaiting time to 15s, but this didn't work.

 

Anyone have the same problem?