Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17921

mongo cluster have five shards.The peak of the business,One server of the cluster load suddenly increased, the abnormal.

XMLWordPrintableJSON

    • Type: Icon: Question Question
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.6.7
    • Component/s: Diagnostics
    • None
    • None
    • 0
    • None
    • None
    • None
    • None
    • None
    • None

      mongos's log:

      2015-04-07T19:20:42.844+0800 [conn1418] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.561+0800 [conn2562] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.563+0800 [conn2692] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.564+0800 [conn2971] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.569+0800 [conn2656] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.570+0800 [conn2376] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.574+0800 [conn2957] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.575+0800 [conn2860] warning: Primary for shard1/xxx.xxx.xxx.xxx:xxxx was down before, bypassing setShardVersio
n. The local replica set view and targeting may be stale.
2015-04-07T19:20:44.578+0800 [conn347] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.579+0800 [conn1177] warning: Failed to connect to xxx.xxx.xxx.xxx:xxxx, reason: errno:106 Transport endpoint is already connected
2015-04-07T19:20:44.589+0800 [conn2669] warning: Primary for shard3/xxx.xxx.xxx.xxx:xxxx was down before, bypassing setShardVersio
n. The local replica set view and targeting may be stale.
2015-04-07T19:20:44.591+0800 [conn82] warning: Primary for shard3/xxx.xxx.xxx.xxx:xxxx was down before, bypassing setShardVersion.
 The local replica set view and targeting may be stale.
2015-04-07T19:20:44.600+0800 [conn2860] warning: Primary for shard1/xxx.xxx.xxx.xxx:xxxx was down before, bypassing setShardVersio
n. The local replica set view and targeting may be stale.

      java's log:

      2015-04-07T19:20:36+08:00 xxxx xxxxx [ERROR] {c.a.d.r.filter.ExceptionFilter} -  [DUBBO] Got unchecked and undeclared exception which called by xxx.xxx.xxx.xxx. service: , exception: org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.mongodb.MongoException$Network: Exception opening the socket, dubbo version: 2.4.9_ZIBO_1.0.1, current host: xxx.xxx.xxx.xxx 	org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.mongodb.MongoException$Network: Exception opening the socket
		at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:56)
		at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1828)
		at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1711)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1522)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1506)
		at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:532)
	Caused by: com.mongodb.MongoException$Network: Exception opening the socket
		at com.mongodb.DBPort.<init>(DBPort.java:117)
		at com.mongodb.DBPort.<init>(DBPort.java:95)
		at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)
		at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)
		at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:183)
		at com.mongodb.ConcurrentPool.createNewAndReleasePermitIfFailure(ConcurrentPool.java:150)
		at com.mongodb.ConcurrentPool.get(ConcurrentPool.java:118)
		at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:75)
		at com.mongodb.DefaultServer.getConnection(DefaultServer.java:61)
		at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:254)
		at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:505)
		at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:448)
		at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:284)
		at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:269)
		at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
		at com.mongodb.TickableDBCollectionImpl.find(TickableDBCollectionImpl.java:78)
		at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
		at com.mongodb.DBCursor._check(DBCursor.java:498)
		at com.mongodb.DBCursor._hasNext(DBCursor.java:621)
		at com.mongodb.DBCursor.hasNext(DBCursor.java:657)
		at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1697)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1522)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1506)
		at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:532)
		at com.mongodb.DBPort.ensureOpen(DBPort.java:287)
		at com.mongodb.DBPort.<init>(DBPort.java:113)
		at com.mongodb.DBPort.<init>(DBPort.java:95)
		at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)
		at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)
		at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:183)
		at com.mongodb.ConcurrentPool.createNewAndReleasePermitIfFailure(ConcurrentPool.java:150)
		at com.mongodb.ConcurrentPool.get(ConcurrentPool.java:118)
		at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:75)
		at com.mongodb.DefaultServer.getConnection(DefaultServer.java:61)
		at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:254)
		at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:505)
		at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:448)
		at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:284)
		at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:269)
		at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
		at com.mongodb.TickableDBCollectionImpl.find(TickableDBCollectionImpl.java:78)
		at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
		at com.mongodb.DBCursor._check(DBCursor.java:498)
		at com.mongodb.DBCursor._hasNext(DBCursor.java:621)
		at com.mongodb.DBCursor.hasNext(DBCursor.java:657)
		at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1697)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1522)
		at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1506)
		at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:532)
	java.lang.RuntimeException: org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.mongodb.MongoException$Network: Exception opening the socket
org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.mongodb.MongoException$Network: Exception opening the socket
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:56)
	at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1828)
	at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1711)
	at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1522)
	at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1506)
	at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:532)
Caused by: com.mongodb.MongoException$Network: Exception opening the socket
	at com.mongodb.DBPort.<init>(DBPort.java:117)
	at com.mongodb.DBPort.<init>(DBPort.java:95)
	at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)
	at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)
	at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:183)
	at com.mongodb.ConcurrentPool.createNewAndReleasePermitIfFailure(ConcurrentPool.java:150)
	at com.mongodb.ConcurrentPool.get(ConcurrentPool.java:118)
	at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:75)
	at com.mongodb.DefaultServer.getConnection(DefaultServer.java:61)
	at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:254)
	at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:505)
	at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:448)
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:284)
	at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:269)
	at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
	at com.mongodb.TickableDBCollectionImpl.find(TickableDBCollectionImpl.java:78)
	at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
	at com.mongodb.DBCursor._check(DBCursor.java:498)
	at com.mongodb.DBCursor._hasNext(DBCursor.java:621)
	at com.mongodb.DBCursor.hasNext(DBCursor.java:657)
	at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1697)
	at com.mongodb.DBPort.ensureOpen(DBPort.java:287)
	at com.mongodb.DBPort.<init>(DBPort.java:113)
	java.lang.RuntimeException: org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.
mongodb.MongoException$Network: Exception opening the socket
org.springframework.dao.DataAccessResourceFailureException: Exception opening the socket; nested exception is com.mongodb.MongoException$Network: Exception opening the socket
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:56)
	at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1828)
	at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1658)
	at org.springframework.data.mongodb.core.MongoTemplate.doFindAndModify(MongoTemplate.java:1586)
	at org.springframework.data.mongodb.core.MongoTemplate.findAndModify(MongoTemplate.java:615)
	at org.springframework.data.mongodb.core.MongoTemplate.findAndModify(MongoTemplate.java:610)
	at com.voxlearning.utopia.dao.mongo.support.AbstractMongoDao.updateById(AbstractMongoDao.java:272)
	at com.voxlearning.utopia.dao.mongo.support.AbstractMongoDao.updateById(AbstractMongoDao.java:246)
Caused by: com.mongodb.MongoException$Network: Exception opening the socket
	at com.mongodb.DBPort.<init>(DBPort.java:117)
	at com.mongodb.DBPort.<init>(DBPort.java:95)
	at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)
	at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)
	at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:183)
	at com.mongodb.ConcurrentPool.createNewAndReleasePermitIfFailure(ConcurrentPool.java:150)
	at com.mongodb.ConcurrentPool.get(ConcurrentPool.java:118)
	at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:75)
	at com.mongodb.DefaultServer.getConnection(DefaultServer.java:61)
	at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:254)
	at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:505)
	at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:448)
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:284)
	at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:269)
	at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
	at com.mongodb.TickableDBCollectionImpl.find(TickableDBCollectionImpl.java:78)
	at com.mongodb.DB.command(DB.java:320)
	at com.mongodb.TickableDBApiLayer.command(TickableDBApiLayer.java:56)
	at com.mongodb.DB.command(DB.java:299)
	at com.mongodb.DB.command(DB.java:374)
	at com.mongodb.DB.command(DB.java:246)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:480)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:424)
	at org.springframework.data.mongodb.core.MongoTemplate$FindAndModifyCallback.doInCollection(MongoTemplate.java:1967)
	at org.springframework.data.mongodb.core.MongoTemplate$FindAndModifyCallback.doInCollection(MongoTemplate.java:1949)
	at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1654)
	at com.mongodb.DBPort.ensureOpen(DBPort.java:287)
	at com.mongodb.DBPort.<init>(DBPort.java:113)

        1. cpu.png
          56 kB
          liangzhang
        2. cpuinfo.txt
          22 kB
          Xiaoguang Wang
        3. crash-0407-09-mongodb.log
          432 kB
          liangzhang
        4. crash-0407-09-sharding.log
          3.09 MB
          liangzhang
        5. interrupts1.log
          1.39 MB
          liangzhang
        6. proc_interrupts.jpeg
          392 kB
          Xiaoguang Wang
        7. soft_irqs.png
          104 kB
          Xiaoguang Wang
        8. softirqs1.log
          320 kB
          liangzhang
        9. top1.log
          80 kB
          liangzhang
        10. vmstat1.log
          9 kB
          liangzhang

            Assignee:
            Unassigned Unassigned
            Reporter:
            liangzhang liangzhang
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: