[SERVER-18458] High cpu usage on secondary with wiredTiger Created: 13/May/15  Updated: 14/May/15  Resolved: 13/May/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.2, 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Damon Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We have a replica set with 3 servers which install Mongodb 2.6.7.
After update one secondary to 3.0.2 with the wiredTiger engine, the CPU usage will increased to 100% slowly.
But if make the 3.0.2 as Primary, everything is OK.



 Comments   
Comment by Damon [ 14/May/15 ]

Hi, Ramon
The issue still exists, after update to 2.6.9.
And seems it's the issue of the Java drive: JAVA-853 .
We are using the Java driver 2.11.4, here is some log of the Mongodb and Java:

Mongo:

2015-05-14T21:56:09.239+0000 I QUERY [conn13679] killcursors keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

Unknown macro: { r}

}, Database: { acquireCount:

Unknown macro: { r}

}, Collection: { acquireCount:

Unknown macro: { r}

} } 21345ms
2015-05-14T21:56:09.240+0000 I NETWORK [rsBackgroundSync] Socket flush send() errno:9 Bad file descriptor 10.0.1.85:27017
2015-05-14T21:56:09.240+0000 I - [rsBackgroundSync] caught exception (socket exception [SEND_ERROR] for 10.0.1.85:27017) in destructor (~PiggyBackData)
2015-05-14T21:56:09.260+0000 I QUERY [conn13721] killcursors keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

Unknown macro: { r}

}, Database: { acquireCount:

Unknown macro: { r}

}, Collection: { acquireCount:

Unknown macro: { r}

} } 20980ms
2015-05-14T21:56:09.312+0000 I QUERY [conn13529] killcursors keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

Unknown macro: { r}

}, Database: { acquireCount:

Unknown macro: { r}

}, Collection: { acquireCount:

Unknown macro: { r}

} } 10244ms
2015-05-14T21:56:09.313+0000 I NETWORK [conn13068] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.1.102:46615]
2015-05-14T21:56:09.313+0000 I NETWORK [conn12945] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.1.102:46564]
2015-05-14T21:56:09.314+0000 I NETWORK [conn13466] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.1.103:34850]
2015-05-14T21:56:09.314+0000 I NETWORK [conn13391] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.1.103:34823]
2015-05-14T21:56:09.314+0000 I NETWORK [conn13340] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.1.102:46726]

Java:

DBPortPool.ConnectionWaitTimeOut: Connection wait timeout after 15000 ms InvalidDataAccessResourceUsageException: Connection wait timeout after 15000 ms; nested exception is com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 15000 ms

Will re-test after update the Java driver.

Thanks.

Comment by Ramon Fernandez Marina [ 13/May/15 ]

fivoot, this is not a supported configuration. As noted in the documentation, you need to upgrade to 2.6.8 or newer before upgrading to 3.0.

We haven't observed the behavior you describe in our internal upgrade testing. If you're interested in pursuing this please re-test with 2.6.8 or 2.6.9 and upload more information about your secondary (logs at logLevel 1, information from top/iostat, etc.).

Cheers,
Ramón.

Generated at Thu Feb 08 03:47:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.