[SERVER-28945] MongoDB load increases when updating 57k large records in one query Created: 24/Apr/17  Updated: 25/Apr/17  Resolved: 25/Apr/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sulabh Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB Server 3.2.8, MongoDB Java Driver 3.0.2, Linux operating system


Participants:

 Description   
Original Summary

MongoDB Crashed while updating 57k records in one query

Original Description

Hi,

We have been struggling with the mongoDB server crash in production which is happening due to one update query which is updating 57000 records in one shot. Application is sending this update query 300 times sequentially but mongoDB server is unusually getting 700-800% of CPU Usage which is causing the mongoDB to crash.

Query which creating the crash to happen:

2017-04-22T18:42:27.896+0000 I WRITE    [conn28248] update hc.Schedule query: { service.bouquetDetails.handle: "dvbBouquet:1.6406E7" } update: { $pull: { service.bouquetDetails: { handle: "dvbBouquet:1.6406E7" } } } keysExamined:57785 docsExamined:57785 nMatched:57785 nModified:57785 keyUpdates:0 writeConflicts:0 numYields:2318 locks:{ Global: { acquireCount: { r: 60104, w: 60104 } }, Database: { acquireCount: { w: 60104 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 118 } }, Collection: { acquireCount: { w: 2319 } }, Metadata: { acquireCount: { w: 57785 } }, oplog: { acquireCount: { w: 57785 } } } 127328ms
 
2017-04-22T18:53:37.395+0000 I WRITE    [conn24466] update hc.Schedule query: { service.bouquetDetails.handle: "dvbBouquet:1.6409E7" } update: { $pull: { service.bouquetDetails: { handle: "dvbBouquet:1.6409E7" } } } keysExamined:57072 docsExamined:57072 nMatched:57072 nModified:57072 keyUpdates:0 writeConflicts:0 numYields:2249 locks:{ Global: { acquireCount: { r: 59322, w: 59322 } }, Database: { acquireCount: { w: 59322 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 100 } }, Collection: { acquireCount: { w: 2250 } }, Metadata: { acquireCount: { w: 57072 } }, oplog: { acquireCount: { w: 57072 } } } 118500ms

Cluster Setup:

  • We have no sharding.
  • One replica set with three Mongod servers (version: 3.2.8) and two Mongos Server
  • All are running on Linux environment on OpenStack cloud.
  • Application is in java 8 and using mongodb driver 3.0.2

Questing: Is MongoDB can't handle 57000 records update in one query or there is a bug in MongoDB 3.2.8 version with about specified query.

Thanks,
Sulabh



 Comments   
Comment by Kelsey Schubert [ 25/Apr/17 ]

Hi sulabh84,

Thank you for clarifying that a crash did not occur. I've updated the ticket summary accordingly. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

See also our Technical Support page for additional support resources.

Kind regards,
Thomas

Comment by sulabh [ 25/Apr/17 ]

Just got the update that there was a high load on mongo db because of the size of the Document which is been updated is high 33K and the number of documents updated are 57000. So that came to high CPU usage 700-800%.

But mongoDB has not crashed but high load. This is also pointing to the storage type which is used in production.

But still the 700-800% CPU usage is in question.

Comment by Kelsey Schubert [ 25/Apr/17 ]

Hi sulabh84,

I've created a secure upload portal for the log files. Files uploaded to this portal are only visible to MongoDB employees and are routinely deleted after some time.

I understand that sharing log files may be a challenge. However, in order for us to better understand what is going on here, and appropriately investigate this issue, we would need to see how this mongod crashed (i.e. the backtrace) and what events preceded the crash.

Thank you,
Thomas

Comment by sulabh [ 25/Apr/17 ]

Hi Ramon,

It is hard to share the logs since mongoDB node is running on customer site. But what i can give as information is bouquetDetails is a sub document of Service Document and bouquetDetails is an array. So, to update entry we used to use $pull the record and then we used to have another query with $addToSet to add the new document in the array.

These query happen 300 time in a sequential way and whenever this process triggered by the application we used to high spike in the mongoDB node of 700-800% of CPU utilisation.

Is $pull and $addToSet are not recommended to used when the records are higher which match the query criteria?

Thanks,
Sulabh

Comment by Ramon Fernandez Marina [ 24/Apr/17 ]

Can you please provide the logs for the affected node from the last restart until the crash you describe?

Thanks,
Ramón.

Generated at Thu Feb 08 04:19:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.