[SERVER-12545] Everything stopped after adding hashed index Created: 30/Jan/14  Updated: 10/Dec/14  Resolved: 03/Jul/14

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Johnny Boy Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04


Operating System: Linux
Participants:

 Description   

After creating a new index to prepare for sharding, everything seemed to stop and afterwards new warnings started to show up in the logs that we haven't seen before.

The index was created like this:

db.visits.ensureIndex({"visited": "hashed"}, {background: true})

The cli froze until it was done building the index. The MMS statistics shows 150% lock for the period (how can you have more than 100?). Then when the lock went down again, our application stopped working and a large spike of connections was created.

After restarting all mongos clients it seemed to resume operation again.

Now we are observing these kinds of warnings in the logs:

warning: ClientCursor::yield can't unlock b/c of recursive lock ns: api.visits top: { opid: 1445932706, active: true, secs_running: 0, op: "query", ns: "api.visits", query: { findAndModify: "visits", new: true, fields: { _id: 1 }, query: { ... }, update: { ... }, ...

While no index was removed or modified, only this new index was added, no queries was changed. And doing explain() on the query presented in the findAndModify shows that it is still using a fast index that was there since before.

Why did we get this behaviour and how do we create indexes in the future without having to let everything go to pieces? Do we have to worry about this warning?



 Comments   
Comment by Ramon Fernandez Marina [ 03/Jul/14 ]

balboah, since we haven't heard back from you we're going to mark this issue as resolved. If you're still seeing the ClientCursor::yield warnings, please feel free to re-open the ticket and provide the output of db.currentOp() when the warnings appear.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 26/Jun/14 ]

balboah, we haven't heard back from you for some time. If this is still an issue for you, can you please send us the output of db.currentOp() when you see ClientCursor::yield warnings?

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 06/Jun/14 ]

When creating indexes from the mongo shell, the session will block until the index build is complete, even if {background:true} is specified.

As for the warnings in the logs, can you send us the output of db.currentOp() when you see the warnings?

Generated at Thu Feb 08 03:28:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.