After creating a new index to prepare for sharding, everything seemed to stop and afterwards new warnings started to show up in the logs that we haven't seen before.

The index was created like this:

db.visits.ensureIndex({"visited": "hashed"}, {background: true})

The cli froze until it was done building the index. The MMS statistics shows 150% lock for the period (how can you have more than 100?). Then when the lock went down again, our application stopped working and a large spike of connections was created.

After restarting all mongos clients it seemed to resume operation again.

Now we are observing these kinds of warnings in the logs:

warning: ClientCursor::yield can't unlock b/c of recursive lock ns: api.visits top: { opid: 1445932706, active: true, secs_running: 0, op: "query", ns: "api.visits", query: { findAndModify: "visits", new: true, fields: { _id: 1 }, query: { ... }, update: { ... }, ...

While no index was removed or modified, only this new index was added, no queries was changed. And doing explain() on the query presented in the findAndModify shows that it is still using a fast index that was there since before.

Why did we get this behaviour and how do we create indexes in the future without having to let everything go to pieces? Do we have to worry about this warning?