[SERVER-14375] never ending "split failed Cause: the collection's metadata lock is taken" Created: 27/Jun/14  Updated: 14/Apr/15  Resolved: 14/Apr/15

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kay Agahd Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We are presplitting our chunks which worked always fine until we have upgraded to v2.6.1_linux_64bit. Since then we encounter never ending "split failed Cause: the collection's metadata lock is taken" error messages.
In the log of the mongod holding the chunk to split we find:

2014-06-27T16:42:57.797+0200 [LockPinger] cluster sx210:20020,sx176:20020,sx177:20020 pinged successfully at Fri Jun 27 16:42:57 2014 by distributed lock pinger 'sx210:20020,sx176:20020,sx177:20020/s484:27017:1403879511:112806737', sleeping for 30000ms
2014-06-27T16:42:58.707+0200 [conn17] received splitChunk request: { splitChunk: "offerStore.offer", keyPattern: { _id: 1.0 }, min: { _id: 2929980021 }, max: { _id: MaxKey }, from: "offerStoreDE5", splitKeys: [ { _id: 2930480021 } ], shardId: "offerStore.offer-_id_2929980021", configdb: "sx210:20020,sx176:20020,sx177:20020" }
2014-06-27T16:43:01.738+0200 [conn17] received splitChunk request: { splitChunk: "offerStore.offer", keyPattern: { _id: 1.0 }, min: { _id: 2929980021 }, max: { _id: MaxKey }, from: "offerStoreDE5", splitKeys: [ { _id: 2930480021 } ], shardId: "offerStore.offer-_id_2929980021", configdb: "sx210:20020,sx176:20020,sx177:20020" }
2014-06-27T16:43:04.771+0200 [conn17] received splitChunk request: { splitChunk: "offerStore.offer", keyPattern: { _id: 1.0 }, min: { _id: 2929980021 }, max: { _id: MaxKey }, from: "offerStoreDE5", splitKeys: [ { _id: 2930480021 } ], shardId: "offerStore.offer-_id_2929980021", configdb: "sx210:20020,sx176:20020,sx177:20020" }

Related to the issue https://jira.mongodb.org/browse/SERVER-14047 , where we learnt that we have to shut down the whole cluster to clean-up noTimeOut cursors because they may block chunk moves, we restarted the whole cluster (which already is quite painful!). We left all routers shut down and started only one router on a "private" port so that only the application which does the presplit was connected to the cluster. Nevertheless, we received the same error messages as above!
How it's possible that there is still a metadata lock? How to deblock it? How can we proceed with our presplitting?



 Comments   
Comment by Sam Kleinman (Inactive) [ 14/Apr/15 ]

Sorry for taking so long to get back to you on this. After reviewing this ticket again, it seems like core issue here is that cleanup operations are taking too long. Improving the time in takes clusters to clean up after migration is a driving factor behind a number of ongoing initiatives to improve the sharded cluster implementation: in the mean time, Asaya's suggestions provide ways to work around this limitation. I think it makes the most sense to close this ticket. If you have further questions about this issue, you can post on the mongodb-users group
or Stack Overflow with the mongodb tag.

Regards,
Sam

Comment by Kay Agahd [ 29/Jul/14 ]

Is there a reason that application can't just re-query if the cursor times out?

Yes, there is. The query wouldn't know where to continue unless the result was sorted by _id which is however a very time consuming operation when the resultset is very large.

it's hard to see how doubling the network round trips (with half size batches) could possibly make measureable difference.

Well, half batch sizes would reduce the timeout risk only by 50%. More important, all queries using half size batches would be penalized independant if the client consumes the result fast enough or not. However, only round about 10% (or even less) of the queries may run into the timeout problem.
We would like to reduce the batchsize only when the client can't consume the result fast enough but we can't know this before the query was issued. To my knowledge, batchsize can't be changed once executed the query, am I right?

For us, a proper solution would be to let the client set a server timeout, which is independent from the processing time on the server. A default value of either 10 minutes or no timeout at all is just a big pain.

Good to hear that you came to the same conclusion/suggestion as we do already for a few weeks.

Comment by Asya Kamsky [ 29/Jul/14 ]

Is there a reason that application can't just re-query if the cursor times out?

> However, we would need that the cursor stays open on the server even when the client didn't ask for the next batch within 10 minutes.

You say that if the batches are too small then it takes "too long" to get process the data, but if it takes over 10 minutes to process data then it's hard to see how doubling the network round trips (with half size batches) could possibly make measureable difference.

Regardless, if your queries tend to run for hours then that's always going to be blocking the clean-up thread after migration.

Your only option then is to presplit and locate empty chunks evenly, disable the balancer and then hope that it won't get too unbalanced - or accept that post-migration cleanups will take a very long time.

Comment by Asya Kamsky [ 29/Jul/14 ]

If you check the source code (or comments) you will see that part of migration of a chunk involves waiting for all the current cursors against the "old" data (that's now been moved) to expire:

https://github.com/mongodb/mongo/blob/v2.6/src/mongo/s/d_migrate.cpp#L810

In your case that's happening never.

Any fix is temporary as you say more noTimeout cursors will show up.

Asya

Comment by Kay Agahd [ 29/Jul/14 ]

asya it might be a bit off topic but I've just encountered another blocking issue: we added a new shard and activated the balancer. The balancer got stuck while moving the first chunk to the new shard.
We could read in config.changelog that chunk moves from server s482 were aborted. So I followed your advice and stepped down s482. s490 became primary. But the balancer got stuck again while moving the first chunk and we could read following in config.changelog:

{
        "_id" : "s490-2014-07-29T00:18:06-53d6e83e9e7802f618eda940",
        "server" : "s490",
        "clientAddr" : "172.16.65.203:57363",
        "time" : ISODate("2014-07-29T00:18:06.783Z"),
        "what" : "moveChunk.from",
        "ns" : "offerStore.offer",
        "details" : {
                "min" : {
                        "_id" : NumberLong(39260450)
                },
                "max" : {
                        "_id" : NumberLong(40065344)
                },
                "step 1 of 6" : 0,
                "note" : "aborted"
        }
}
{
        "_id" : "s490-2014-07-29T00:18:01-53d6e8399e7802f618eda93f",
        "server" : "s490",
        "clientAddr" : "172.16.65.204:43756",
        "time" : ISODate("2014-07-29T00:18:01.836Z"),
        "what" : "moveChunk.from",
        "ns" : "offerStore.offer",
        "details" : {
                "min" : {
                        "_id" : NumberLong(39260450)
                },
                "max" : {
                        "_id" : NumberLong(40065344)
                },
                "step 1 of 6" : 0,
                "step 2 of 6" : 494,
                "step 3 of 6" : 18,
                "step 4 of 6" : 2051,
                "note" : "aborted"
        }
}
{
        "_id" : "s490-2014-07-29T00:17:49-53d6e82d9e7802f618eda93c",
        "server" : "s490",
        "clientAddr" : "172.16.65.202:49716",
        "time" : ISODate("2014-07-29T00:17:49.096Z"),
        "what" : "moveChunk.from",
        "ns" : "offerStore.offer",
        "details" : {
                "min" : {
                        "_id" : NumberLong(39260450)
                },
                "max" : {
                        "_id" : NumberLong(40065344)
                },
                "step 1 of 6" : 0,
                "note" : "aborted"
        }
}

The logs of s490 repeated "moveChunk waiting for full cleanup after move":

2014-07-29T02:12:11.040+0200 [conn632885] moveChunk waiting for full cleanup after move
2014-07-29T02:12:11.040+0200 [conn632885] received moveChunk request: { moveChunk: "offerStore.offer", from: "offerStoreUK3/s482:27017,s486:27017,s490:27017", to: "offerStoreUK4/s558:27017,s559:27017,s560:27017", fromShard: "offerStoreUK3", toShard: "offerStoreUK4", min: { _id: 39260450 }, max: { _id: 40065344 }, maxChunkSizeBytes: 67108864, shardId: "offerStore.offer-_id_39260450", configdb: "sx350:20019,sx351:20019,sx352:20019", secondaryThrottle: true, waitForDelete: true, maxTimeMS: 0 }
2014-07-29T02:12:11.043+0200 [conn632885] about to log metadata event: { _id: "s490-2014-07-29T00:12:11-53d6e6db9e7802f618eda8b4", server: "s490", clientAddr: "172.16.65.202:49719", time: new Date(1406592731043), what: "moveChunk.from", ns: "offerStore.offer", details: { min: { _id: 39260450 }, max: { _id: 40065344 }, step 1 of 6: 0, note: "aborted" } }
2014-07-29T02:12:11.439+0200 [initandlisten] connection accepted from 172.16.65.202:49866 #633336 (725 connections now open)
2014-07-29T02:12:11.440+0200 [initandlisten] connection accepted from 172.16.65.202:49867 #633337 (726 connections now open)
2014-07-29T02:12:11.440+0200 [conn633336]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-07-29T02:12:11.441+0200 [conn633337]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-07-29T02:12:12.608+0200 [conn633276] moveChunk waiting for full cleanup after move
2014-07-29T02:12:12.608+0200 [conn633276] received moveChunk request: { moveChunk: "offerStore.offer", from: "offerStoreUK3/s482:27017,s486:27017,s490:27017", to: "offerStoreUK4/s558:27017,s559:27017,s560:27017", fromShard: "offerStoreUK3", toShard: "offerStoreUK4", min: { _id: 39260450 }, max: { _id: 40065344 }, maxChunkSizeBytes: 67108864, shardId: "offerStore.offer-_id_39260450", configdb: "sx350:20019,sx351:20019,sx352:20019", secondaryThrottle: true, waitForDelete: true, maxTimeMS: 0 }
2014-07-29T02:12:12.610+0200 [conn633276] about to log metadata event: { _id: "s490-2014-07-29T00:12:12-53d6e6dc9e7802f618eda8b5", server: "s490", clientAddr: "172.16.65.204:43914", time: new Date(1406592732610), what: "moveChunk.from", ns: "offerStore.offer", details: { min: { _id: 39260450 }, max: { _id: 40065344 }, step 1 of 6: 0, note: "aborted" } }
2014-07-29T02:12:12.764+0200 [initandlisten] connection accepted from 172.16.65.202:49868 #633338 (727 connections now open)
2014-07-29T02:12:12.766+0200 [conn633338]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-07-29T02:12:14.071+0200 [conn633320] end connection 172.16.66.4:47170 (726 connections now open)
2014-07-29T02:12:14.072+0200 [initandlisten] connection accepted from 172.16.66.4:47174 #633339 (727 connections now open)
2014-07-29T02:12:14.073+0200 [conn633339]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-07-29T02:12:15.801+0200 [initandlisten] connection accepted from 172.16.64.36:37183 #633340 (728 connections now open)
2014-07-29T02:12:15.805+0200 [conn633340]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "xxx", key: "xxx" }
2014-07-29T02:12:15.810+0200 [conn633340] end connection 172.16.64.36:37183 (727 connections now open)
2014-07-29T02:12:17.602+0200 [conn632885] moveChunk waiting for full cleanup after move


Well, it seemed that the cleanup would take forever as usual. Running cleanupOrphaned was not an option because it would wait also forever for the lock, so we removed the corresponding metalock from config.locks. Since then, the balancer was able to move chunks to the new shard.

These experiences show that mongo get stuck very often while migrating chunks, splitting chunks, cleaning up moved chunks etc. Since v2.6 it's seems to be worse than with former versions. What do we wrong that we have so much difficulties with locking? Is this all due to noTimeOut cursors? Do we risk anything by deleting the blocking metalock from config.locks?

Comment by Asya Kamsky [ 28/Jul/14 ]

.bq How could we have force the migration to abort first?

Stepping down the primary on the "from" shard (which would failover to another member of the set) would abort the migration. After that you always want to run cleanupOrphaned on that shard (if it was deleting already) or on "to" shard if the migration hadn't committed yet.

cleanupOrphaned waits for the metadata lock, which is probably why it didn't delete any orphans (they are not orphans until migration is done/aborts).

Comment by Kay Agahd [ 28/Jul/14 ]

asya I forgot to says that maxTimeMS is not helpful for us because it relates to processing time on the server. However, we would need that the cursor stays open on the server even when the client didn't ask for the next batch within 10 minutes. As I wrote in SERVER-14389 already, it's very difficult to know the optimal batchsize: if its too big, the server closes the cursor, if its too small, the iteration over the large result set takes too much time. If we could set the cursor timeout to, say one hour or two, our application could throw away the noTimeOut cursors.

Comment by Kay Agahd [ 28/Jul/14 ]

asya we removed the balancer lock from the config.locks collection while the migration from offerStoreDE5 to offerStoreDE3 was still running because there was no progress for over 70 hours. How could we have force the migration to abort first?

Comment by Asya Kamsky [ 28/Jul/14 ]

maxTimeMS is a user settable parameter introduced in 2.6: http://docs.mongodb.org/manual/reference/method/cursor.maxTimeMS/

To clarify something you said (as it may have impact on other scenarios we're discussing in other tickets:

When we removed the balancer lock from config.lock, presplitting started to run normally

You removed the balancer lock from the config.locks collection while the migration from offerStoreDE5 to offerStoreDE3 was still running? Or did you force the migration to abort first?

Comment by Kay Agahd [ 04/Jul/14 ]

From currentOp we see the migration op which is running for over 70 hours already. We can't kill it.

{
	"opid" : 177515865,
	"active" : true,
	"secs_running" : 258347,
	"op" : "query",
	"ns" : "offerStore.offer",
	"query" : {
		"moveChunk" : "offerStore.offer",
		"from" : "offerStoreDE5/s484:27017,s487:27017,s488:27017",
		"to" : "offerStoreDE3/s124:27017,s129:27017,s132:27017",
		"fromShard" : "offerStoreDE5",
		"toShard" : "offerStoreDE3",
		"min" : {
			"_id" : NumberLong(106276007)
		},
		"max" : {
			"_id" : NumberLong(115364216)
		},
		"maxChunkSizeBytes" : NumberLong(67108864),
		"shardId" : "offerStore.offer-_id_106276007",
		"configdb" : "sx210:20020,sx176:20020,sx177:20020",
		"secondaryThrottle" : true,
		"waitForDelete" : true,
		"maxTimeMS" : 0
	},
	"client" : "172.16.64.161:58522",
	"desc" : "conn95048",
	"threadId" : "0x7f9ed3bac700",
	"connectionId" : 95048,
	"waitingForLock" : false,
	"msg" : "step 5 of 6",
	"numYields" : 11,
	"lockStats" : {
		"timeLockedMicros" : {
			"R" : NumberLong(0),
			"W" : NumberLong(26),
			"r" : NumberLong(6363887),
			"w" : NumberLong(13717)
		},
		"timeAcquiringMicros" : {
			"R" : NumberLong(0),
			"W" : NumberLong(108),
			"r" : NumberLong("2666936677"),
			"w" : NumberLong(4)
		}
	}
},


When we removed the balancer lock from config.lock, presplitting started to run normally

	{
		"_id" : "offerStore.offer",
		"process" : "s487:27017:1404203890:394948458",
		"state" : 2,
		"ts" : ObjectId("53b2865251c391dd7dfef9fe"),
		"when" : ISODate("2014-07-01T09:58:42.513Z"),
		"who" : "s487:27017:1404203890:394948458:conn95048:1512015390",
		"why" : "migrate-{ _id: 106276007 }"
	},


We wonder why maxTimeMS was set to 0, means no limit (see above currentOp) because it does not make any sense to let run one chunk migration for over 70 hours! It's better to kill such operation than to block presplitting or other things. How can we set it to, say one hour?

Comment by Kay Agahd [ 03/Jul/14 ]

thomasr, is there any progress on this? We still have the same problem. We can't presplit because of the issue.

Comment by Kay Agahd [ 27/Jun/14 ]

This entry looks weird because we don't have a process running on sx210:27020. Moreover it's an old mongo version (2.4.6). What does this mean?

{
		"_id" : "sx210:27020",
		"mongoVersion" : "2.4.6",
		"ping" : ISODate("2014-06-05T12:47:58.264Z"),
		"up" : 60,
		"waiting" : true
},

The balancer has the lock. Do you suggest us to turn off the balancer before we execute our presplitting script in order to free the lock?

		
{
	"_id" : "balancer",
	"process" : "sx177:27018:1403879528:1804289383",
	"state" : 2,
	"ts" : ObjectId("53adab85c14ed3f76a6647bc"),
	"when" : ISODate("2014-06-27T17:36:05.379Z"),
	"who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
	"why" : "doing balance round"
},		

Here is also a lock from the migration of chunks. Does this block our presplitting?

		
{
		"_id" : "offerStore.offer",
		"process" : "s216:27017:1403882309:2082556053",
		"state" : 2,
		"ts" : ObjectId("53adab8667d25ab86ba1822e"),
		"when" : ISODate("2014-06-27T17:36:06.017Z"),
		"who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
		"why" : "migrate-{ _id: 11605025 }"
},

The same balancer-lock on the second router:

		
{
		"_id" : "balancer",
		"process" : "sx177:27018:1403879528:1804289383",
		"state" : 2,
		"ts" : ObjectId("53adab85c14ed3f76a6647bc"),
		"when" : ISODate("2014-06-27T17:36:05.379Z"),
		"who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
		"why" : "doing balance round"
},

Same as above but on the second router:

		
{
		"_id" : "offerStore.offer",
		"process" : "s216:27017:1403882309:2082556053",
		"state" : 2,
		"ts" : ObjectId("53adab8667d25ab86ba1822e"),
		"when" : ISODate("2014-06-27T17:36:06.017Z"),
		"who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
		"why" : "migrate-{ _id: 11605025 }"
},

The same balancer-lock on the third router:

		
{
		"_id" : "balancer",
		"process" : "sx177:27018:1403879528:1804289383",
		"state" : 2,
		"ts" : ObjectId("53adab85c14ed3f76a6647bc"),
		"when" : ISODate("2014-06-27T17:36:05.379Z"),
		"who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
		"why" : "doing balance round"
},

Same as above but on the third router:

		
{
		"_id" : "offerStore.offer",
		"process" : "s216:27017:1403882309:2082556053",
		"state" : 2,
		"ts" : ObjectId("53adab8667d25ab86ba1822e"),
		"when" : ISODate("2014-06-27T17:36:06.017Z"),
		"who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
		"why" : "migrate-{ _id: 11605025 }"
}

So, finally, only the first entry lock weird to me. What do you think?

Comment by Kay Agahd [ 27/Jun/14 ]

Hi Thomas,
thanks for your fast reply. This incindent is recurrent. We have 4 sharded system where we are presplitting. We regulary encounter the issue on 3 of 4 clusters.
Here comes the output you requested:

mongos> var config_servers = db.adminCommand("getCmdLineOpts").parsed.sharding.configDB.split(",");
mongos> config_servers.forEach(function (server) {
...     var mongo = new Mongo(server);
...     mongo.auth("admin", "admin", "xxxxx");
...     print(server);
...     print("----------------------------");
...     printjson(mongo.getDB("config").mongos.find().toArray());
...     printjson(mongo.getDB("config").locks.find().toArray());
...     printjson(mongo.getDB("config").lockpings.find().toArray());
...     print();
... })
sx210:20020
----------------------------
[
        {
                "_id" : "sx176:37018",
                "ping" : ISODate("2012-08-29T20:50:04.991Z"),
                "up" : 15781
        },
        {
                "_id" : "sx210:37018",
                "ping" : ISODate("2012-08-29T20:50:19.865Z"),
                "up" : 15776
        },
        {
                "_id" : "sx177:37018",
                "ping" : ISODate("2012-08-29T20:50:23.593Z"),
                "up" : 15750
        },
        {
                "_id" : "ws30810:27018",
                "ping" : ISODate("2013-07-11T15:32:08.980Z"),
                "up" : 656,
                "waiting" : false
        },
        {
                "_id" : "b33:27023",
                "ping" : ISODate("2013-07-31T10:05:07.847Z"),
                "up" : 228,
                "waiting" : true
        },
        {
                "_id" : "hostb11-2:27018",
                "ping" : ISODate("2013-12-19T15:22:50.110Z"),
                "up" : 2523547,
                "waiting" : false
        },
        {
                "_id" : "sx210:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.714Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx176:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.649Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx177:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:36:05.371Z"),
                "up" : 11037,
                "waiting" : false
        },
        {
                "_id" : "sx470:27017",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:24.197Z"),
                "up" : 11108,
                "waiting" : true
        },
        {
                "_id" : "sx210:27020",
                "mongoVersion" : "2.4.6",
                "ping" : ISODate("2014-06-05T12:47:58.264Z"),
                "up" : 60,
                "waiting" : true
        },
        {
                "_id" : "sx470:27015",
                "ping" : ISODate("2014-06-27T17:37:22.379Z"),
                "up" : 11135,
                "waiting" : true,
                "mongoVersion" : "2.6.1"
        }
]
[
        {
                "_id" : "balancer",
                "process" : "sx177:27018:1403879528:1804289383",
                "state" : 2,
                "ts" : ObjectId("53adab85c14ed3f76a6647bc"),
                "when" : ISODate("2014-06-27T17:36:05.379Z"),
                "who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
                "why" : "doing balance round"
        },
        {
                "_id" : "configUpgrade",
                "process" : "sx210:27020:1401972507:1804289383",
                "state" : 0,
                "ts" : ObjectId("5390671b3d4688b53071b855"),
                "when" : ISODate("2014-06-05T12:48:27.825Z"),
                "who" : "sx210:27020:1401972507:1804289383:mongosMain:846930886",
                "why" : "upgrading config database to new format v5"
        },
        {
                "_id" : "OfferManagerRepo.job",
                "process" : "s480:27017:1382980992:1822762246",
                "state" : 0,
                "ts" : ObjectId("52b301a78cf01673922914ef"),
                "when" : ISODate("2013-12-19T14:24:39.027Z"),
                "who" : "s480:27017:1382980992:1822762246:conn2389989:2092921418",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.jobLog",
                "process" : "s131:27017:1393095973:1549804603",
                "state" : 0,
                "ts" : ObjectId("536c26164f130a2492abc9de"),
                "when" : ISODate("2014-05-09T00:49:26.825Z"),
                "who" : "s131:27017:1393095973:1549804603:conn2493399:2104746431",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.task",
                "process" : "sx210:27018:1378897282:1804289383",
                "state" : 0,
                "ts" : ObjectId("52304d83a7e317e17cd6b0da"),
                "when" : ISODate("2013-09-11T11:01:23.728Z"),
                "who" : "sx210:27018:1378897282:1804289383:mongosMain:846930886",
                "why" : "ensuring epochs for config upgrade (52304d82a7e317e17cd6b0d6)"
        },
        {
                "_id" : "offerStore.offer",
                "process" : "s216:27017:1403882309:2082556053",
                "state" : 2,
                "ts" : ObjectId("53adab8667d25ab86ba1822e"),
                "when" : ISODate("2014-06-27T17:36:06.017Z"),
                "who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
                "why" : "migrate-{ _id: 11605025 }"
        },
        {
                "_id" : "authorizationData",
                "state" : 0,
                "who" : "sx210:27018:1401972734:1804289383:conn252925:1957747793",
                "process" : "sx210:27018:1401972734:1804289383",
                "when" : ISODate("2014-06-06T08:50:53.869Z"),
                "why" : "Create user",
                "ts" : ObjectId("539180ed10d4f14361c64be9")
        }
]
[
        {
                "_id" : "hostb11-2:27018:1384943023:557530271",
                "ping" : ISODate("2014-06-27T17:37:13.514Z")
        },
        {
                "_id" : "sx470:27017:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.012Z")
        },
        {
                "_id" : "sx176:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.550Z")
        },
        {
                "_id" : "sx210:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.217Z")
        },
        {
                "_id" : "sx177:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.857Z")
        },
        {
                "_id" : "s484:27017:1403105433:1738930378",
                "ping" : ISODate("2014-06-27T14:25:55.835Z")
        },
        {
                "_id" : "s480:27017:1403105462:1655255111",
                "ping" : ISODate("2014-06-27T14:25:59.042Z")
        },
        {
                "_id" : "s136:27017:1403105462:2110642602",
                "ping" : ISODate("2014-06-27T14:31:05.320Z")
        },
        {
                "_id" : "s129:27017:1403105462:486103315",
                "ping" : ISODate("2014-06-27T14:25:46.113Z")
        },
        {
                "_id" : "s125:27017:1403109667:1529428776",
                "ping" : ISODate("2014-06-27T14:30:59.761Z")
        },
        {
                "_id" : "s132:27017:1403546249:1842495236",
                "ping" : ISODate("2014-06-27T14:30:43.357Z")
        },
        {
                "_id" : "s127:27017:1403546302:1384301518",
                "ping" : ISODate("2014-06-27T14:25:46.045Z")
        },
        {
                "_id" : "sx470:27015:1403878642:1804289383",
                "ping" : ISODate("2014-06-27T14:21:54.769Z")
        },
        {
                "_id" : "sx470:27015:1403878933:1804289383",
                "ping" : ISODate("2014-06-27T14:22:13.171Z")
        },
        {
                "_id" : "sx470:27015:1403879230:1804289383",
                "ping" : ISODate("2014-06-27T14:29:41.239Z")
        },
        {
                "_id" : "s487:27017:1403879236:286455613",
                "ping" : ISODate("2014-06-27T14:30:48.304Z")
        },
        {
                "_id" : "sx176:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.049Z")
        },
        {
                "_id" : "sx177:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.083Z")
        },
        {
                "_id" : "sx210:27018:1403879262:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.151Z")
        },
        {
                "_id" : "sx470:27017:1403879283:1804289383",
                "ping" : ISODate("2014-06-27T14:30:35.137Z")
        },
        {
                "_id" : "sx470:27015:1403879507:1804289383",
                "ping" : ISODate("2014-06-27T17:37:10.712Z")
        },
        {
                "_id" : "s484:27017:1403879511:112806737",
                "ping" : ISODate("2014-06-27T17:37:14.604Z")
        },
        {
                "_id" : "sx177:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx176:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx210:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.443Z")
        },
        {
                "_id" : "sx470:27017:1403879536:1804289383",
                "ping" : ISODate("2014-06-27T17:37:09.994Z")
        },
        {
                "_id" : "s129:27017:1403882252:681112909",
                "ping" : ISODate("2014-06-27T17:37:04.399Z")
        },
        {
                "_id" : "s216:27017:1403882309:2082556053",
                "ping" : ISODate("2014-06-27T17:37:02.144Z")
        },
        {
                "_id" : "s136:27017:1403882339:414926390",
                "ping" : ISODate("2014-06-27T17:37:02.999Z")
        },
        {
                "_id" : "s479:27017:1403882398:1167576080",
                "ping" : ISODate("2014-06-27T17:37:01.223Z")
        }
]
 
sx176:20020
----------------------------
[
        {
                "_id" : "sx176:37018",
                "ping" : ISODate("2012-08-29T20:50:04.991Z"),
                "up" : 15781
        },
        {
                "_id" : "sx210:37018",
                "ping" : ISODate("2012-08-29T20:50:19.865Z"),
                "up" : 15776
        },
        {
                "_id" : "sx177:37018",
                "ping" : ISODate("2012-08-29T20:50:23.593Z"),
                "up" : 15750
        },
        {
                "_id" : "ws30810:27018",
                "ping" : ISODate("2013-07-11T15:32:08.980Z"),
                "up" : 656,
                "waiting" : false
        },
        {
                "_id" : "b33:27023",
                "ping" : ISODate("2013-07-31T10:05:07.847Z"),
                "up" : 228,
                "waiting" : true
        },
        {
                "_id" : "hostb11-2:27018",
                "ping" : ISODate("2013-12-19T15:22:50.110Z"),
                "up" : 2523547,
                "waiting" : false
        },
        {
                "_id" : "sx210:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.714Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx176:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.649Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx177:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:36:05.371Z"),
                "up" : 11037,
                "waiting" : false
        },
        {
                "_id" : "sx470:27017",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:24.197Z"),
                "up" : 11108,
                "waiting" : true
        },
        {
                "_id" : "sx210:27020",
                "mongoVersion" : "2.4.6",
                "ping" : ISODate("2014-06-05T12:47:58.264Z"),
                "up" : 60,
                "waiting" : true
        },
        {
                "_id" : "sx470:27015",
                "ping" : ISODate("2014-06-27T17:37:22.379Z"),
                "up" : 11135,
                "waiting" : true,
                "mongoVersion" : "2.6.1"
        }
]
[
        {
                "_id" : "balancer",
                "process" : "sx177:27018:1403879528:1804289383",
                "state" : 2,
                "ts" : ObjectId("53adab85c14ed3f76a6647bc"),
                "when" : ISODate("2014-06-27T17:36:05.379Z"),
                "who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
                "why" : "doing balance round"
        },
        {
                "_id" : "configUpgrade",
                "process" : "sx210:27020:1401972507:1804289383",
                "state" : 0,
                "ts" : ObjectId("5390671b3d4688b53071b855"),
                "when" : ISODate("2014-06-05T12:48:27.825Z"),
                "who" : "sx210:27020:1401972507:1804289383:mongosMain:846930886",
                "why" : "upgrading config database to new format v5"
        },
        {
                "_id" : "OfferManagerRepo.job",
                "process" : "s480:27017:1382980992:1822762246",
                "state" : 0,
                "ts" : ObjectId("52b301a78cf01673922914ef"),
                "when" : ISODate("2013-12-19T14:24:39.027Z"),
                "who" : "s480:27017:1382980992:1822762246:conn2389989:2092921418",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.jobLog",
                "process" : "s131:27017:1393095973:1549804603",
                "state" : 0,
                "ts" : ObjectId("536c26164f130a2492abc9de"),
                "when" : ISODate("2014-05-09T00:49:26.825Z"),
                "who" : "s131:27017:1393095973:1549804603:conn2493399:2104746431",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.task",
                "process" : "sx210:27018:1378897282:1804289383",
                "state" : 0,
                "ts" : ObjectId("52304d83a7e317e17cd6b0da"),
                "when" : ISODate("2013-09-11T11:01:23.728Z"),
                "who" : "sx210:27018:1378897282:1804289383:mongosMain:846930886",
                "why" : "ensuring epochs for config upgrade (52304d82a7e317e17cd6b0d6)"
        },
        {
                "_id" : "offerStore.offer",
                "process" : "s216:27017:1403882309:2082556053",
                "state" : 2,
                "ts" : ObjectId("53adab8667d25ab86ba1822e"),
                "when" : ISODate("2014-06-27T17:36:06.017Z"),
                "who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
                "why" : "migrate-{ _id: 11605025 }"
        },
        {
                "_id" : "authorizationData",
                "state" : 0,
                "who" : "sx210:27018:1401972734:1804289383:conn252925:1957747793",
                "process" : "sx210:27018:1401972734:1804289383",
                "when" : ISODate("2014-06-06T08:50:53.869Z"),
                "why" : "Create user",
                "ts" : ObjectId("539180ed10d4f14361c64be9")
        }
]
[
        {
                "_id" : "hostb11-2:27018:1384943023:557530271",
                "ping" : ISODate("2014-06-27T17:37:13.514Z")
        },
        {
                "_id" : "sx177:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.857Z")
        },
        {
                "_id" : "sx470:27017:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.012Z")
        },
        {
                "_id" : "sx176:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.550Z")
        },
        {
                "_id" : "sx210:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.217Z")
        },
        {
                "_id" : "s484:27017:1403105433:1738930378",
                "ping" : ISODate("2014-06-27T14:25:55.835Z")
        },
        {
                "_id" : "s480:27017:1403105462:1655255111",
                "ping" : ISODate("2014-06-27T14:25:59.042Z")
        },
        {
                "_id" : "s136:27017:1403105462:2110642602",
                "ping" : ISODate("2014-06-27T14:31:05.320Z")
        },
        {
                "_id" : "s129:27017:1403105462:486103315",
                "ping" : ISODate("2014-06-27T14:25:46.113Z")
        },
        {
                "_id" : "s125:27017:1403109667:1529428776",
                "ping" : ISODate("2014-06-27T14:30:59.761Z")
        },
        {
                "_id" : "s132:27017:1403546249:1842495236",
                "ping" : ISODate("2014-06-27T14:30:43.357Z")
        },
        {
                "_id" : "s127:27017:1403546302:1384301518",
                "ping" : ISODate("2014-06-27T14:25:46.045Z")
        },
        {
                "_id" : "sx470:27015:1403878642:1804289383",
                "ping" : ISODate("2014-06-27T14:21:54.769Z")
        },
        {
                "_id" : "sx470:27015:1403878933:1804289383",
                "ping" : ISODate("2014-06-27T14:22:13.171Z")
        },
        {
                "_id" : "sx470:27015:1403879230:1804289383",
                "ping" : ISODate("2014-06-27T14:29:41.239Z")
        },
        {
                "_id" : "s487:27017:1403879236:286455613",
                "ping" : ISODate("2014-06-27T14:30:48.304Z")
        },
        {
                "_id" : "sx176:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.049Z")
        },
        {
                "_id" : "sx177:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.083Z")
        },
        {
                "_id" : "sx210:27018:1403879262:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.151Z")
        },
        {
                "_id" : "sx470:27017:1403879283:1804289383",
                "ping" : ISODate("2014-06-27T14:30:35.137Z")
        },
        {
                "_id" : "sx470:27015:1403879507:1804289383",
                "ping" : ISODate("2014-06-27T17:37:10.712Z")
        },
        {
                "_id" : "s484:27017:1403879511:112806737",
                "ping" : ISODate("2014-06-27T17:37:14.604Z")
        },
        {
                "_id" : "sx177:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx176:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx210:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.443Z")
        },
        {
                "_id" : "sx470:27017:1403879536:1804289383",
                "ping" : ISODate("2014-06-27T17:37:09.994Z")
        },
        {
                "_id" : "s129:27017:1403882252:681112909",
                "ping" : ISODate("2014-06-27T17:37:04.399Z")
        },
        {
                "_id" : "s216:27017:1403882309:2082556053",
                "ping" : ISODate("2014-06-27T17:37:02.144Z")
        },
        {
                "_id" : "s136:27017:1403882339:414926390",
                "ping" : ISODate("2014-06-27T17:37:02.999Z")
        },
        {
                "_id" : "s479:27017:1403882398:1167576080",
                "ping" : ISODate("2014-06-27T17:37:01.223Z")
        }
]
 
sx177:20020
----------------------------
[
        {
                "_id" : "sx176:37018",
                "ping" : ISODate("2012-08-29T20:50:04.991Z"),
                "up" : 15781
        },
        {
                "_id" : "sx210:37018",
                "ping" : ISODate("2012-08-29T20:50:19.865Z"),
                "up" : 15776
        },
        {
                "_id" : "sx177:37018",
                "ping" : ISODate("2012-08-29T20:50:23.593Z"),
                "up" : 15750
        },
        {
                "_id" : "ws30810:27018",
                "ping" : ISODate("2013-07-11T15:32:08.980Z"),
                "up" : 656,
                "waiting" : false
        },
        {
                "_id" : "b33:27023",
                "ping" : ISODate("2013-07-31T10:05:07.847Z"),
                "up" : 228,
                "waiting" : true
        },
        {
                "_id" : "hostb11-2:27018",
                "ping" : ISODate("2013-12-19T15:22:50.110Z"),
                "up" : 2523547,
                "waiting" : false
        },
        {
                "_id" : "sx210:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.714Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx176:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:23.649Z"),
                "up" : 11115,
                "waiting" : true
        },
        {
                "_id" : "sx177:27018",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:36:05.371Z"),
                "up" : 11037,
                "waiting" : false
        },
        {
                "_id" : "sx470:27017",
                "mongoVersion" : "2.6.1",
                "ping" : ISODate("2014-06-27T17:37:24.197Z"),
                "up" : 11108,
                "waiting" : true
        },
        {
                "_id" : "sx210:27020",
                "mongoVersion" : "2.4.6",
                "ping" : ISODate("2014-06-05T12:47:58.264Z"),
                "up" : 60,
                "waiting" : true
        },
        {
                "_id" : "sx470:27015",
                "ping" : ISODate("2014-06-27T17:37:22.379Z"),
                "up" : 11135,
                "waiting" : true,
                "mongoVersion" : "2.6.1"
        }
]
[
        {
                "_id" : "balancer",
                "process" : "sx177:27018:1403879528:1804289383",
                "state" : 2,
                "ts" : ObjectId("53adab85c14ed3f76a6647bc"),
                "when" : ISODate("2014-06-27T17:36:05.379Z"),
                "who" : "sx177:27018:1403879528:1804289383:Balancer:846930886",
                "why" : "doing balance round"
        },
        {
                "_id" : "configUpgrade",
                "process" : "sx210:27020:1401972507:1804289383",
                "state" : 0,
                "ts" : ObjectId("5390671b3d4688b53071b855"),
                "when" : ISODate("2014-06-05T12:48:27.825Z"),
                "who" : "sx210:27020:1401972507:1804289383:mongosMain:846930886",
                "why" : "upgrading config database to new format v5"
        },
        {
                "_id" : "OfferManagerRepo.job",
                "process" : "s480:27017:1382980992:1822762246",
                "state" : 0,
                "ts" : ObjectId("52b301a78cf01673922914ef"),
                "when" : ISODate("2013-12-19T14:24:39.027Z"),
                "who" : "s480:27017:1382980992:1822762246:conn2389989:2092921418",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.jobLog",
                "process" : "s131:27017:1393095973:1549804603",
                "state" : 0,
                "ts" : ObjectId("536c26164f130a2492abc9de"),
                "when" : ISODate("2014-05-09T00:49:26.825Z"),
                "who" : "s131:27017:1393095973:1549804603:conn2493399:2104746431",
                "why" : "migrate-{ _id: MinKey }"
        },
        {
                "_id" : "OfferManagerRepo.task",
                "process" : "sx210:27018:1378897282:1804289383",
                "state" : 0,
                "ts" : ObjectId("52304d83a7e317e17cd6b0da"),
                "when" : ISODate("2013-09-11T11:01:23.728Z"),
                "who" : "sx210:27018:1378897282:1804289383:mongosMain:846930886",
                "why" : "ensuring epochs for config upgrade (52304d82a7e317e17cd6b0d6)"
        },
        {
                "_id" : "offerStore.offer",
                "process" : "s216:27017:1403882309:2082556053",
                "state" : 2,
                "ts" : ObjectId("53adab8667d25ab86ba1822e"),
                "when" : ISODate("2014-06-27T17:36:06.017Z"),
                "who" : "s216:27017:1403882309:2082556053:conn29:1058363483",
                "why" : "migrate-{ _id: 11605025 }"
        },
        {
                "_id" : "authorizationData",
                "state" : 0,
                "who" : "sx210:27018:1401972734:1804289383:conn252925:1957747793",
                "process" : "sx210:27018:1401972734:1804289383",
                "when" : ISODate("2014-06-06T08:50:53.869Z"),
                "why" : "Create user",
                "ts" : ObjectId("539180ed10d4f14361c64be9")
        }
]
[
        {
                "_id" : "hostb11-2:27018:1384943023:557530271",
                "ping" : ISODate("2014-06-27T17:37:13.514Z")
        },
        {
                "_id" : "sx177:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.857Z")
        },
        {
                "_id" : "sx470:27017:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.012Z")
        },
        {
                "_id" : "sx210:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:49.217Z")
        },
        {
                "_id" : "sx176:27018:1403105432:1804289383",
                "ping" : ISODate("2014-06-27T14:25:34.550Z")
        },
        {
                "_id" : "s484:27017:1403105433:1738930378",
                "ping" : ISODate("2014-06-27T14:25:55.835Z")
        },
        {
                "_id" : "s480:27017:1403105462:1655255111",
                "ping" : ISODate("2014-06-27T14:25:59.042Z")
        },
        {
                "_id" : "s136:27017:1403105462:2110642602",
                "ping" : ISODate("2014-06-27T14:31:05.320Z")
        },
        {
                "_id" : "s129:27017:1403105462:486103315",
                "ping" : ISODate("2014-06-27T14:25:46.113Z")
        },
        {
                "_id" : "s125:27017:1403109667:1529428776",
                "ping" : ISODate("2014-06-27T14:30:59.761Z")
        },
        {
                "_id" : "s132:27017:1403546249:1842495236",
                "ping" : ISODate("2014-06-27T14:30:43.357Z")
        },
        {
                "_id" : "s127:27017:1403546302:1384301518",
                "ping" : ISODate("2014-06-27T14:25:46.045Z")
        },
        {
                "_id" : "sx470:27015:1403878642:1804289383",
                "ping" : ISODate("2014-06-27T14:21:54.769Z")
        },
        {
                "_id" : "sx470:27015:1403878933:1804289383",
                "ping" : ISODate("2014-06-27T14:22:13.171Z")
        },
        {
                "_id" : "sx470:27015:1403879230:1804289383",
                "ping" : ISODate("2014-06-27T14:29:41.239Z")
        },
        {
                "_id" : "s487:27017:1403879236:286455613",
                "ping" : ISODate("2014-06-27T14:30:48.304Z")
        },
        {
                "_id" : "sx176:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.049Z")
        },
        {
                "_id" : "sx177:27018:1403879261:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.083Z")
        },
        {
                "_id" : "sx210:27018:1403879262:1804289383",
                "ping" : ISODate("2014-06-27T14:30:43.151Z")
        },
        {
                "_id" : "sx470:27017:1403879283:1804289383",
                "ping" : ISODate("2014-06-27T14:30:35.137Z")
        },
        {
                "_id" : "sx470:27015:1403879507:1804289383",
                "ping" : ISODate("2014-06-27T17:37:10.712Z")
        },
        {
                "_id" : "s484:27017:1403879511:112806737",
                "ping" : ISODate("2014-06-27T17:37:14.604Z")
        },
        {
                "_id" : "sx177:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx176:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.067Z")
        },
        {
                "_id" : "sx210:27018:1403879528:1804289383",
                "ping" : ISODate("2014-06-27T17:37:03.443Z")
        },
        {
                "_id" : "sx470:27017:1403879536:1804289383",
                "ping" : ISODate("2014-06-27T17:37:09.994Z")
        },
        {
                "_id" : "s129:27017:1403882252:681112909",
                "ping" : ISODate("2014-06-27T17:37:04.399Z")
        },
        {
                "_id" : "s216:27017:1403882309:2082556053",
                "ping" : ISODate("2014-06-27T17:37:02.144Z")
        },
        {
                "_id" : "s136:27017:1403882339:414926390",
                "ping" : ISODate("2014-06-27T17:37:02.999Z")
        },
        {
                "_id" : "s479:27017:1403882398:1167576080",
                "ping" : ISODate("2014-06-27T17:37:01.223Z")
        }
]
 
mongos> 

Do you see there a stale lock? How can we remove it/them?
Thanks!

Comment by Thomas Rueckstiess [ 27/Jun/14 ]

Hi Kay,

Is this issue a single incident or does it happen repeatedly?

The metadata locks are stored in the config.locks collection. A state value of 2 means the lock is taken, 0 means the lock is open. There may still be a lock that's being held from a previous attempt.

Can you please share the content of your config.mongos and config.lockpings and config.locks collections from all three config servers individually? You can get these via a mongos, with the following script:

var config_servers = db.adminCommand("getCmdLineOpts").parsed.sharding.configDB.split(",");
config_servers.forEach(function (server) {
    var mongo = new Mongo(server);
    print(server);
    print("----------------------------");
    printjson(mongo.getDB("config").mongos.find().toArray());
    printjson(mongo.getDB("config").locks.find().toArray());
    printjson(mongo.getDB("config").lockpings.find().toArray());
    print();
})

This should tell us if there is a stuck migration or a stale lock.

Thanks,
Thomas

Generated at Thu Feb 08 03:34:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.