[SERVER-17243] Dropped collection & database still has metadata Created: 10/Feb/15  Updated: 06/Dec/22  Resolved: 07/Mar/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.0.0-rc8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Duplicate Votes: 3
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File configdata.tar.gz     File mongos-1.log.gz     File mongos-2.log.gz    
Issue Links:
Depends
Duplicate
duplicates SERVER-33973 Force cleanup of possibly remaining p... Closed
Related
is related to SERVER-17398 Deadlock in MigrateStatus::startCommit Closed
Assigned Teams:
Sharding
Operating System: ALL
Participants:
Case:

 Description   

After dropping the collection for a sharded collection, via mongos, the chunks are still visible in the config server. Same if the database is dropped.

This collection had balancing failures, which seems to have left a lock in place, preventing full cleanup.

mongos> db.cap1833hashed.drop()
2015-02-10T20:37:33.450+0000 E QUERY    Error: drop failed: {
	"code" : 16338,
	"ok" : 0,
	"errmsg" : "exception: Dropping collection failed on the following hosts: shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423598324000|1, electionId: ObjectId('54d968cb8a8d61f9c59083ff') } }, shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423600492000|1, electionId: ObjectId('54d4c393625d1974fec18a28') } }"
}
    at Error (<anonymous>)
    at DBCollection.drop (src/mongo/shell/collection.js:619:15)
    at (shell):1:18 at src/mongo/shell/collection.js:619
mongos> use config
switched to db config
mongos> db.collections.find()
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-06T17:55:44.651Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54d5002050c6400fb0af3055") }
{ "_id" : "t.foo", "lastmod" : ISODate("2015-02-10T20:35:07.767Z"), "dropped" : true, "lastmodEpoch" : ObjectId("000000000000000000000000") }
mongos> db.locks.find()
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:mongosMain:846930886", "ts" : ObjectId("54d4ffa250c6400fb0af3035"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-06T17:53:38.080Z"), "why" : "upgrading config database to new format v6" }
{ "_id" : "balancer", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:Balancer:1681692777", "ts" : ObjectId("54da507d50c6400fb0b0592c"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T18:39:57.034Z"), "why" : "doing balance round" }
{ "_id" : "test.cap1833hashed", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6c0e50c6400fb0b05945"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:37:34.650Z"), "why" : "drop" }
{ "_id" : "t.foo", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6b7b50c6400fb0b05942"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:35:07.203Z"), "why" : "drop" }
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("54d4ffa250c6400fb0af3037")
}
  shards:
	{  "_id" : "shard0",  "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
	{  "_id" : "shard1",  "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
  balancer:
	Currently enabled:  no
	Currently running:  no
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours:
		19 : Success
		1 : Failed with error 'data transfer error', from shard1 to shard0
		3 : Failed with error 'migration already in progress', from shard1 to shard0
		25 : Failed with error 'chunk too big to move', from shard1 to shard0
		315 : Failed with error 'could not acquire collection lock for test.cap1833hashed to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed is taken.', from shard1 to shard0
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }
		test.cap1833hashed
			shard key: { "idx" : "hashed" }
			chunks:
				shard0	77978
				shard1	93448
			too many chunks to print, use verbose if you want to force print
	{  "_id" : "t",  "partitioned" : true,  "primary" : "shard1" }
	{  "_id" : "db",  "partitioned" : false,  "primary" : "shard1" }



 Comments   
Comment by David Bord [ 01/Oct/15 ]

On PTO until October 13th
For URGENT DB issues please contact Alon Horev

On Sep 29, 2015, at 2:43 PM, Adam Schwartz (JIRA) <jira@mongodb.org> wrote:

[ https://jira.mongodb.org/browse/SERVER-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramon Fernandez updated SERVER-17243:
-------------------------------------
Comment: was deleted

(was: On PTO until October 13th
For URGENT DB issues please contact Alon Horev

On Sep 28, 2015, at 11:41 PM, Andy Schwerin (JIRA) <jira@mongodb.org> wrote:

[ https://jira.mongodb.org/browse/SERVER-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andy Schwerin updated SERVER-17243:
-----------------------------------
Fix Version/s: Planning Bucket A
(was: 3.1 Desired)

Dropped collection & database still has metadata
------------------------------------------------

Key: SERVER-17243
URL: https://jira.mongodb.org/browse/SERVER-17243
Project: Core Server
Issue Type: Bug
Components: Sharding
Affects Versions: 3.0.0-rc8
Reporter: Jonathan Abrahams
Labels: 28qa
Fix For: Planning Bucket A

Attachments: configdata.tar.gz, mongos-1.log.gz, mongos-2.log.gz

After dropping the collection for a sharded collection, via mongos, the chunks are still visible in the config server. Same if the database is dropped.
This collection had balancing failures, which seems to have left a lock in place, preventing full cleanup.

mongos> db.cap1833hashed.drop()
2015-02-10T20:37:33.450+0000 E QUERY    Error: drop failed: {
       "code" : 16338,
       "ok" : 0,
       "errmsg" : "exception: Dropping collection failed on the following hosts: shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423598324000|1, electionId: ObjectId('54d968cb8a8d61f9c59083ff') } }, shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423600492000|1, electionId: ObjectId('54d4c393625d1974fec18a28') } }"
}
  at Error (<anonymous>)
  at DBCollection.drop (src/mongo/shell/collection.js:619:15)
  at (shell):1:18 at src/mongo/shell/collection.js:619
mongos> use config
switched to db config
mongos> db.collections.find()
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-06T17:55:44.651Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54d5002050c6400fb0af3055") }
{ "_id" : "t.foo", "lastmod" : ISODate("2015-02-10T20:35:07.767Z"), "dropped" : true, "lastmodEpoch" : ObjectId("000000000000000000000000") }
mongos> db.locks.find()
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:mongosMain:846930886", "ts" : ObjectId("54d4ffa250c6400fb0af3035"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-06T17:53:38.080Z"), "why" : "upgrading config database to new format v6" }
{ "_id" : "balancer", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:Balancer:1681692777", "ts" : ObjectId("54da507d50c6400fb0b0592c"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T18:39:57.034Z"), "why" : "doing balance round" }
{ "_id" : "test.cap1833hashed", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6c0e50c6400fb0b05945"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:37:34.650Z"), "why" : "drop" }
{ "_id" : "t.foo", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6b7b50c6400fb0b05942"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:35:07.203Z"), "why" : "drop" }
mongos> sh.status()
--- Sharding Status ---
sharding version: {
       "_id" : 1,
       "minCompatibleVersion" : 5,
       "currentVersion" : 6,
       "clusterId" : ObjectId("54d4ffa250c6400fb0af3037")
}
shards:
       {  "_id" : "shard0",  "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
       {  "_id" : "shard1",  "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
balancer:
       Currently enabled:  no
       Currently running:  no
       Failed balancer rounds in last 5 attempts:  0
       Migration Results for the last 24 hours:
               19 : Success
               1 : Failed with error 'data transfer error', from shard1 to shard0
               3 : Failed with error 'migration already in progress', from shard1 to shard0
               25 : Failed with error 'chunk too big to move', from shard1 to shard0
               315 : Failed with error 'could not acquire collection lock for test.cap1833hashed to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed is taken.', from shard1 to shard0
databases:
       {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
       {  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }
               test.cap1833hashed
                       shard key: { "idx" : "hashed" }
                       chunks:
                               shard0  77978
                               shard1  93448
                       too many chunks to print, use verbose if you want to force print
       {  "_id" : "t",  "partitioned" : true,  "primary" : "shard1" }
       {  "_id" : "db",  "partitioned" : false,  "primary" : "shard1" }

----------------------
This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to https://jira.mongodb.org using your JIRA or MMS credentials.
)

Dropped collection & database still has metadata
------------------------------------------------

Key: SERVER-17243
URL: https://jira.mongodb.org/browse/SERVER-17243
Project: Core Server
Issue Type: Bug
Components: Sharding
Affects Versions: 3.0.0-rc8
Reporter: Jonathan Abrahams
Labels: 28qa
Fix For: Planning Bucket A

Attachments: configdata.tar.gz, mongos-1.log.gz, mongos-2.log.gz

After dropping the collection for a sharded collection, via mongos, the chunks are still visible in the config server. Same if the database is dropped.
This collection had balancing failures, which seems to have left a lock in place, preventing full cleanup.

mongos> db.cap1833hashed.drop()
2015-02-10T20:37:33.450+0000 E QUERY    Error: drop failed: {
        "code" : 16338,
        "ok" : 0,
        "errmsg" : "exception: Dropping collection failed on the following hosts: shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423598324000|1, electionId: ObjectId('54d968cb8a8d61f9c59083ff') } }, shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423600492000|1, electionId: ObjectId('54d4c393625d1974fec18a28') } }"
}
   at Error (<anonymous>)
   at DBCollection.drop (src/mongo/shell/collection.js:619:15)
   at (shell):1:18 at src/mongo/shell/collection.js:619
mongos> use config
switched to db config
mongos> db.collections.find()
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-06T17:55:44.651Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54d5002050c6400fb0af3055") }
{ "_id" : "t.foo", "lastmod" : ISODate("2015-02-10T20:35:07.767Z"), "dropped" : true, "lastmodEpoch" : ObjectId("000000000000000000000000") }
mongos> db.locks.find()
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:mongosMain:846930886", "ts" : ObjectId("54d4ffa250c6400fb0af3035"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-06T17:53:38.080Z"), "why" : "upgrading config database to new format v6" }
{ "_id" : "balancer", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:Balancer:1681692777", "ts" : ObjectId("54da507d50c6400fb0b0592c"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T18:39:57.034Z"), "why" : "doing balance round" }
{ "_id" : "test.cap1833hashed", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6c0e50c6400fb0b05945"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:37:34.650Z"), "why" : "drop" }
{ "_id" : "t.foo", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6b7b50c6400fb0b05942"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:35:07.203Z"), "why" : "drop" }
mongos> sh.status()
--- Sharding Status ---
 sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("54d4ffa250c6400fb0af3037")
}
 shards:
        {  "_id" : "shard0",  "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
        {  "_id" : "shard1",  "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
 balancer:
        Currently enabled:  no
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                19 : Success
                1 : Failed with error 'data transfer error', from shard1 to shard0
                3 : Failed with error 'migration already in progress', from shard1 to shard0
                25 : Failed with error 'chunk too big to move', from shard1 to shard0
                315 : Failed with error 'could not acquire collection lock for test.cap1833hashed to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed is taken.', from shard1 to shard0
 databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }
                test.cap1833hashed
                        shard key: { "idx" : "hashed" }
                        chunks:
                                shard0  77978
                                shard1  93448
                        too many chunks to print, use verbose if you want to force print
        {  "_id" : "t",  "partitioned" : true,  "primary" : "shard1" }
        {  "_id" : "db",  "partitioned" : false,  "primary" : "shard1" }

----------------------
This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to https://jira.mongodb.org using your JIRA or MMS credentials.

Comment by Randolph Tan [ 27/Feb/15 ]

I think the new repro steps is slightly different from the original. In the original report, the shards doesn't have the collection anymore (the drop was actually executed, in fact, it was in the oplog). However, the drop command somehow failed before it changed the config metadata. In the new repro, it failed even before it starts sending drops the shards.

Comment by Jonathan Abrahams [ 26/Feb/15 ]

After running into SERVER-17393, I tried to drop the collection and then the database:

mongos> db.cap1833hashed.drop()
2015-02-26T21:43:51.461+0000 E QUERY    Error: drop failed: {
	"code" : 13331,
	"ok" : 0,
	"errmsg" : "exception: collection's metadata is undergoing changes. Please try again."
}
    at Error (<anonymous>)
    at DBCollection.drop (src/mongo/shell/collection.js:619:15)
    at (shell):1:18 at src/mongo/shell/collection.js:619
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("54edd63deae8d68b1c02b2bf")
}
  shards:
	{  "_id" : "shard0",  "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
	{  "_id" : "shard1",  "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
  balancer:
	Currently enabled:  yes
	Currently running:  yes
		Balancer lock taken at Wed Feb 25 2015 19:30:06 GMT+0000 (UTC) by ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777
	Collections with active migrations:
		test.cap1833hashed started at Wed Feb 25 2015 19:30:07 GMT+0000 (UTC)
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours:
		No recent migrations
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0" }
		test.cap1833hashed
			shard key: { "idx" : "hashed" }
			chunks:
				shard0	57324
				shard1	64971
			too many chunks to print, use verbose if you want to force print
 
mongos> db.dropDatabase()
{
	"code" : 13331,
	"ok" : 0,
	"errmsg" : "exception: collection's metadata is undergoing changes. Please try again."
}
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("54edd63deae8d68b1c02b2bf")
}
  shards:
	{  "_id" : "shard0",  "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
	{  "_id" : "shard1",  "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
  balancer:
	Currently enabled:  yes
	Currently running:  yes
		Balancer lock taken at Wed Feb 25 2015 19:30:06 GMT+0000 (UTC) by ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777
	Collections with active migrations:
		test.cap1833hashed started at Wed Feb 25 2015 19:30:07 GMT+0000 (UTC)
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours:
		No recent migrations
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
mongos> use config
switched to db config
mongos> db.chunks.count()
122295
mongos> db.collections.find()
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-25T14:05:22.027Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54edd6a2eae8d68b1c02b2c5") }
mongos> db.locks.find()
{ "_id" : "balancer", "state" : 2, "who" : "ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777", "ts" : ObjectId("54ee22beeae8d68b1c02bfaa"), "process" : "ip-10-123-129-75:37017:1424873021:1804289383", "when" : ISODate("2015-02-25T19:30:06.046Z"), "why" : "doing balance round" }
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:37017:1424873021:1804289383:mongosMain:846930886", "ts" : ObjectId("54edd63deae8d68b1c02b2bd"), "process" : "ip-10-123-129-75:37017:1424873021:1804289383", "when" : ISODate("2015-02-25T14:03:41.332Z"), "why" : "upgrading config database to new format v6" }
{ "_id" : "test.cap1833hashed", "state" : 2, "who" : "ip-10-203-173-62:30000:1424877009:481270752:conn61:269687465", "ts" : ObjectId("54ee22bfcf8fe88c92d09625"), "process" : "ip-10-203-173-62:30000:1424877009:481270752", "when" : ISODate("2015-02-25T19:30:07.013Z"), "why" : "migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed" }

It seemed this is caused SERVER-17398.

Comment by Randolph Tan [ 23/Feb/15 ]

It looks like the dropDatabase/dropCollection command cannot be resumed without some manual intervention. If the command failed on one of the shards and was thus unable to successfully complete, successive calls to the drop command will fail because the previous run has already removed the collection metadata in some of the shards (there is a check at the beginning of the command that the ns to drop still exists).

jonathan.abrahams I would like to find out what the initial failure was when you reproduced this and when you try to reproduce this again, I would like to request for:

1. Log level 1 for mongos and the primaries.
2. Always log the return object from the drop commands in your application so we can see the error details of the first failure.

Comment by Jonathan Abrahams [ 11/Feb/15 ]

I still have the environment active....we can look at it tomorrow.

Comment by Randolph Tan [ 11/Feb/15 ]

jonathan.abrahams Do you still have the dump of the shards? It looks like there are no database/collection metadata (note: I'm referring to the storage engine metadata, not sharding metadata) in the shard0 (possibly no data?), which made the drop fail. The reason why the collection and database metadata still exists in the config servers is because the drop was not executed successfully.

Comment by Andy Schwerin [ 11/Feb/15 ]

renctan, can you do an analysis of this error?

Comment by Jonathan Abrahams [ 11/Feb/15 ]

The scenario could not be reproduced in the simple way, i.e., create collection, shard collection, add splits, drop collection /database. Perhaps adding lock failures during migration would help.

This problem was observed from a long running mongos stress test, which ran as follows:

  • 2 node sharded clustert, with shard nodes running WiredTiger storage engine
  • Set shard chunk size to 2MB
  • Create collection
  • Created hashed index
  • Shard collection on hashed index
  • Populate collection with 1000000 documents, with average document size of 10K
  • 5 clients perform the following workload:
    • 50% find on random value for hashed index
    • 50% bulk insert of 10000 documents, averaging 10K size
Comment by Scott Hernandez (Inactive) [ 11/Feb/15 ]

Can you please attach a dump of the config database, the mongos logs, and a set of reproduction steps?

Generated at Thu Feb 08 03:43:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.