|
On PTO until October 13th
For URGENT DB issues please contact Alon Horev
On Sep 29, 2015, at 2:43 PM, Adam Schwartz (JIRA) <jira@mongodb.org> wrote:
[ https://jira.mongodb.org/browse/SERVER-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramon Fernandez updated SERVER-17243:
-------------------------------------
Comment: was deleted
(was: On PTO until October 13th
For URGENT DB issues please contact Alon Horev
On Sep 28, 2015, at 11:41 PM, Andy Schwerin (JIRA) <jira@mongodb.org> wrote:
[ https://jira.mongodb.org/browse/SERVER-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Schwerin updated SERVER-17243:
-----------------------------------
Fix Version/s: Planning Bucket A
(was: 3.1 Desired)
Dropped collection & database still has metadata
------------------------------------------------
Key: SERVER-17243
URL: https://jira.mongodb.org/browse/SERVER-17243
Project: Core Server
Issue Type: Bug
Components: Sharding
Affects Versions: 3.0.0-rc8
Reporter: Jonathan Abrahams
Labels: 28qa
Fix For: Planning Bucket A
Attachments: configdata.tar.gz, mongos-1.log.gz, mongos-2.log.gz
After dropping the collection for a sharded collection, via mongos, the chunks are still visible in the config server. Same if the database is dropped.
This collection had balancing failures, which seems to have left a lock in place, preventing full cleanup.
mongos> db.cap1833hashed.drop()
|
2015-02-10T20:37:33.450+0000 E QUERY Error: drop failed: {
|
"code" : 16338,
|
"ok" : 0,
|
"errmsg" : "exception: Dropping collection failed on the following hosts: shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423598324000|1, electionId: ObjectId('54d968cb8a8d61f9c59083ff') } }, shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423600492000|1, electionId: ObjectId('54d4c393625d1974fec18a28') } }"
|
}
|
at Error (<anonymous>)
|
at DBCollection.drop (src/mongo/shell/collection.js:619:15)
|
at (shell):1:18 at src/mongo/shell/collection.js:619
|
mongos> use config
|
switched to db config
|
mongos> db.collections.find()
|
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-06T17:55:44.651Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54d5002050c6400fb0af3055") }
|
{ "_id" : "t.foo", "lastmod" : ISODate("2015-02-10T20:35:07.767Z"), "dropped" : true, "lastmodEpoch" : ObjectId("000000000000000000000000") }
|
mongos> db.locks.find()
|
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:mongosMain:846930886", "ts" : ObjectId("54d4ffa250c6400fb0af3035"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-06T17:53:38.080Z"), "why" : "upgrading config database to new format v6" }
|
{ "_id" : "balancer", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:Balancer:1681692777", "ts" : ObjectId("54da507d50c6400fb0b0592c"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T18:39:57.034Z"), "why" : "doing balance round" }
|
{ "_id" : "test.cap1833hashed", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6c0e50c6400fb0b05945"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:37:34.650Z"), "why" : "drop" }
|
{ "_id" : "t.foo", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6b7b50c6400fb0b05942"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:35:07.203Z"), "why" : "drop" }
|
mongos> sh.status()
|
--- Sharding Status ---
|
sharding version: {
|
"_id" : 1,
|
"minCompatibleVersion" : 5,
|
"currentVersion" : 6,
|
"clusterId" : ObjectId("54d4ffa250c6400fb0af3037")
|
}
|
shards:
|
{ "_id" : "shard0", "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
|
{ "_id" : "shard1", "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
|
balancer:
|
Currently enabled: no
|
Currently running: no
|
Failed balancer rounds in last 5 attempts: 0
|
Migration Results for the last 24 hours:
|
19 : Success
|
1 : Failed with error 'data transfer error', from shard1 to shard0
|
3 : Failed with error 'migration already in progress', from shard1 to shard0
|
25 : Failed with error 'chunk too big to move', from shard1 to shard0
|
315 : Failed with error 'could not acquire collection lock for test.cap1833hashed to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed is taken.', from shard1 to shard0
|
databases:
|
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
|
{ "_id" : "test", "partitioned" : true, "primary" : "shard1" }
|
test.cap1833hashed
|
shard key: { "idx" : "hashed" }
|
chunks:
|
shard0 77978
|
shard1 93448
|
too many chunks to print, use verbose if you want to force print
|
{ "_id" : "t", "partitioned" : true, "primary" : "shard1" }
|
{ "_id" : "db", "partitioned" : false, "primary" : "shard1" }
|
----------------------
This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to https://jira.mongodb.org using your JIRA or MMS credentials.
)
Dropped collection & database still has metadata
------------------------------------------------
Key: SERVER-17243
URL: https://jira.mongodb.org/browse/SERVER-17243
Project: Core Server
Issue Type: Bug
Components: Sharding
Affects Versions: 3.0.0-rc8
Reporter: Jonathan Abrahams
Labels: 28qa
Fix For: Planning Bucket A
Attachments: configdata.tar.gz, mongos-1.log.gz, mongos-2.log.gz
After dropping the collection for a sharded collection, via mongos, the chunks are still visible in the config server. Same if the database is dropped.
This collection had balancing failures, which seems to have left a lock in place, preventing full cleanup.
mongos> db.cap1833hashed.drop()
|
2015-02-10T20:37:33.450+0000 E QUERY Error: drop failed: {
|
"code" : 16338,
|
"ok" : 0,
|
"errmsg" : "exception: Dropping collection failed on the following hosts: shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423598324000|1, electionId: ObjectId('54d968cb8a8d61f9c59083ff') } }, shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000: { ok: 0.0, errmsg: \"ns not found\", $gleStats: { lastOpTime: Timestamp 1423600492000|1, electionId: ObjectId('54d4c393625d1974fec18a28') } }"
|
}
|
at Error (<anonymous>)
|
at DBCollection.drop (src/mongo/shell/collection.js:619:15)
|
at (shell):1:18 at src/mongo/shell/collection.js:619
|
mongos> use config
|
switched to db config
|
mongos> db.collections.find()
|
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-06T17:55:44.651Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54d5002050c6400fb0af3055") }
|
{ "_id" : "t.foo", "lastmod" : ISODate("2015-02-10T20:35:07.767Z"), "dropped" : true, "lastmodEpoch" : ObjectId("000000000000000000000000") }
|
mongos> db.locks.find()
|
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:mongosMain:846930886", "ts" : ObjectId("54d4ffa250c6400fb0af3035"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-06T17:53:38.080Z"), "why" : "upgrading config database to new format v6" }
|
{ "_id" : "balancer", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:Balancer:1681692777", "ts" : ObjectId("54da507d50c6400fb0b0592c"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T18:39:57.034Z"), "why" : "doing balance round" }
|
{ "_id" : "test.cap1833hashed", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6c0e50c6400fb0b05945"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:37:34.650Z"), "why" : "drop" }
|
{ "_id" : "t.foo", "state" : 0, "who" : "ip-10-123-129-75:27017:1423245217:1804289383:conn23279:1957747793", "ts" : ObjectId("54da6b7b50c6400fb0b05942"), "process" : "ip-10-123-129-75:27017:1423245217:1804289383", "when" : ISODate("2015-02-10T20:35:07.203Z"), "why" : "drop" }
|
mongos> sh.status()
|
--- Sharding Status ---
|
sharding version: {
|
"_id" : 1,
|
"minCompatibleVersion" : 5,
|
"currentVersion" : 6,
|
"clusterId" : ObjectId("54d4ffa250c6400fb0af3037")
|
}
|
shards:
|
{ "_id" : "shard0", "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
|
{ "_id" : "shard1", "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
|
balancer:
|
Currently enabled: no
|
Currently running: no
|
Failed balancer rounds in last 5 attempts: 0
|
Migration Results for the last 24 hours:
|
19 : Success
|
1 : Failed with error 'data transfer error', from shard1 to shard0
|
3 : Failed with error 'migration already in progress', from shard1 to shard0
|
25 : Failed with error 'chunk too big to move', from shard1 to shard0
|
315 : Failed with error 'could not acquire collection lock for test.cap1833hashed to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed is taken.', from shard1 to shard0
|
databases:
|
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
|
{ "_id" : "test", "partitioned" : true, "primary" : "shard1" }
|
test.cap1833hashed
|
shard key: { "idx" : "hashed" }
|
chunks:
|
shard0 77978
|
shard1 93448
|
too many chunks to print, use verbose if you want to force print
|
{ "_id" : "t", "partitioned" : true, "primary" : "shard1" }
|
{ "_id" : "db", "partitioned" : false, "primary" : "shard1" }
|
----------------------
This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to https://jira.mongodb.org using your JIRA or MMS credentials.
|
|
After running into SERVER-17393, I tried to drop the collection and then the database:
mongos> db.cap1833hashed.drop()
|
2015-02-26T21:43:51.461+0000 E QUERY Error: drop failed: {
|
"code" : 13331,
|
"ok" : 0,
|
"errmsg" : "exception: collection's metadata is undergoing changes. Please try again."
|
}
|
at Error (<anonymous>)
|
at DBCollection.drop (src/mongo/shell/collection.js:619:15)
|
at (shell):1:18 at src/mongo/shell/collection.js:619
|
mongos> sh.status()
|
--- Sharding Status ---
|
sharding version: {
|
"_id" : 1,
|
"minCompatibleVersion" : 5,
|
"currentVersion" : 6,
|
"clusterId" : ObjectId("54edd63deae8d68b1c02b2bf")
|
}
|
shards:
|
{ "_id" : "shard0", "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
|
{ "_id" : "shard1", "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
|
balancer:
|
Currently enabled: yes
|
Currently running: yes
|
Balancer lock taken at Wed Feb 25 2015 19:30:06 GMT+0000 (UTC) by ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777
|
Collections with active migrations:
|
test.cap1833hashed started at Wed Feb 25 2015 19:30:07 GMT+0000 (UTC)
|
Failed balancer rounds in last 5 attempts: 0
|
Migration Results for the last 24 hours:
|
No recent migrations
|
databases:
|
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
|
{ "_id" : "test", "partitioned" : true, "primary" : "shard0" }
|
test.cap1833hashed
|
shard key: { "idx" : "hashed" }
|
chunks:
|
shard0 57324
|
shard1 64971
|
too many chunks to print, use verbose if you want to force print
|
|
mongos> db.dropDatabase()
|
{
|
"code" : 13331,
|
"ok" : 0,
|
"errmsg" : "exception: collection's metadata is undergoing changes. Please try again."
|
}
|
mongos> sh.status()
|
--- Sharding Status ---
|
sharding version: {
|
"_id" : 1,
|
"minCompatibleVersion" : 5,
|
"currentVersion" : 6,
|
"clusterId" : ObjectId("54edd63deae8d68b1c02b2bf")
|
}
|
shards:
|
{ "_id" : "shard0", "host" : "shard0/ip-10-102-151-133:30000,ip-10-203-175-233:30000,ip-10-63-43-228:30000" }
|
{ "_id" : "shard1", "host" : "shard1/ip-10-180-37-181:30000,ip-10-187-63-140:30000,ip-10-203-173-62:30000" }
|
balancer:
|
Currently enabled: yes
|
Currently running: yes
|
Balancer lock taken at Wed Feb 25 2015 19:30:06 GMT+0000 (UTC) by ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777
|
Collections with active migrations:
|
test.cap1833hashed started at Wed Feb 25 2015 19:30:07 GMT+0000 (UTC)
|
Failed balancer rounds in last 5 attempts: 0
|
Migration Results for the last 24 hours:
|
No recent migrations
|
databases:
|
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
|
mongos> use config
|
switched to db config
|
mongos> db.chunks.count()
|
122295
|
mongos> db.collections.find()
|
{ "_id" : "test.cap1833hashed", "lastmod" : ISODate("2015-02-25T14:05:22.027Z"), "dropped" : false, "key" : { "idx" : "hashed" }, "unique" : false, "lastmodEpoch" : ObjectId("54edd6a2eae8d68b1c02b2c5") }
|
mongos> db.locks.find()
|
{ "_id" : "balancer", "state" : 2, "who" : "ip-10-123-129-75:37017:1424873021:1804289383:Balancer:1681692777", "ts" : ObjectId("54ee22beeae8d68b1c02bfaa"), "process" : "ip-10-123-129-75:37017:1424873021:1804289383", "when" : ISODate("2015-02-25T19:30:06.046Z"), "why" : "doing balance round" }
|
{ "_id" : "configUpgrade", "state" : 0, "who" : "ip-10-123-129-75:37017:1424873021:1804289383:mongosMain:846930886", "ts" : ObjectId("54edd63deae8d68b1c02b2bd"), "process" : "ip-10-123-129-75:37017:1424873021:1804289383", "when" : ISODate("2015-02-25T14:03:41.332Z"), "why" : "upgrading config database to new format v6" }
|
{ "_id" : "test.cap1833hashed", "state" : 2, "who" : "ip-10-203-173-62:30000:1424877009:481270752:conn61:269687465", "ts" : ObjectId("54ee22bfcf8fe88c92d09625"), "process" : "ip-10-203-173-62:30000:1424877009:481270752", "when" : ISODate("2015-02-25T19:30:07.013Z"), "why" : "migrating chunk [{ : MinKey }, { : MaxKey }) in test.cap1833hashed" }
|
It seemed this is caused SERVER-17398.
|