Details
Description
MongoS will continue to issue chunk split commands on insert even when config servers are inconsistent.
These splits will appear as failed in the log and no entry is found in the changelog, however the chunks collection is updated.
Logs from the MongoS. Note the chunk ranges.
Tue Dec 3 14:41:23.346 [conn5] about to initiate autosplit: ns:test.testshard: shard0000:Pixl.local:30000lastmod: 2|985||000000000000000000000000min: { _id: -7767558900086237716 }max: { _id: -7177852217467415019 } dataWritten: 209766 splitThreshold: 1048576
|
Tue Dec 3 14:41:23.476 [Balancer] distributed lock 'balancer/Pixl.local:30005:1386041035:16807' acquired, ts : 529d52e3a397e550c993feab
|
Tue Dec 3 14:41:23.476 [Balancer] warning: Skipping balancing round because data inconsistency was detected amongst the config servers.
|
Tue Dec 3 14:41:23.929 [conn5] warning: splitChunk failed - cmd: { splitChunk: "test.test", keyPattern: { _id: "hashed" }, min: { _id: -7767558900086237716 }, max: { _id: -7177852217467415019 }, from: "shard0000", splitKeys: [ { _id: -7751497167210968041 } ], shardId: "test.test-_id_-7767558900086237716", configdb: "Pixl.local:30002,Pixl.local:30003,Pixl.local:30004" } result: { errmsg: "exception: write $cmd failed on a node: { "got" : { "_id" : "test.test-_id_0", "lastmod" : { "$timestamp" : { "t" : 2, "i" : 4 } }, "lastmodEpoch" : {...", code: 13105, ok: 0.0 }
|
Find of the left half of the split chunk:
>db.getSiblingDB("config").chunks.find({"min._id":-7767558900086237716});
|
{ "_id" : "test.test-_id_-7767558900086237716", "lastmod" : Timestamp(2, 1006), "lastmodEpoch" : ObjectId("529d4efea397e550c993f69e"), "ns" : "test.test", "min" : { "_id" : NumberLong("-7767558900086237716") }, "max" : { "_id" : NumberLong("-7751497167210968041") }, "shard" : "shard0000" }
|
Find of the right half of the split chunk:
db.getSiblingDB("config").chunks.find({"min._id":-7751497167210968041});
|
{ "_id" : "test.test-_id_-7751497167210968041", "lastmod" : Timestamp(2, 1007), "lastmodEpoch" : ObjectId("529d4efea397e550c993f69e"), "ns" : "test.test", "min" : { "_id" : NumberLong("-7751497167210968041") }, "max" : { "_id" : NumberLong("-7177852217467415019") }, "shard" : "shard0000" }
|
MongoD log for splitchunk:
Tue Dec 3 14:41:23.346 [conn5] request split points lookup for chunk test.test { : -7767558900086237716 } -->> { : -7177852217467415019 }
|
Tue Dec 3 14:41:23.348 [conn5] max number of requested split points reached (2) before the end of chunk test.test { : -7767558900086237716 } -->> { : -7177852217467415019 }
|
Tue Dec 3 14:41:23.348 [conn5] received splitChunk request: { splitChunk: "test.test", keyPattern: { _id: "hashed" }, min: { _id: -7767558900086237716 }, max: { _id: -7177852217467415019 }, from: "shard0000", splitKeys: [ { _id: -7751497167210968041 } ], shardId: "test.test-_id_-7767558900086237716", configdb: "Pixl.local:30002,Pixl.local:30003,Pixl.local:30004" }
|
Tue Dec 3 14:41:23.649 [conn5] distributed lock 'test.test/Pixl.local:30000:1386041087:1349921075' acquired, ts : 529d52e302b120a57cca2e2b
|
Tue Dec 3 14:41:23.649 [conn5] SyncClusterConnection connecting to [Pixl.local:30002]
|
Tue Dec 3 14:41:23.650 [conn5] SyncClusterConnection connecting to [Pixl.local:30003]
|
Tue Dec 3 14:41:23.650 [conn5] SyncClusterConnection connecting to [Pixl.local:30004]
|
Tue Dec 3 14:41:23.652 [conn5] splitChunk accepted at version 2|1005||529d4efea397e550c993f69e
|
Tue Dec 3 14:41:23.789 [conn5] scoped connection to Pixl.local:30002,Pixl.local:30003,Pixl.local:30004 not being returned to the pool
|
Tue Dec 3 14:41:23.928 [conn5] distributed lock 'test.test/Pixl.local:30000:1386041087:1349921075' unlocked.
|
Tue Dec 3 14:41:23.928 [conn5] command admin.$cmd command: { splitChunk: "test.test", keyPattern: { _id: "hashed" }, min: { _id: -7767558900086237716 }, max: { _id: -7177852217467415019 }, from: "shard0000", splitKeys: [ { _id: -7751497167210968041 } ], shardId: "test.test-_id_-7767558900086237716", configdb: "Pixl.local:30002,Pixl.local:30003,Pixl.local:30004" } ntoreturn:1 keyUpdates:0 locks(micros) r:2 reslen:1459 579ms
|
Attachments
Issue Links
- is related to
-
SERVER-10015 balancer should stop when ConfigServerCheck indicates inconsistency
-
- Closed
-