[SERVER-22301] Sh.shardCollection() for hashed shard fails to report error when move chunk fails. Created: 25/Jan/16  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: william hutton Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

1) Create file to create and hash shard 20 “xxx” collections

use hstest
sh.enableSharding("hstest")
db.createCollection("xxx10")
sh.shardCollection("hstest.xxx10",{ENTITY_ID:"hashed"})
db.createCollection("xxx11")
sh.shardCollection("hstest.xxx11",{ENTITY_ID:"hashed"})
db.createCollection("xxx12")
sh.shardCollection("hstest.xxx12",{ENTITY_ID:"hashed"})
db.createCollection("xxx13")
sh.shardCollection("hstest.xxx13",{ENTITY_ID:"hashed"})
db.createCollection("xxx14")
sh.shardCollection("hstest.xxx14",{ENTITY_ID:"hashed"})
db.createCollection("xxx15")
sh.shardCollection("hstest.xxx15",{ENTITY_ID:"hashed"})
db.createCollection("xxx16")
sh.shardCollection("hstest.xxx16",{ENTITY_ID:"hashed"})
db.createCollection("xxx17")
sh.shardCollection("hstest.xxx17",{ENTITY_ID:"hashed"})
db.createCollection("xxx18")
sh.shardCollection("hstest.xxx18",{ENTITY_ID:"hashed"})
db.createCollection("xxx19")
sh.shardCollection("hstest.xxx19",{ENTITY_ID:"hashed"})
db.createCollection("xxx20")
sh.shardCollection("hstest.xxx20",{ENTITY_ID:"hashed"})
db.createCollection("xxx21")
sh.shardCollection("hstest.xxx21",{ENTITY_ID:"hashed"})
db.createCollection("xxx22")
sh.shardCollection("hstest.xxx22",{ENTITY_ID:"hashed"})
db.createCollection("xxx23")
sh.shardCollection("hstest.xxx23",{ENTITY_ID:"hashed"})
db.createCollection("xxx24")
sh.shardCollection("hstest.xxx24",{ENTITY_ID:"hashed"})
db.createCollection("xxx25")
sh.shardCollection("hstest.xxx25",{ENTITY_ID:"hashed"})
db.createCollection("xxx26")
sh.shardCollection("hstest.xxx26",{ENTITY_ID:"hashed"})
db.createCollection("xxx27")
sh.shardCollection("hstest.xxx27",{ENTITY_ID:"hashed"})
db.createCollection("xxx28")
sh.shardCollection("hstest.xxx28",{ENTITY_ID:"hashed"})
db.createCollection("xxx29")
sh.shardCollection("hstest.xxx29",{ENTITY_ID:"hashed"})

2) Create file to create and hash shard 20 “yyy” collections

use hstest
sh.enableSharding("hstest")
db.createCollection("xxx10")
sh.shardCollection("hstest.xxx10",{ENTITY_ID:"hashed"})
db.createCollection("xxx11")
sh.shardCollection("hstest.xxx11",{ENTITY_ID:"hashed"})
db.createCollection("xxx12")
sh.shardCollection("hstest.xxx12",{ENTITY_ID:"hashed"})
db.createCollection("xxx13")
sh.shardCollection("hstest.xxx13",{ENTITY_ID:"hashed"})
db.createCollection("xxx14")
sh.shardCollection("hstest.xxx14",{ENTITY_ID:"hashed"})
db.createCollection("xxx15")
sh.shardCollection("hstest.xxx15",{ENTITY_ID:"hashed"})
db.createCollection("xxx16")
sh.shardCollection("hstest.xxx16",{ENTITY_ID:"hashed"})
db.createCollection("xxx17")
sh.shardCollection("hstest.xxx17",{ENTITY_ID:"hashed"})
db.createCollection("xxx18")
sh.shardCollection("hstest.xxx18",{ENTITY_ID:"hashed"})
db.createCollection("xxx19")
sh.shardCollection("hstest.xxx19",{ENTITY_ID:"hashed"})
db.createCollection("xxx20")
sh.shardCollection("hstest.xxx20",{ENTITY_ID:"hashed"})
db.createCollection("xxx21")
sh.shardCollection("hstest.xxx21",{ENTITY_ID:"hashed"})
db.createCollection("xxx22")
sh.shardCollection("hstest.xxx22",{ENTITY_ID:"hashed"})
db.createCollection("xxx23")
sh.shardCollection("hstest.xxx23",{ENTITY_ID:"hashed"})
db.createCollection("xxx24")
sh.shardCollection("hstest.xxx24",{ENTITY_ID:"hashed"})
db.createCollection("xxx25")
sh.shardCollection("hstest.xxx25",{ENTITY_ID:"hashed"})
db.createCollection("xxx26")
sh.shardCollection("hstest.xxx26",{ENTITY_ID:"hashed"})
db.createCollection("xxx27")
sh.shardCollection("hstest.xxx27",{ENTITY_ID:"hashed"})
db.createCollection("xxx28")
sh.shardCollection("hstest.xxx28",{ENTITY_ID:"hashed"})
db.createCollection("xxx29")
sh.shardCollection("hstest.xxx29",{ENTITY_ID:"hashed"})

3) execute both files in mongo at the same time (two shells)
note : balancer is disabled

cat x | mongo
MongoDB shell version: 3.2.0
connecting to: test
switched to db hstest
{ "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz10", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz11", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz12", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz13", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz14", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz15", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz16", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz17", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz18", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz19", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz20", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz21", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz22", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz23", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz24", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz25", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz26", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz27", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz28", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.zzz29", "ok" : 1 }
Bye
 
cat y | mongo
MongoDB shell version: 3.2.0
connecting to: test
switched to db hstest
{
        "ok" : 0,
        "errmsg" : "sharding already enabled for database hstest",
        "code" : 23
}
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx10", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx11", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx12", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx13", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx14", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx15", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx16", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx17", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx18", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx19", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx20", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx21", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx22", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx23", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx24", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx25", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx26", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx27", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx28", "ok" : 1 }
{ "ok" : 1 }
{ "collectionsharded" : "hstest.xxx29", "ok" : 1 }
Bye

4) Shard status show hash split range correct for all collections, but distribution among the 3 shards is inconsistent.
Expected 2-2-2 distribution is often not present.
Failed with error 'aborted' balancer errors indicate moveChunk() failing for many hash ranges
Enabling balancer redistributes all collections to 2-2-2 distribution.

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("568bf1758584c5c1635a88e0")
}
  shards:
        {  "_id" : "shard0000",  "host" : "weedat02.idco.intdata.com:27018",  "tags" : [ "entity_range_0", "entity_range_3", "entity_range_6", "entity_range_9", "entity_range_12", "entity_range_15", "entity_range_18", "entity_range_21", "entity_range_24", "entity_range_27", "entity_range_30", "entity_range_33", "entity_range_36", "entity_range_39", "entity_range_42", "entity_range_45", "entity_range_48" ] }
        {  "_id" : "shard0001",  "host" : "weedat03.idco.intdata.com:27018",  "tags" : [ "entity_range_1", "entity_range_4", "entity_range_7", "entity_range_10", "entity_range_13", "entity_range_16", "entity_range_19", "entity_range_22", "entity_range_25", "entity_range_28", "entity_range_31", "entity_range_34", "entity_range_37", "entity_range_40", "entity_range_43", "entity_range_46", "entity_range_49" ] }
        {  "_id" : "shard0002",  "host" : "weedat04.idco.intdata.com:27018",  "tags" : [ "entity_range_2", "entity_range_5", "entity_range_8", "entity_range_11", "entity_range_14", "entity_range_17", "entity_range_20", "entity_range_23", "entity_range_26", "entity_range_29", "entity_range_32", "entity_range_35", "entity_range_38", "entity_range_41", "entity_range_44", "entity_range_47" ] }
  active mongoses:
        "3.2.0" : 3
  balancer:
        Currently enabled:  no
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                222 : Success
                78 : Failed with error 'aborted', from shard0001 to shard0002
                76 : Failed with error 'aborted', from shard0001 to shard0000
                19 : Failed with error 'aborted', from shard0002 to shard0001
                24 : Failed with error 'aborted', from shard0002 to shard0000
  databases:
        {  "_id" : "pocdb",  "primary" : "shard0000",  "partitioned" : true }
        {  "_id" : "OMEGA",  "primary" : "shard0001",  "partitioned" : true }
        {  "_id" : "hstest",  "primary" : "shard0001",  "partitioned" : true }
                hstest.xxx10
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx11
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx12
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       4
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(2, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(2, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(2, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(2, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(2, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(2, 7)
                hstest.xxx13
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx14
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx15
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx16
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx17
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx18
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx19
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx20
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx21
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       4
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(2, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(2, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(2, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(2, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(2, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 7)
                hstest.xxx22
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx23
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx24
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx25
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx26
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.xxx27
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx28
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.xxx29
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       4
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(2, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(2, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(2, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(2, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(2, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 7)
                hstest.zzz10
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz11
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz12
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz13
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz14
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz15
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz16
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz17
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       4
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(2, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(2, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(2, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(2, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(2, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 7)
                hstest.zzz18
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz19
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       4
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(2, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(2, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(2, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(2, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(2, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 7)
                hstest.zzz20
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz21
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz22
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz23
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz24
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz25
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz26
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0001       6
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0001 Timestamp(1, 3)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0001 Timestamp(1, 4)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(1, 5)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(1, 6)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0001 Timestamp(1, 7)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 8)
                hstest.zzz27
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz28
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)
                hstest.zzz29
                        shard key: { "ENTITY_ID" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       2
                                shard0001       2
                                shard0002       2
                        { "ENTITY_ID" : { "$minKey" : 1 } } -->> { "ENTITY_ID" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2)
                        { "ENTITY_ID" : NumberLong("-6148914691236517204") } -->> { "ENTITY_ID" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3)
                        { "ENTITY_ID" : NumberLong("-3074457345618258602") } -->> { "ENTITY_ID" : NumberLong(0) } on : shard0001 Timestamp(3, 4)
                        { "ENTITY_ID" : NumberLong(0) } -->> { "ENTITY_ID" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5)
                        { "ENTITY_ID" : NumberLong("3074457345618258602") } -->> { "ENTITY_ID" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6)
                        { "ENTITY_ID" : NumberLong("6148914691236517204") } -->> { "ENTITY_ID" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7)

5 ) Run same test and retrieve number of errors

balancer:
        Currently enabled:  no
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                339 : Success
                105 : Failed with error 'aborted', from shard0001 to shard0002
                102 : Failed with error 'aborted', from shard0001 to shard0000
                19 : Failed with error 'aborted', from shard0002 to shard0001
                24 : Failed with error 'aborted', from shard0002 to shard0000
 
	[before]
                78 : Failed with error 'aborted', from shard0001 to shard0002
                76 : Failed with error 'aborted', from shard0001 to shard0000
                19 : Failed with error 'aborted', from shard0002 to shard0001
                24 : Failed with error 'aborted', from shard0002 to shard000

Participants:

 Description   

shardCollection({"db.collname":"hashed"})

silently fails to equally distribute chunks (ranges) when multiple shardCollections are issued.

Shard status show hash split range correct for all collections, but distribution among the 3 shards is inconsistent.
Expected 2-2-2 distribution is often not present.
Failed with error 'aborted' balancer errors indicate moveChunk() failing for many hash ranges
Enabling balancer redistributes all collections to 2-2-2 distribution.



 Comments   
Comment by Kelsey Schubert [ 10/Feb/16 ]

Hi davidson.hutton@interactivedata.com,

Thank you for the report. This is behavior is expected. A new chunk migration may not occur while a previous chunk migration is in progress. This is noted in documentation here

If you are continuing to experience this issue, I would recommend executing chunk migrations from a single thread. Or, as you mentioned, enable the balancer after failed migrations to balance the chunks.

Kind regards,
Thomas

Comment by william hutton [ 03/Feb/16 ]

One script creates collections hstest.xxxnn and the other script creates different collections hstest.zzznn.
Script in step (2) should be zzznn, not xxxnn.

Examinig config.changelog shows aborted moveChunk entries.

Comment by william hutton [ 25/Jan/16 ]

I know similar behavior occurs when creating multiple collections with pre-split ranges & moveChunk() in multiple simultaneous shells.
But in this case, the moveChunk() command explicitly returns error.
In general, after initial error, it is unlikely retrying moveChuck will work within a reasonable amount of time.
When programatically defining many tables dynamically, continued failures are unacceptable.
If moveChunk() error is ignore, balancer will eventually redistribute chunks across shards.

I suspect similar behavior occurs when creating multiple collections using tag aware shards & subsequent shard key range definitions in multiple simultaneous shells.

Generated at Thu Feb 08 04:00:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.