-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.0.6
-
Component/s: Sharding
-
Labels:None
-
ALL
We are migrating data from a legacy set of shards to a new set. Looking at the logs I see that it is moving chunks from one shard to another which is also in the set of shards to be removed. Of course this means that the contents of that chunk are going to be migrated more than 1 time, making the already painfully slow balancing process even worse.
Here's an excerpt from the log showing that it is moving a chunk from Shard_18 to Shard_83:
Apr 28 12:33:48 chron-queryb ip-10-43-48-90: docker/mongowt[5237]: 2016-04-28T19:33:48.295+0000 I NETWORK [conn1822] end connection 172.17.0.1:41782 (3 connections now open)
Apr 28 12:34:01 chron-queryb ip-10-43-48-90: docker/shardstrap[5237]: [INFO] - Shard(s) to remove: set([u'Shard_26', u'Shard_73', u'Shard_52', u'Shard_77', u'Shard_89', u'Shard_88', u'Shard_87', u'Shard_86', u'Shard_85', u'Shard_84', u'Shard_83', u'Shard_82', u'Shard_81', u'Shard_80', u'Shard_29', u'Shard_28', u'Shard_25', u'Shard_24', u'Shard_27', u'Shard_53', u'Shard_21', u'Shard_20', u'Shard_23', u'Shard_22', u'Shard_61', u'Shard_60', u'Shard_63', u'Shard_62', u'Shard_65', u'Shard_64', u'Shard_67', u'Shard_66', u'Shard_69', u'Shard_68', u'Shard_41', u'Shard_40', u'Shard_47', u'Shard_46', u'Shard_45', u'Shard_44', u'Shard_48', u'Shard_71', u'Shard_54', u'Shard_79', u'Shard_100', u'Shard_50', u'Shard_98', u'Shard_99', u'Shard_49', u'Shard_42', u'Shard_94', u'Shard_95', u'Shard_96', u'Shard_97', u'Shard_90', u'Shard_91', u'Shard_92', u'Shard_93', u'Shard_38', u'Shard_39', u'Shard_43', u'Shard_55', u'Shard_32', u'Shard_72', u'Shard_30', u'Shard_31', u'Shard_36', u'Shard_37', u'Shard_34', u'Shard_35', u'Shard_58', u'Shard_59', u'Shard_70', u'Shard_78', u'Shard_33', u'Shard_76', u'Shard_51', u'Shard_74', u'Shard_75', u'Shard_18', u'Shard_19', u'Shard_56', u'Shard_57'])
Apr 28 12:34:01 chron-queryb ip-10-43-48-90: docker/shardstrap[5237]: [ERROR] - Manual shard removal needed/in progress -OR- Critical error is occuring
Apr 28 12:34:01 chron-queryb ip-10-43-48-90: docker/shardstrap[5237]: [INFO] - Sleeping for 600
Apr 28 12:34:04 chron-queryb ip-10-43-48-90: docker/mongowt[5237]: 2016-04-28T19:34:04.005+0000 I SHARDING [Balancer] ChunkManager: time to load chunks for google_us.review: 22ms sequenceNumber: 60 version: 8598|1||557fc21623a54038a6930db5 based on: 8597|1||557fc21623a54038a6930db5
Apr 28 12:34:04 chron-queryb ip-10-43-48-90: docker/mongowt[5237]: 2016-04-28T19:34:04.025+0000 I SHARDING [Balancer] moving chunk ns: google_us.recommended moving ( ns: google_us.recommended, shard: Shard_18:Shard_18/10.37.28.238:27018,10.37.41.191:27018,10.43.26.234:27018, lastmod: 12|0||000000000000000000000000, min: { app_id: "com.amazesoft.collage.mania" }, max: { app_id: "com.anip.wallpaper.live.moonlight" }) Shard_18:Shard_18/10.37.28.238:27018,10.37.41.191:27018,10.43.26.234:27018 -> Shard_83:Shard_83/10.37.24.79:27018,10.37.44.232:27018,10.43.31.196:27018
- duplicates
-
SERVER-11328 Allow concurrent draining shards
-
- Closed
-
- is related to
-
SERVER-24006 Allow multiple shards to be marked as 'draining' at the same time
-
- Closed
-