Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-23975

migrating chunks from one shard to another which is also set to be removed

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.6
    • Component/s: Sharding
    • None
    • ALL

      We are migrating data from a legacy set of shards to a new set. Looking at the logs I see that it is moving chunks from one shard to another which is also in the set of shards to be removed. Of course this means that the contents of that chunk are going to be migrated more than 1 time, making the already painfully slow balancing process even worse.

      Here's an excerpt from the log showing that it is moving a chunk from Shard_18 to Shard_83:

      Apr 28 12:33:48 chron-queryb ip-10-43-48-90:  docker/mongowt[5237]:  2016-04-28T19:33:48.295+0000 I NETWORK  [conn1822] end connection 172.17.0.1:41782 (3 connections now open)
      Apr 28 12:34:01 chron-queryb ip-10-43-48-90:  docker/shardstrap[5237]:  [INFO] - Shard(s) to remove: set([u'Shard_26', u'Shard_73', u'Shard_52', u'Shard_77', u'Shard_89', u'Shard_88', u'Shard_87', u'Shard_86', u'Shard_85', u'Shard_84', u'Shard_83', u'Shard_82', u'Shard_81', u'Shard_80', u'Shard_29', u'Shard_28', u'Shard_25', u'Shard_24', u'Shard_27', u'Shard_53', u'Shard_21', u'Shard_20', u'Shard_23', u'Shard_22', u'Shard_61', u'Shard_60', u'Shard_63', u'Shard_62', u'Shard_65', u'Shard_64', u'Shard_67', u'Shard_66', u'Shard_69', u'Shard_68', u'Shard_41', u'Shard_40', u'Shard_47', u'Shard_46', u'Shard_45', u'Shard_44', u'Shard_48', u'Shard_71', u'Shard_54', u'Shard_79', u'Shard_100', u'Shard_50', u'Shard_98', u'Shard_99', u'Shard_49', u'Shard_42', u'Shard_94', u'Shard_95', u'Shard_96', u'Shard_97', u'Shard_90', u'Shard_91', u'Shard_92', u'Shard_93', u'Shard_38', u'Shard_39', u'Shard_43', u'Shard_55', u'Shard_32', u'Shard_72', u'Shard_30', u'Shard_31', u'Shard_36', u'Shard_37', u'Shard_34', u'Shard_35', u'Shard_58', u'Shard_59', u'Shard_70', u'Shard_78', u'Shard_33', u'Shard_76', u'Shard_51', u'Shard_74', u'Shard_75', u'Shard_18', u'Shard_19', u'Shard_56', u'Shard_57'])
      Apr 28 12:34:01 chron-queryb ip-10-43-48-90:  docker/shardstrap[5237]:  [ERROR] - Manual shard removal needed/in progress -OR- Critical error is occuring
      Apr 28 12:34:01 chron-queryb ip-10-43-48-90:  docker/shardstrap[5237]:  [INFO] - Sleeping for 600
      Apr 28 12:34:04 chron-queryb ip-10-43-48-90:  docker/mongowt[5237]:  2016-04-28T19:34:04.005+0000 I SHARDING [Balancer] ChunkManager: time to load chunks for google_us.review: 22ms sequenceNumber: 60 version: 8598|1||557fc21623a54038a6930db5 based on: 8597|1||557fc21623a54038a6930db5
      Apr 28 12:34:04 chron-queryb ip-10-43-48-90:  docker/mongowt[5237]:  2016-04-28T19:34:04.025+0000 I SHARDING [Balancer] moving chunk ns: google_us.recommended moving ( ns: google_us.recommended, shard: Shard_18:Shard_18/10.37.28.238:27018,10.37.41.191:27018,10.43.26.234:27018, lastmod: 12|0||000000000000000000000000, min: { app_id: "com.amazesoft.collage.mania" }, max: { app_id: "com.anip.wallpaper.live.moonlight" }) Shard_18:Shard_18/10.37.28.238:27018,10.37.41.191:27018,10.43.26.234:27018 -> Shard_83:Shard_83/10.37.24.79:27018,10.37.44.232:27018,10.43.31.196:27018
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            robreid Rob Reid
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: