Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3393

Dropping a sharded collection and re-sharding it leads to inconsistent inserts

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.8.2
    • Component/s: Sharding
    • None
    • Environment:
      Ubuntu Natty Narwhal, EC2
    • ALL

      Steps to repro (has happened 3 times to us):

      Given a cluster of 2 shards, (let's call them S0, S1)
      And two application servers (let's call them A0, A1), each running mongos locally.
      And finally, one non-application server, T0, that also has a mongos running, where we're running the admin commands
      And a collection 'comments'

      1. On T0, issue enablesharding and shardcollection
      2. Run a script to pre-split chunks ( splitChunk and moveChunk )
      3. 'comments' is now presplit evenly into 102 chunks, 51 on each
      4. Start throwing live traffic from A0 and A1

      At this point, we noticed the pre-split wasn't quite correct and was only writing to S1 (our mistake), so we did:

      1. Stopped throwing writes at cluster
      2. On T0, issued a collection .drop()
      3. Issued another shardcollection with that collection name, which re-created the collection
      4. Ran the corrected pre-split
      5. Threw live traffic from A0 and A1

      Expected behavior would be to have the mongos on A0 and A1 pick up the new chunk information; instead, all the inserts went into S1, as if the mongos on A0 and A1 never picked up the changes from the second pre-split (even though printShardingStatus() on A0 and A1's mongos looked okay).

      The only solution we found was to restart mongos after issuing our presplit commands. After restarting mongos, all the writes went to the correct places, split between S0 and S1.

      Seems like either a bug in mongos failing to pick up these changes, or the docs should be updated specifying that mongos must be restarted after a collection has been dropped and re-sharded.

            Assignee:
            Unassigned Unassigned
            Reporter:
            mikeyk Mike K
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: