[SERVER-7963] mongos does not find migrated data after sharding init Created: 17/Dec/12  Updated: 06/Dec/22  Resolved: 19/Apr/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kristina Chodorow (Inactive) Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Sharding
Operating System: ALL
Participants:
Case:

 Description   

If you have a mongos that you create the sharded cluster with and another mongos that the application is accessing, if you add a shard and data is migrated to the new shard, an app doing queries through the non-config mongos will not update its config and so will not find migrated documents.

Working on writing up the reproduction steps.



 Comments   
Comment by Kristina Chodorow (Inactive) [ 17/Dec/12 ]

Summary: it seems like a read-only load isn't flushing the router config correctly.

I think this was all the steps I did, but it was quite a process so I may have missed something. Basically, I set up two replica set shards, 3 config servers, and two mongoses. I used one mongos as a "admin" mongos that just does setup. I configured another to be an "application" mongos that just does RW (and, once the collection is populated, only reads).

  1. Start two replica sets:

    > // shell 1
    > rs1 = new ReplSetTest({nodes:3, name:"rs1", startPort:31000})
    > rs1.startSet()
    > rs1.initiate()
    >
    > // shell 2
    > rs2 = new ReplSetTest({nodes:3, name:"rs2", startPort:31100})
    > rs2.startSet()
    > rs2.initiate()

  2. Start 3 config servers:

    $ # shell 3
    $ ./mongod --port 28000 --dbpath /tmp/config0
    $
    $ # shell 4
    $ ./mongod --port 28001 --dbpath /tmp/config1
    $
    $ # shell 5
    $ ./mongod --port 28002 --dbpath /tmp/config2

  3. Start two mongos processes:

    $ # shell 6
    $ ./mongos --port 30999 --configdb ubuntu:28000,ubuntu:28001,ubuntu:28002
    $
    $ # shell 5
    $ ./mongos --port 40000 --configdb ubuntu:28000,ubuntu:28001,ubuntu:28002

  4. Now, down to business. Connect a new shell to the 30999 mongos. This is the "admin mongos": all sharding config commands will go through here. Add r1 as a shard.
  5. Open up another shell, this one connected to the mongos at port 40000. This is the "app mongos": all inserts and queries go through this one. Add some data that is trackable:

    > ids = []
    > write = function () { 
        rid=ObjectId(); 
        db.foo.insert({_rid:rid}); 
        ids.push(rid);
    }
    > for (i=0; i<100000; i++) { write(); }

  6. Make the chunk size 1 (through the admin shell).
  7. Create a loop in the app shell to do random reads of the writes:

    > x = function () { 
        var target = ids[Math.floor(Math.random()*ids.length)];
        if (db.foo.findOne({_rid:target}) == null) {
            print("Couldn't find "+target);
        } 
    }
    > while (true) { x(); sleep(1000); }

    At this point, the loop should just be sitting there, printing nothing.

  8. Add the other shard (through the admin shell):

    > sh.addShard("rs2/ubuntu:31100")

  9. Enable sharding on the db (through the admin shell):

    > sh.enableSharding("test")

  10. Create an index on _rid (through the admin shell):

    > use test
    > db.foo.ensureIndex({_rid:1})

  11. Finally, shard the collection (through the admin shell):

    > sh.shardCollection("test.foo", {_rid:1})

After ~30 seconds, the infinite "app" loop starts printing that it cannot find ObjectIds!

Generated at Thu Feb 08 03:16:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.