Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27360

Replicaset primary flapping when using high priority on primary and network split happens

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.11
    • Component/s: Replication
    • Labels:
      None
    • Environment:
      OSX 10.11.4
    • ALL
    • Hide

      Toxiproxy repoduction

      • Install toxiproxy on your machine (brew or whatever package manager you have)
      • Open a terminal terminal and run
        toxiproxy-server &
        toxiproxy-cli create mongod_primary -l localhost:12345 -u localhost:31000
        
      • Create a replicaset with one primary and 2 secondaries with ports 31000, 31001, 31002 where the primary has priority 100.
        {
            "_id" : "rs",
            "version" : 1,
            "members" : [
                {
                    "_id" : 0,
                    "host" : "localhost:12345",
                    "priority" : 100
                },
                {
                    "_id" : 1,
                    "host" : "localhost:31001",
                    "priority" : 1
                },
                {
                    "_id" : 2,
                    "host" : "localhost:31002",
                    "priority" : 1
                }
            ]
        }
        
      • Wait for election to finish and a primary to be selected.
      • Now throw the primary in the black hole.
        toxiproxy-cli toxic add mongod_primary -t timeout -a timeout=0 --upstream
        toxiproxy-cli toxic add mongod_primary -t timeout -a timeout=0 --downstream
        
      • Watch as the new primary is elected and then the flapping starts with new elections happening ever X seconds.

      You can turn off the toxics by running the following commands

      toxiproxy-cli toxic remove mongod_primary -n timeout_downstream
      toxiproxy-cli toxic remove mongod_primary -n timeout_upstream
      
      Show
      Toxiproxy repoduction Install toxiproxy on your machine (brew or whatever package manager you have) Open a terminal terminal and run toxiproxy-server & toxiproxy-cli create mongod_primary -l localhost:12345 -u localhost:31000 Create a replicaset with one primary and 2 secondaries with ports 31000, 31001, 31002 where the primary has priority 100. { "_id" : "rs" , "version" : 1, "members" : [ { "_id" : 0, "host" : "localhost:12345" , "priority" : 100 }, { "_id" : 1, "host" : "localhost:31001" , "priority" : 1 }, { "_id" : 2, "host" : "localhost:31002" , "priority" : 1 } ] } Wait for election to finish and a primary to be selected. Now throw the primary in the black hole. toxiproxy-cli toxic add mongod_primary -t timeout -a timeout=0 --upstream toxiproxy-cli toxic add mongod_primary -t timeout -a timeout=0 --downstream Watch as the new primary is elected and then the flapping starts with new elections happening ever X seconds. You can turn off the toxics by running the following commands toxiproxy-cli toxic remove mongod_primary -n timeout_downstream toxiproxy-cli toxic remove mongod_primary -n timeout_upstream
    • Storage 2017-01-23

      Replicaset with 1 primary, 2 secondaries where primary has priority 100 and a network split happens causes election flapping.

        1. 31000.log
          30 kB
        2. 31000.log
          404 kB
        3. 31001.log
          21 kB
        4. 31001.log
          378 kB
        5. 31002.log
          20 kB
        6. 31002.log
          448 kB

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            christkv Christian Amor Kvalheim
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: