Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16808

rs.reconfig makes whole cluster as Secondary

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major - P3 Major - P3
    • None
    • 2.4.3
    • Admin, Replication
    • None
    • ALL
    • Hide

      Setup cluster with 4 nodes.
      Make one node hidden and 24 hours delay.
      Put lot of load.
      Change 24 hours delay to 0.

      Show
      Setup cluster with 4 nodes. Make one node hidden and 24 hours delay. Put lot of load. Change 24 hours delay to 0.

    Description

      We have the following config for replica set.

      ecset01:PRIMARY> cfg = rs.conf()
      {
      	"_id" : "ecset01",
      	"version" : 25,
      	"members" : [
      		{
      			"_id" : 1,
      			"host" : "10.98.2.21:27017",
      			"priority" : 2
      		},
      		{
      			"_id" : 2,
      			"host" : "10.98.2.22:27017",
      			"priority" : 3
      		},
      		{
      			"_id" : 3,
      			"host" : "10.98.2.20:27017",
      			"priority" : 4
      		},
      		{
      			"_id" : 4,
      			"host" : "10.98.141.170:27017",
      			"priority" : 0,
      			"hidden" : true
      		},
      		{
      			"_id" : 5,
      			"host" : "10.98.141.120:27017",
      			"priority" : 0,
      			"slaveDelay" : 86400,
      			"hidden" : true
      		},
      		{
      			"_id" : 6,
      			"host" : "10.10.14.158:27017",
      			"priority" : 0,
      			"slaveDelay" : 86400,
      			"hidden" : true
      		}
      	]
      }

      we try to reset slaveDelay from 86400 to 0 .

      cfg = rs.conf()
      cfg.members[5].slaveDelay = 0
      rs.reconfig(cfg)
      Mon Jan 12 11:30:15.802 DBClientCursor::init call() failed
      Mon Jan 12 11:30:15.804 trying reconnect to 127.0.0.1:27017
      Mon Jan 12 11:30:15.804 reconnect 127.0.0.1:27017 ok
      Mon Jan 12 11:30:16.007 DBClientCursor::init call() failed
      Mon Jan 12 11:30:16.008 JavaScript execution failed: Error: DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { authenticate: 1, nonce: "fe555b6fcb676ba7", user: "admin", key: "a2d59cbc51cf8c61b4cb45b7f4f8db80" } at src/mongo/shell/query.js:L78
      > 
      Mon Jan 12 11:30:20.139 trying reconnect to 127.0.0.1:27017
      Mon Jan 12 11:30:20.139 reconnect 127.0.0.1:27017 ok
      ecset01:SECONDARY> 

      It makes whole cluster as Secondary. Error on MongoLog:

      Mon Jan 12 11:30:15.803 [conn5988790] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.98.0.30:51311
      9]
      Mon Jan 12 11:30:15.803 [conn6128487] replSet replSetReconfig new config saved locally
      Mon Jan 12 11:30:15.803 [conn6124807] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1421066044000|13 } } cursorid:6909070915536108841 ntorett
      urn:0 keyUpdates:0 numYields: 1 locks(micros) r:132 nreturned:1 reslen:117 1054ms
      Mon Jan 12 11:30:15.803 [conn6128487] command admin.$cmd command: { replSetReconfig: { _id: "ecset01", version: 26, members: [ { _id: 1, host: "10.99
      8.2.21:27017", priority: 2.0 }, { _id: 2, host: "10.98.2.22:27017", priority: 3.0 }, { _id: 3, host: "10.98.2.20:27017", priority: 4.0 }, { _id: 4,  
      host: "10.98.141.170:27017", priority: 0.0, hidden: true }, { _id: 5, host: "10.98.141.120:27017", priority: 0.0, slaveDelay: 86400, hidden: true },,
       { _id: 6, host: "10.10.14.158:27017", priority: 0.0, slaveDelay: 0, hidden: true } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:977929 reslen:37  
      1065ms
      Mon Jan 12 11:30:15.803 BackgroundJob starting: rsHealthPoll
      Mon Jan 12 11:30:15.803 [conn6128487] Socket say send() errno:9 Bad file descriptor 127.0.0.1:35581
      Mon Jan 12 11:30:15.803 [conn6121755] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1421065202000|5 } } cursorid:6905521952664444159 ntoretuu
      rn:0 keyUpdates:0 numYields: 1 locks(micros) r:141 nreturned:1 reslen:37 1018ms
      Mon Jan 12 11:30:15.803 [conn6116610] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1421062683000|18 } } cursorid:6894912989969242768 ntorett
      urn:0 keyUpdates:0 numYields: 1 locks(micros) r:133 nreturned:1 reslen:37 1038ms
      Mon Jan 12 11:30:15.803 [conn6128487] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:355811
      ]
      Mon Jan 12 11:30:15.803 [conn6116610] Socket say send() errno:9 Bad file descriptor 10.98.141.170:56737
      Mon Jan 12 11:30:15.803 [conn6124807] Socket say send() errno:9 Bad file descriptor 10.98.2.22:52034
      Mon Jan 12 11:30:15.803 [conn6121755] Socket say send() errno:9 Bad file descriptor 10.98.2.21:43302
      Mon Jan 12 11:30:15.803 [slaveTracking] User Assertion: 11000:E11000 duplicate key error index: local.slaves.$_id_  dup key: { : ObjectId('53aa302c77
      1e48a32f08ebb5d') }
      Mon Jan 12 11:30:15.804 [conn6124807] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.98.2.22:52033
      4]
      Mon Jan 12 11:30:15.804 [conn6121755] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.98.2.21:43300
      2]
      Mon Jan 12 11:30:15.804 [conn6116610] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.98.141.170:55
      6737]
      Mon Jan 12 11:30:15.804 [slaveTracking] update local.slaves query: { _id: ObjectId('53aa302c71e48a32f08ebb5d'), config: { _id: 1, host: "10.98.2.21::
      27017", priority: 2.0 }, ns: "local.oplog.rs" } update: { $set: { syncedTo: Timestamp 1421080214000|28 } } nscanned:1 fastmodinsert:1 keyUpdates:0 ee
      xception: E11000 duplicate key error index: local.slaves.$_id_  dup key: { : ObjectId('53aa302c71e48a32f08ebb5d') } code:11000 locks(micros) w:416 00
      ms

      We are able to make Primary after shutting down "host" : "10.10.14.158:27017". We like to know why MongoDB behaves like this.

      Attachments

        Activity

          People

            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            sarbamangal Sarbamangal Choudhury
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: