|
This happened in a three member replica set:
1. Member_1: Data node
2. Member_2 : Data node
3. Member_3: Arbiter-Only node
At some point of time when Member_1 was Primary and Member_2 was Secondary and Member_3 was Arbiter I brought down Member_2 node to do some hardware repairs and upgrades.
Member_2 was down for a couple of hours. What I noticed just before bringing Member_2 node up again is that Member_1 is, as expected, Primary, but Member_3 is not Arbiter anymore but rather SECONDAY!
Somehow it is promoted itself. Its Secondary status was evident from connecting to it from the shell: it would display a prompt with “SECONDARY” label in it. Also it would show up as Secondary when rs.status() command was executed.
Furthermore, the data directory if Member_3 was showing some new files which were collections data files from the rest of the replica set. The total size of these files was 10 GB at that time.
I brought up Member_2 and continued monitoring the situation. After several hours my replica set had one Primary and two Secondaries: Member_3 continued to be secondary.
Furthermore, the data directory for Member_3 continued to grow to 60GB (which is the size of Member_1 data directory). Which means that by that time it fully synced with the Primary.
And so Member_3 continued to be secondary until I removed it from replica set, restarted Member_3 instance and added it back to the Replica Set. After that it carried on as Arbiter.
|