MongoDB Replica Sets with Odd and Even Membership
I noticed a few things about MongoDB Replica Sets.
In the 3-node replSet, if PRIMARY is omitted, I can see that the set selects the new PRIMARY and everything is fine, without any downtime. But if the other member goes down (2 total down), one remaining member will not become PRIMARY, and a complete disconnection will occur. I understand this is because replSet does not have a majority for choices.
But that seems silly. Wouldn't my one surviving member be able to work on its own? Is there a way to tweak it so that I get this behavior?
I understand that arbitrators can be used to achieve a majority, however if I add an arbiter for just 4 members, an even number, then wouldn't that also run into problems with the majority? Or if I add 2 arbitrators for a total of 5 voting members, but 1 goes down, would I be left with an even number of voting members and still suspect that replSet won't be able to select PRIMARY?
Overall, I am a little confused about how the "majority" is set in relation to what happens when participants go up or down, and what configuration options I have. My specific questions:
- How to protect against failures in 3-node replSet when 2 participants go down, and / or what is the best practice to safely resolve failures that occur in this scenario?
- In an odd member replSet, what happens when an odd number of members goes down and leaves a replSet with an even number of members on the internet (in the sense that replSet can do most of the choices)?
source to share
How to protect against crashes in 3-node replSet when 2 members go down
Not. If two members are omitted, your replica set becomes read-only, and rightly so. down can be relative - server 1 might say 2 and 3 are down, but really 1 is on the other side of the network partition. If server 1 is protected from 2 member disconnection, it will become the primary and will receive records. However, one of 2 or 3 is also primary, so the set now has two primary values. How do you reconcile conflicting records sent to 1 and sent to primary 2, 3 when the section ends? Probability is your shield against the majority of replica set members declining if a server is down 1% of the time and each server down is independent of the other down (a guess which is probably true except wherewhen servers are hosted), then at least 2 will be down only 1/10000 of the time. If you want the best ratios, use 5 servers per replica set.
what happens when an odd number of members goes down and leaves a replSet with an even number of members online
The replica set requires a majority (in terms of the total number of members of the replica set, not the number currently starting from any one point of view of the member) to select the primary. If any group of members of a replica set, whether it is an even or odd number, see that they make up the majority of a replica set, they will try to select the primary. The majority condition guarantees that there can only be one primary. Thus, 8/11 members talking to each other will choose primary, just like 7/11 or 9/11.
source to share
What is the best practice to safely troubleshoot this scenario?
As mentioned in the previous answer, MongoDB is very diligent in avoiding having two primaries in a replica set, as this can cause serious corruption. If you know that a node is down and does not come back, you can remove it from the replica set. Even if you only have one surviving node, you can say that node is to remove down nodes from the configuration, so that you get a one-node replica set, and one node becomes the primary.If you have no primary information, you will need to use "force" parameter to rs.reconfig () to remove downstream nodes. After that, you can add new nodes to the replica set and they will start copying data from the surviving node. You may need to tweak your application's configuration to reference the new nodes.
source to share