[TIL]

Difference between Chubby and Zookeeper

• Consensus Algorithm:
  • Chubby: Using Paxos consensus algorithm
  • Zookeeper: Using ZAB (which is a modified algorithm of Paxos)
• Access Path:
  • Chubby: Must through leader, follower don’t accept any command directly.
  • Zookeeper: Can accept any command from all follower, but will go back to leader.
• Data Out-Of-Date:
  • Chubby: No rish on out-of-date, because all data fro Leader.
  • Zookeeper: Will get out-of-date data, if read without sync command. Sync command will force ask follower ask Leader first before return result to client.

The tolerance of Quorum-Backup and Primary-Backup:

Majority Quorums:

For Paxos or ZAB, A leader election need meet majority quorum ( n/2 +1 ) . It means if n=2f at most the service can tolerance f service failed at the same time. ( Quorum-Backup Replica ).

Primary-Backup

• All data must confirm by all follower confirmed.
• Much slower than “Quorum-Backup”.

Kafka ISR (In-Sync Replica)

In-Sync Replica is a backup policy which similar with Primary-Backup but in sychronous replication.

Server store two copy of replica data:

• ISR (The latest one) store in Leader.
• Follower store replica data might later than ISR.
• Use Zookeeper to update all follower replica data from ISR.

In this case, if total server is n=f+1, we could tolerance number of f service failed. ( Primary-Backup replica )

trade-off:

• Follower data critical late than leader, but in Kafka this case might be acceptable.
• If follower cannot catch up data from leader, will drop ISR by leader.
• If failed on leader, it has high possibility lost data if next leader. (Choose leader from non-ISR list, because all follower in ISR leave behind.)
• Use Zookeeper to store ISR will introduce split brain issue. (two majority group issue).

Refer to Kafka ISR

Reference

• Distributed Coordination with Zookeeper
• paxos(chubby) vs zab(Zookeeper)
• Zookeeper 和 Chubby 有哪些不同点？
• Big Data: Beyond the storage, what’s else?
• Kafka ISR
• Kafka设计解析（二）：Kafka High Availability （上）