What does it mean to optimize Kafka for High Availability?
This means that the Kafka applications should recover as quickly as possible from failures to ensure reads and writes are always available from and to topic partitions by the respective Kafka clients.
Minimum In-sync Replicas
When Producer configuration acks is set to all, then min.insync.replicas specifies the minimum number of ISRs that must acknowledge a write before a produce request (which is a write request) can be considered successful. If this set minimum is not met, then the produce request fails and the producer will throw an exception. Thus, this decreases the availability of a Kafka partition for writes. So setting a lower value of this configuration makes the cluster capable of tolerating more replica failures. Kindly check the following section to understand the trade-offs that need to be taken care of while settings its value.
Trade-offs
Write Availability vs Durability
Read Availability vs Durability
No trade-off. More details - https://krishnakrmahto.com/configure-kafka-for-high-durability#heading-durability-vs-read-availability
Consumer Failures
Kakfa can detect consumer failures and initiate rebalance so that the partitions that the failed consumers were consuming from are distributed amongst the remaining consumers belonging to the same consumer group.
Consumer failures can be seen as:
Hard failures (for e.g. - SIGKILL) which imply a certain failure of the consumer process.
Soft failures (for e.g. - consumer session time out) which imply the consumer process is deemed to have failed by Kafka, however, the consumer application might still be processing the messages it had polled last.
Both of the above failures are detected by Kafka cluster after which a rebalance is triggered.
The soft failures can be mitigated when the reasons are related to network, and/or message processing time, or any configuration in general by tuning the following configurations.
heartbeat.interval.ms, session.timeout.ms
heartbeat.interval.msis the interval between two successive heartbeats that are sent to the consumer group coordinator (which is one of the Kafka brokers) to let it know that the consumer process is alive. These heartbeats are sent by a background thread.heartbeat.interval.msmust be set lower thansession.timeout.ms. It is recommended to setheartbeat.interval.ms<= (1/3) *session.timeout.ms[1].If the consumer group coordinator does not receive a heartbeat within
session.timeout.msfrom a consumer, then the consumer will be removed from the group and a rebalance will be initiated by the consumer group coordinator.Setting lower values for
session.timeout.msandheartbeat.interval.mswill help detect failed consumers and recovery faster, but this could also cause undesirable rebalances - for e.g. if the consumer usually takes a only few more seconds than thesession.timeout.msto process the messages polled.Setting higher values for these configurations would reduce the chances of unwanted rebalances, but it also entails a longer time to detect real failures.
Note that the value must be in the allowable range as configured in the broker configuration by
group.min.session.timeout.msandgroup.max.session.timeout.ms.
max.poll.interval.ms,max.poll.recordsmax.poll.interval.mssets the maximum interval between two successive calls to poll(). When this timeout occurs, then the consumer also stops sending heartbeats and the background thread sends an explicit LeaveGroup request eventually resulting in a rebalance. If the consumer is supposed to take longer to process a batch of messages, then we should increase the value of this configuration.The max poll interval configuration is for the main thread that processes the messages in contrast to
session.timeout.mswhich is for the background heartbeat thread. It is possible, for e.g., that the main thread is deadlocked while the background thread is still sending heartbeats -max.poll.interval.msensures that such cases are addressed and acted upon.We can also reduce
max.poll.recordswhich should decrease the total time the consumer takes to process the batch.Even if we try configuring
max.poll.records, it is still difficult to predict the exact interval between two poll() requests. Ideally,max.poll.interval.msvalue should be set to a value that is high enough so that it can be reached only rarely by a healthy consumer application, but also it has to be low enough so that we do not end up with a hanging consumer main thread for too long.The fact that
session.timeout.msis configured to be less than or equal tomax.poll.interval.msensures that an unhealthy consumer is detected and recovered from much earlier.
References
Kafka - The Definitive Guide
If references to declarative claims made in this doc are not found in the references above, then they may have been taken directly from the Kafka code base - https://github.com/apache/kafka.