Defined by QnA
Lately in my work, I got here throughout a situation the place the appliance wanted to devour messages from a number of queues. I discovered it attention-grabbing, how Kafka manages to supply such capacity to the customers in a dependable method.
I had a whole lot of questions in my thoughts that I wanted solutions to earlier than continuing with the implementation of such a design and wished to grasp how Kafka works in such a situation.
Let’s take a look at a few of these questions and the solutions I discovered throughout my search.
Q1. Is it allowed in Kafka for one shopper group to devour messages from a number of matters directly?
A. Sure, Kafka’s design permits customers from one shopper group to devour messages from a number of matters.
The protocol underlying shopper.ballot() permits sending requests for a number of partitions(throughout matters as effectively) in a single request.
When shopper.ballot() is known as, FetchRequest objects are despatched to brokers which are internet hosting partition leaders and all i.e. max.ballot.data enqueued messages are returned.
shopper.ballot() returns a map of matter to data for the subscribed checklist of matters and partitions.
Q2. Find out how to subscribe to a number of matters?
A.
shopper.subscribe(Arrays.asList(topic1, topic2))
Yeah, that is easy. After realizing that the patron API gives this capacity. I used to be positive that Kafka have to be dealing with such a situation underneath the hood however, I used to be curious to know extra.
Q3. How does Kafka make sure that matters are usually not starved?
A. Messages are enqueued ranging from the primary partition, if there are not any extra messages within the present partition left, however there are nonetheless bytes to fill, messages from the following partition can be enqueued till there are not any extra messages or the buffer is full.
After the patron receives the buffer, it should cut up it into CompletedFetches, the place one CompletedFetch comprises all of the messages of 1 matter partition, the CompletedFetches are enqueued
The enqueued CompletedFetches are logically flattened into one large queue, and for the reason that requests to every partition are despatched in parallel they could be combined collectively
shopper.ballot() will learn and dequeue at most max.ballot.data from that flattened large queue.
Subsequent fetch requests exclude all the subject partitions which are already within the flattened queue.
Because of this you’ll don’t have any ravenous messages and message consumption occurs in a spherical robin method, however you will have numerous messages from one matter, earlier than you’ll get numerous messages for the following matter.
This autumn. How are offsets dedicated?
A. Offsets dedicated by customers are saved in a particular Kafka matter known as __consumer_offsets which persists offsets for every partition of every matter.
Q5. How to make sure optimum throughput?
A. To make sure message consumption from numerous matters occur optimally and keep away from massive batches of messages from one matter after the opposite you might want to take a look at the next properties:
- Message measurement — The scale of the message being consumed from a subject
fetch.min.bytes— The minimal quantity of bytes a shopper needs to obtain from the dealermax.partition.fetch.bytes— The utmost variety of bytes to be consumed per partition
