Skip to content

Latest commit

 

History

History
207 lines (145 loc) · 7.93 KB

GLOSSARY.md

File metadata and controls

207 lines (145 loc) · 7.93 KB

Kafka client

The domain language of this repository's Kafka client and the tools built on it. Use these words in code, docs, issues and reviews.

Cluster and protocol

Broker: One Kafka server that stores partitions and serves clients, identified in cluster metadata by its node id. Avoid: server, host, node (when you mean the broker rather than its id)

Bootstrap broker: A broker address the client is given up front and uses only to discover the cluster. Avoid: seed broker, bootstrap server

Controller: The node that manages cluster metadata and applies cluster-wide changes such as creating topics. Under KRaft it is one of a quorum of controllers; under ZooKeeper it is one elected broker. Avoid: master, leader

Bootstrap controller: A controller address an admin client connects to directly, instead of a bootstrap broker, to manage a KRaft cluster (KIP-919).

Metadata: The client's current view of the cluster: its brokers, topics, partitions and partition leaders. Avoid: topology, cluster state

Coordinator: The broker responsible for one consumer group, share group or transactional id. Name it by its kind: group coordinator or transaction coordinator. Avoid: controller (for this role)

Kafka API: One request and response type of the Kafka protocol, named as the protocol names it (Produce, Fetch, JoinGroup) and identified by its api key. Avoid: endpoint, command, RPC

API version: The version of a Kafka API used with one broker: the highest version both the client and that broker support.

Flexible version: The first version of a Kafka API that uses compact encodings and tagged fields (KIP-482).

Topics and messages

Topic: A named, append-only stream of messages, split into partitions.

Partition: One ordered, append-only log within a topic. Ordering holds within a partition, not across a topic.

Replica: A copy of a partition held by one broker. One replica is the partition leader; the others follow it.

In-sync replica: A replica that is caught up with its partition leader closely enough to take over as leader. Avoid: ISR (in prose)

Partition leader: The replica, and so the broker, that accepts produce and fetch requests for one partition. Avoid: partition owner

Leader epoch: The number of a partition leader's current term. A request based on an older epoch is stale.

Topic id: The broker-assigned unique id of a topic, used in place of its name by newer API versions.

Message: One key, value, headers and timestamp entry that a user produces or consumes. Avoid: event

Record batch: The wire unit that carries messages to or from one partition (format v2, Kafka 0.11+). Older brokers use the legacy message set format (v0/v1) instead.

Offset: A message's position within its partition. Avoid: index, sequence

High watermark: The offset just after the last message replicated to all in-sync replicas: the end of what consumers may read.

Last stable offset: The offset before which no transaction is still open: the end of what a read_committed consumer may read.

Isolation level: Whether a consumer reads messages from open or aborted transactions (read_uncommitted) or only committed ones (read_committed).

Producing

Linger: The time the producer waits to collect more messages into a batch before sending it.

Partitioner: The strategy that picks a partition for a message sent without one.

Idempotent producer: A producer whose retries cannot write the same message twice to a partition. Avoid: exactly-once producer (that also needs transactions)

Producer id: The broker-assigned identity of an idempotent or transactional producer. Avoid: PID (in prose)

Producer epoch: A counter on the producer id that rises each time the identity is re-initialized.

Epoch bump: Re-initializing the producer id to get a higher producer epoch, which also restarts sequence numbers (KIP-360).

Sequence number: A per-partition counter on produced batches that the broker uses to drop duplicate retries.

Transactional id: The user-chosen, stable name that carries a transactional producer's identity across restarts.

Transaction: A set of produced messages and consumer offset commits that become visible together or not at all.

Consumer groups

Consumer group: Consumers sharing a group id that split a topic's partitions so each partition belongs to one member. Avoid: group (alone, where a share group could be meant)

Member: One consumer within a consumer group or share group, identified by its member id. Avoid: client, instance (when you mean the member rather than its group instance id)

Static member: A member with a stable group instance id, so a restart within its session timeout does not trigger a rebalance.

Group protocol: The membership protocol a consumer group uses: classic (JoinGroup/SyncGroup with client-side assigners) or consumer (KIP-848, broker-side assignment through heartbeats). Avoid: old protocol, new protocol, next-gen protocol

Generation: The classic protocol's count of completed rebalances. Commits tagged with an older generation are rejected. Avoid: epoch (for the classic protocol)

Member epoch: The consumer protocol's per-member counter of assignment changes, its equivalent of the generation. Avoid: generation (for the consumer protocol)

Fenced: Rejected by the broker because a newer instance holds the same identity: a higher producer epoch, member epoch or generation. Avoid: kicked, evicted, zombie

Rebalance: A redistribution of partitions among a consumer group's members after membership or subscription changes.

Eager rebalance: A rebalance in which every member revokes its whole assignment before partitions are redistributed.

Cooperative rebalance: A rebalance in which members give up only the partitions that move to another member (KIP-429). Avoid: incremental rebalance

Assignment: The set of partitions a member currently owns.

Assigner: A client-side strategy that computes assignments under the classic group protocol: range, round-robin, sticky or cooperative-sticky. Avoid: assignor (except for the broker-side strategy of the consumer protocol, which Kafka calls an assignor)

Revoked: Given up cleanly by a member during a rebalance, after it had the chance to commit.

Lost: Taken from a member without a clean revoke, because its session expired or it was fenced. Pending commits for lost partitions are abandoned.

Assign mode: Consuming exact partitions chosen by the application, with no group membership and no rebalances. Avoid: standalone consumer

Committed offset: The offset a consumer group has stored for a partition: where consumption resumes after a restart or reassignment.

Position: The offset of the next message the consumer will hand to the application for a partition. Avoid: current offset

Lag: How far a consumer's position trails the end of its partition.

Seek: Moving a partition's position to a chosen offset.

Pause: Stopping fetches for a partition without giving it up. Avoid: suspend

Fetch session: Per-broker incremental fetch state (KIP-227) that lets a fetch list only the partitions that changed.

Share groups

Share group: A group in which any member can receive any message from any partition, giving queue-like delivery (KIP-932). It has no partition assignment and no seek. Avoid: queue (as a name), consumer group

Acquired: Held by one share group member under a broker-side lock until the member acknowledges the message or the lock expires. Avoid: leased, claimed

Acknowledgement: A share group member's verdict on an acquired message: accept, release (make it available for redelivery), reject, or renew the lock. Avoid: ack (in prose), commit

Share session: A share group member's per-broker fetch state (KIP-932), the share group counterpart of a fetch session. Opening a new one releases the messages the previous one acquired.