kafka-topics

Wat het doet

Zo schrijf je het

kafka-topics.sh --bootstrap-server HOST:9092 --create|--list|--describe|--alter|--delete [--topic NAME]
kafka-console-producer.sh / kafka-console-consumer.sh --bootstrap-server HOST:9092 --topic NAME

Belangrijkste delen

Tools

kafka-topics.sh

Create, list, describe topics.

kafka-console-producer.sh

Write messages from the terminal.

kafka-console-consumer.sh

Read messages.

kafka-consumer-groups.sh

Groups, offsets, lag.

kafka-configs.sh

Topic and broker settings (retention).

kafka-acls.sh

Access control lists.

kafka-storage.sh

Format storage for KRaft.

Meest gebruikt

Niets past bij dit filter.

$ als gewone gebruiker# als root (of met sudo)

1. 1Run Kafka (KRaft) in a container for a lab

   $podman run -d --name kafka -p 127.0.0.1:9092:9092 docker.io/apache/kafka:3.8.1

2. 2Create and describe a topic

   $kafka-topics.sh --bootstrap-server localhost:9092 --create --topic shop.orders --partitions 6 --replication-factor 1
   $kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic shop.orders

3. 3Send and read test messages

   $echo '{"id": 1, "total": 49.90}' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic shop.orders
   $kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic shop.orders --from-beginning --max-messages 5

4. 4Check consumer lag (are consumers behind?)

   $kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group invoice-service

5. 5Reset a group to re-read a topic

   $kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group invoice-service --topic shop.orders --reset-offsets --to-earliest --execute

6. 6Set retention to 7 days

   $kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name shop.orders --add-config retention.ms=604800000

7. 7Connect with TLS and SASL

   $cat client.properties
   security.protocol=SASL_SSL
   sasl.mechanism=SCRAM-SHA-512
   sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="shop" password="Example-Only";
   ssl.truststore.location=/etc/kafka/truststore.jks
   $kafka-topics.sh --bootstrap-server kafka.lab.local:9093 --command-config client.properties --list

8. 8Allow a user to read one topic (ACL)

   $kafka-acls.sh --bootstrap-server kafka.lab.local:9093 --command-config admin.properties --add --allow-principal User:invoice --operation Read --topic shop.orders --group invoice-service

9. 9List all topics

   $kafka-topics.sh --bootstrap-server localhost:9092 --list

10. 10Strimzi on Kubernetes: a topic as YAML

    $cat topic.yaml
    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaTopic
    metadata: { name: shop.orders, labels: { strimzi.io/cluster: lab } }
    spec: { partitions: 6, replicas: 3 }

Niets past bij dit filter.

Bestanden

config/server.properties

Broker settings.

client.properties

Client TLS/SASL settings (keep private).

Exitcodes