Apache Kafka

Apache Kafka

Event streaming for Kubernetes. Strimzi operator, KRaft mode, no ZooKeeper.

Quick Start (Tested)

make install    # Deploy Strimzi + Kafka
make test       # Verify everything works
make status     # Show resources
make uninstall  # Clean up

Requirements: Kubernetes cluster, Helm 3+

Versions: Strimzi 0.49+, Kafka 4.1.1

---

Resource Detection & Adaptation

Before generating manifests, detect the target environment:

# Detect machine memory
sysctl -n hw.memsize 2>/​dev/​null | awk '{print $0/​1024/​1024/​1024 " GB"}' || \
  grep MemTotal /​proc/​meminfo | awk '{print $2/​1024/​1024 " GB"}'

# Detect Docker Desktop allocation
docker info --format '{{.MemTotal}}' 2>/​dev/​null | awk '{print $0/​1024/​1024/​1024 " GB"}'

# Detect Kubernetes node capacity
kubectl get nodes -o jsonpath='{.items[0].status.capacity.memory}' 2>/​dev/​null

Adapt resource configuration based on detection:

Adaptive Resource Templates

Minimal (detected < 12GB):

resources:
  requests:
    memory: 512Mi
    cpu: 200m
  limits:
    memory: 1Gi
    cpu: 500m

⚠️ Agent should warn: "Limited resources detected. Kafka may be unstable under load."

Standard (detected 12-24GB):

resources:
  requests:
    memory: 1Gi
    cpu: 250m
  limits:
    memory: 2Gi
    cpu: 1000m

Production (detected > 24GB or real cluster):

resources:
  requests:
    memory: 4Gi
    cpu: 1000m
  limits:
    memory: 8Gi
    cpu: 4000m

Agent Behavior

1. Always detect before generating manifests
2. Adapt resource configs to detected environment
3. Warn if resources are insufficient for requested workload
4. Suggest Docker Desktop settings if running locally

---

What This Skill Does

What This Skill Does NOT Do

• Deploy ZooKeeper (KRaft only)
• Manage Kafka Streams applications
• Configure multi-datacenter replication

---

Deployment

Install Strimzi Operator

helm repo add strimzi https://strimzi.io/​charts
helm install strimzi-operator strimzi/​strimzi-kafka-operator -n kafka --create-namespace --wait

Deploy Kafka Cluster

kubectl apply -f manifests/​kafka-cluster.yaml -n kafka
kubectl wait kafka/​dev-cluster --for=condition=Ready --timeout=300s -n kafka

Create Topic

kubectl apply -f manifests/​kafka-topic.yaml -n kafka

Verify

kubectl get kafka,kafkatopic,pods -n kafka

---

Core Concepts

Topic      = Named stream (like a database table)
Partition  = Ordered log within topic (parallelism unit)
Consumer Group = Consumers sharing work (partition → one consumer)
Offset     = Consumer position (commit to track progress)
Broker     = Kafka server
Controller = Metadata manager (KRaft replaces ZooKeeper)

---

Local Development

Connect from your host machine (no port-forward needed):

# From your local machine (outside Kubernetes)
producer = Producer({'bootstrap.servers': 'localhost:30092'})

Connect from inside Kubernetes (pod-to-pod):

# From another pod in the cluster
producer = Producer({'bootstrap.servers': 'dev-cluster-kafka-bootstrap.kafka:9092'})

---

Producer/​Consumer (Python)

from confluent_kafka import Producer, Consumer

# Producer (production config)
producer = Producer({
    'bootstrap.servers': 'localhost:30092',  # Or K8s service for pods
    'acks': 'all',
    'enable.idempotence': True,
})
producer.produce('my-topic', key='key', value='message')
producer.flush()

# Consumer
consumer = Consumer({
    'bootstrap.servers': 'localhost:30092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,
})
consumer.subscribe(['my-topic'])
msg = consumer.poll(1.0)

See assets/​templates/​producer-consumer.py for async FastAPI integration.

---

Debugging

# Check consumer lag
kubectl exec -n kafka dev-cluster-dual-role-0 -- \
  bin/​kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group <group-name>

# List topics
kubectl exec -n kafka dev-cluster-dual-role-0 -- \
  bin/​kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describe topic
kubectl exec -n kafka dev-cluster-dual-role-0 -- \
  bin/​kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic <topic-name>

See references/​debugging-runbooks.md for detailed troubleshooting.

---

Delivery Semantics

Default: At-least-once with idempotent consumers.

---

File Structure

kafka/
├── Makefile                 # Tested deployment commands
├── manifests/
│   ├── kafka-cluster.yaml   # KRaft cluster (tested)
│   └── kafka-topic.yaml     # Topic CRD
├── assets/​templates/
│   └── producer-consumer.py # Python async templates
└── references/              # Deep knowledge
    ├── core-concepts.md
    ├── producers.md
    ├── consumers.md
    ├── debugging-runbooks.md
    ├── gotchas.md
    └── ... (15 files)

---

Architecture Analysis

When analyzing synchronous architectures for coupling:

Scenario: Service A calls B, C, D directly (500ms each)

Temporal Coupling?
└── Does caller wait for all responses? → YES = coupled

Availability Coupling?
└── If B is down, does A fail? → YES = coupled

Behavioral Coupling?
└── Does A import B, C, D clients? → YES = coupled

Solution: Publish domain event, services consume independently.

See references/​architecture-patterns.md for detailed analysis templates.

---

References

---

Related Skills