Kafka Infrastructure as Code (IaC) Deployment

Kafka Infrastructure as Code (IaC) Deployment

Expert guidance for deploying Apache Kafka using Terraform across multiple platforms.

When to Use This Skill

I activate when you need help with:

• Terraform deployments: "Deploy Kafka with Terraform", "provision Kafka cluster"
• Platform selection: "Should I use AWS MSK or self-hosted Kafka?", "compare Kafka platforms"
• Infrastructure planning: "How to size Kafka infrastructure", "Kafka on AWS vs Azure"
• IaC automation: "Automate Kafka deployment", "CI/​CD for Kafka infrastructure"

What I Know

Available Terraform Modules

This plugin provides 3 production-ready Terraform modules:

1. Apache Kafka (Self-Hosted, KRaft Mode)

• Location: plugins/​specweave-kafka/​terraform/​apache-kafka/
• Platform: AWS EC2 (can adapt to other clouds)
• Architecture: KRaft mode (no ZooKeeper dependency)
• Features:
  • Multi-broker cluster (3-5 brokers recommended)
  • Security groups with SASL_SSL
  • IAM roles for S3 backups
  • CloudWatch metrics and alarms
  • Auto-scaling group support
  • Custom VPC and subnet configuration
• Use When:
  • ✅ You need full control over Kafka configuration
  • ✅ Running Kafka 3.6+ (KRaft mode)
  • ✅ Want to avoid ZooKeeper operational overhead
  • ✅ Multi-cloud or hybrid deployments
• Variables:

  module "kafka" {
    source = "../​../​plugins/​specweave-kafka/​terraform/​apache-kafka"
  
    environment         = "production"
    broker_count        = 3
    kafka_version       = "3.7.0"
    instance_type       = "m5.xlarge"
    vpc_id              = var.vpc_id
    subnet_ids          = var.subnet_ids
    domain              = "example.com"
    enable_s3_backups   = true
    enable_monitoring   = true
  }
  

2. AWS MSK (Managed Streaming for Kafka)

• Location: plugins/​specweave-kafka/​terraform/​aws-msk/
• Platform: AWS Managed Service
• Features:
  • Fully managed Kafka service
  • IAM authentication + SASL/​SCRAM
  • Auto-scaling (provisioned throughput)
  • Built-in monitoring (CloudWatch)
  • Multi-AZ deployment
  • Encryption in transit and at rest
• Use When:
  • ✅ You want AWS to manage Kafka operations
  • ✅ Need tight AWS integration (IAM, KMS, CloudWatch)
  • ✅ Prefer operational simplicity over cost
  • ✅ Running in AWS VPC
• Variables:

  module "msk" {
    source = "../​../​plugins/​specweave-kafka/​terraform/​aws-msk"
  
    cluster_name           = "my-kafka-cluster"
    kafka_version          = "3.6.0"
    number_of_broker_nodes = 3
    broker_node_instance_type = "kafka.m5.large"
  
    vpc_id     = var.vpc_id
    subnet_ids = var.private_subnet_ids
  
    enable_iam_auth      = true
    enable_scram_auth    = false
    enable_auto_scaling  = true
  }
  

3. Azure Event Hubs (Kafka API)

• Location: plugins/​specweave-kafka/​terraform/​azure-event-hubs/
• Platform: Azure Managed Service
• Features:
  • Kafka 1.0+ protocol support
  • Auto-inflate (elastic scaling)
  • Premium SKU for high throughput
  • Zone redundancy
  • Private endpoints (VNet integration)
  • Event capture to Azure Storage
• Use When:
  • ✅ Running on Azure cloud
  • ✅ Need Kafka-compatible API without Kafka operations
  • ✅ Want serverless scaling (auto-inflate)
  • ✅ Integrating with Azure ecosystem
• Variables:

  module "event_hubs" {
    source = "../​../​plugins/​specweave-kafka/​terraform/​azure-event-hubs"
  
    namespace_name        = "my-event-hub-ns"
    resource_group_name   = var.resource_group_name
    location              = "eastus"
  
    sku                   = "Premium"
    capacity              = 1
    kafka_enabled         = true
    auto_inflate_enabled  = true
    maximum_throughput_units = 20
  }
  

Platform Selection Decision Tree

Need Kafka deployment? START HERE:

├─ Running on AWS?
│  ├─ YES → Want managed service?
│  │  ├─ YES → Use AWS MSK module (terraform/​aws-msk)
│  │  └─ NO → Use Apache Kafka module (terraform/​apache-kafka)
│  └─ NO → Continue...
│
├─ Running on Azure?
│  ├─ YES → Use Azure Event Hubs module (terraform/​azure-event-hubs)
│  └─ NO → Continue...
│
├─ Multi-cloud or hybrid?
│  └─ YES → Use Apache Kafka module (most portable)
│
├─ Need maximum control?
│  └─ YES → Use Apache Kafka module
│
└─ Default → Use Apache Kafka module (self-hosted, KRaft mode)

Deployment Workflows

Workflow 1: Deploy Self-Hosted Kafka (Apache Kafka Module)

Scenario: You want full control over Kafka on AWS EC2

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "kafka_cluster" {
  source = "../​../​plugins/​specweave-kafka/​terraform/​apache-kafka"

  environment         = "production"
  broker_count        = 3
  kafka_version       = "3.7.0"
  instance_type       = "m5.xlarge"

  vpc_id     = "vpc-12345678"
  subnet_ids = ["subnet-abc", "subnet-def", "subnet-ghi"]
  domain     = "kafka.example.com"

  enable_s3_backups = true
  enable_monitoring = true

  tags = {
    Project     = "MyApp"
    Environment = "Production"
  }
}

output "broker_endpoints" {
  value = module.kafka_cluster.broker_endpoints
}
EOF

# 2. Initialize Terraform
terraform init

# 3. Plan deployment (review what will be created)
terraform plan

# 4. Apply (create infrastructure)
terraform apply

# 5. Get broker endpoints
terraform output broker_endpoints
# Output: ["kafka-0.kafka.example.com:9093", "kafka-1.kafka.example.com:9093", ...]

Workflow 2: Deploy AWS MSK (Managed Service)

Scenario: You want AWS to manage Kafka operations

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "msk_cluster" {
  source = "../​../​plugins/​specweave-kafka/​terraform/​aws-msk"

  cluster_name           = "my-msk-cluster"
  kafka_version          = "3.6.0"
  number_of_broker_nodes = 3
  broker_node_instance_type = "kafka.m5.large"

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  enable_iam_auth     = true
  enable_auto_scaling = true

  tags = {
    Project = "MyApp"
  }
}

output "bootstrap_brokers" {
  value = module.msk_cluster.bootstrap_brokers_sasl_iam
}
EOF

# 2. Deploy
terraform init && terraform apply

# 3. Configure IAM authentication
# (module outputs IAM policy, attach to your application role)

Workflow 3: Deploy Azure Event Hubs (Kafka API)

Scenario: You're on Azure and want Kafka-compatible API

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "event_hubs" {
  source = "../​../​plugins/​specweave-kafka/​terraform/​azure-event-hubs"

  namespace_name      = "my-kafka-namespace"
  resource_group_name = "my-resource-group"
  location            = "eastus"

  sku                  = "Premium"
  capacity             = 1
  kafka_enabled        = true
  auto_inflate_enabled = true
  maximum_throughput_units = 20

  # Create hubs (topics) for your use case
  hubs = [
    { name = "user-events",    partitions = 12 },
    { name = "order-events",   partitions = 6 },
    { name = "payment-events", partitions = 3 }
  ]
}

output "connection_string" {
  value = module.event_hubs.connection_string
  sensitive = true
}
EOF

# 2. Deploy
terraform init && terraform apply

# 3. Get connection details
terraform output connection_string

Infrastructure Sizing Recommendations

Small Environment (Dev/​Test)

# Self-hosted: 1 broker, m5.large
broker_count  = 1
instance_type = "m5.large"

# AWS MSK: 1 broker per AZ, kafka.m5.large
number_of_broker_nodes = 3
broker_node_instance_type = "kafka.m5.large"

# Azure Event Hubs: Basic SKU
sku = "Basic"
capacity = 1

Medium Environment (Staging/​Production)

# Self-hosted: 3 brokers, m5.xlarge
broker_count  = 3
instance_type = "m5.xlarge"

# AWS MSK: 3 brokers, kafka.m5.xlarge
number_of_broker_nodes = 3
broker_node_instance_type = "kafka.m5.xlarge"

# Azure Event Hubs: Standard SKU with auto-inflate
sku = "Standard"
capacity = 2
auto_inflate_enabled = true
maximum_throughput_units = 10

Large Environment (High-Throughput Production)

# Self-hosted: 5+ brokers, m5.2xlarge or m5.4xlarge
broker_count  = 5
instance_type = "m5.2xlarge"

# AWS MSK: 6+ brokers, kafka.m5.2xlarge, auto-scaling
number_of_broker_nodes = 6
broker_node_instance_type = "kafka.m5.2xlarge"
enable_auto_scaling = true

# Azure Event Hubs: Premium SKU with zone redundancy
sku = "Premium"
capacity = 4
zone_redundant = true
maximum_throughput_units = 20

Best Practices

Security Best Practices

1. Always use encryption in transit

   • Self-hosted: Enable SASL_SSL listener
   • AWS MSK: Set encryption_in_transit_client_broker = "TLS"
   • Azure Event Hubs: HTTPS/​TLS enabled by default

2. Use IAM authentication (when possible)

   • AWS MSK: enable_iam_auth = true
   • Azure Event Hubs: Managed identities

3. Network isolation

   • Deploy in private subnets
   • Use security groups/​NSGs restrictively
   • Azure: Enable private endpoints for Premium SKU

High Availability Best Practices

1. Multi-AZ deployment

   • Self-hosted: Distribute brokers across 3+ AZs
   • AWS MSK: Automatically multi-AZ
   • Azure Event Hubs: Enable zone_redundant = true (Premium)

2. Replication factor = 3

   • Self-hosted: default.replication.factor=3
   • AWS MSK: Configured automatically
   • Azure Event Hubs: N/​A (fully managed)

3. min.insync.replicas = 2

   • Ensures durability even if 1 broker fails

Cost Optimization

1. Right-size instances

   • Use ClusterSizingCalculator utility (in kafka-architecture skill)
   • Start small, scale up based on metrics

2. Auto-scaling (where available)

   • AWS MSK: enable_auto_scaling = true
   • Azure Event Hubs: auto_inflate_enabled = true

3. Retention policies

   • Set log.retention.hours based on actual needs (default: 168 hours = 7 days)
   • Shorter retention = lower storage costs

Monitoring Integration

All modules integrate with monitoring:

Self-Hosted Kafka

• CloudWatch metrics (via JMX Exporter)
• Prometheus + Grafana dashboards (see kafka-observability skill)
• Custom CloudWatch alarms

AWS MSK

• Built-in CloudWatch metrics
• Enhanced monitoring available
• Integration with CloudWatch Alarms

Azure Event Hubs

• Built-in Azure Monitor metrics
• Diagnostic logs to Log Analytics
• Integration with Azure Alerts

Troubleshooting

"Terraform destroy fails on security groups"

Cause: Resources using security groups still exist Fix:

# 1. Find dependent resources
aws ec2 describe-network-interfaces --filters "Name=group-id,Values=sg-12345678"

# 2. Delete dependent resources first
# 3. Retry terraform destroy

"AWS MSK cluster takes 20+ minutes to create"

Cause: MSK provisioning is inherently slow (AWS behavior) Fix: This is normal. Use --auto-approve for automation:

terraform apply -auto-approve

"Azure Event Hubs: Connection refused"

Cause: Kafka protocol not enabled OR incorrect connection string Fix:

1. Verify kafka_enabled = true in Terraform
2. Use Kafka connection string (not Event Hubs connection string)
3. Check firewall rules (Premium SKU supports private endpoints)

Integration with Other Skills

• kafka-architecture: For cluster sizing and partitioning strategy
• kafka-observability: For Prometheus + Grafana setup after deployment
• kafka-kubernetes: For deploying Kafka on Kubernetes (alternative to Terraform)
• kafka-cli-tools: For testing deployed clusters with kcat

Quick Reference Commands

# Terraform workflow
terraform init          # Initialize modules
terraform plan          # Preview changes
terraform apply         # Create infrastructure
terraform output        # Get outputs (endpoints, etc.)
terraform destroy       # Delete infrastructure

# AWS MSK specific
aws kafka list-clusters # List MSK clusters
aws kafka describe-cluster --cluster-arn <arn> # Get cluster details

# Azure Event Hubs specific
az eventhubs namespace list # List namespaces
az eventhubs eventhub list --namespace-name <name> --resource-group <rg> # List hubs

---

Next Steps After Deployment:

1. Use kafka-observability skill to set up Prometheus + Grafana monitoring
2. Use kafka-cli-tools skill to test cluster with kcat
3. Deploy your producer/​consumer applications
4. Monitor cluster health and performance