Skip to content
expanso-kafka-to-s3 logo

Kafka to S3

expanso-kafka-to-s3

Stream Kafka topics to S3 with partitioning and batching

SKILL.md

Full skill instructions

Kafka to S3

Stream data from Kafka topics to S3 buckets with intelligent partitioning, batching, and compression.

Category

data-routing

Quick Start

# Configure environment
export KAFKA_BROKERS=localhost:9092
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export S3_BUCKET=your-bucket

# Run the pipeline
./​run.sh

Pipeline

The pipeline.yaml streams from Kafka to S3 with:

  • Consumer group management
  • Time-based partitioning (hourly/​daily)
  • Gzip compression
  • Batching for efficient S3 writes

Requirements

  • Expanso Edge installed (clawhub install expanso)
  • Kafka broker access
  • AWS credentials with S3 write permissions

Related

More Other skills

Delegate independent tasks to specialized agents working concurrently with isolated context.

295.4K 206.3K
View

Applies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.

179.7K 100.1K
View

Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors/fonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.

179.7K 89.5K
View

A set of resources to help me write all kinds of internal communications, using the formats that my company likes to use. Claude should use this skill whenever asked to write some sort of internal communications (status reports, leadership updates, 3P updates, company newsletters, FAQs, incident reports, project updates, etc.).

179.7K 76.1K
View

Fundamental design principles for writing maintainable, testable Python code.

40.2K 21.9K
View

Modern responsive layouts using container queries, fluid typography, CSS Grid, and mobile-first strategies.

40.2K 20K
View

Generate Playwright tests from scenarios using interactive browser exploration and validation.

39.7K 18.2K
View

Modern React state management with Redux Toolkit, Zustand, Jotai, and React Query for every state category.

40.2K 14.7K
View

GDPR-compliant data handling with consent management, data subject rights, and privacy controls.

40.2K 14.3K
View

Python error handling patterns including input validation, exception hierarchies, and partial failure handling. Use when implementing validation logic, designing exception strategies, handling batch processing failures, or building robust APIs.

40.2K 13.5K
View

Detailed multi-file refactor planning with safety checks and rollback guidance.

39.7K 13K
View