Skip to content
LiveNew tools this week
Submit
AIMarketly
gradle-build logo

gradle-build

Guides understanding and using the Gradle build system in Apache Beam. Use when building projects, understanding dependencies, or troubleshooting build issues.

NeverSight/skills_feed0installs213stars

SKILL.md

Full skill instructions

Gradle Build System in Apache Beam

Overview

Apache Beam is a mono-repo using Gradle as its build system. The entire project (Java, Python, Go, website) is managed as a single Gradle project.

Key Files

  • build.gradle.kts - Root build configuration
  • settings.gradle.kts - Project structure and module definitions
  • gradle.properties - Global properties and versions
  • buildSrc/ - Custom Gradle plugins including BeamModulePlugin

BeamModulePlugin

Located at buildSrc/​src/​main/​groovy/​org/​apache/​beam/​gradle/​BeamModulePlugin.groovy

Purpose

  • Manages Java dependencies centrally
  • Configures project types (Java, Python, Go, Proto, Docker, etc.)
  • Defines common custom tasks

Java Project Configuration

apply plugin: 'org.apache.beam.module'
applyJavaNature(
    automaticModuleName: 'org.apache.beam.sdk.io.kafka'
)

Common Commands

Build

# Build entire project
./​gradlew build

# Build specific project
./​gradlew :sdks:java:core:build
./​gradlew -p sdks/​java/​core build

# Compile only (no tests)
./​gradlew :sdks:java:core:compileJava

Test

# Run tests
./​gradlew :sdks:java:core:test

# Run specific test
./​gradlew :sdks:java:core:test --tests *MyTest

# Skip tests
./​gradlew build -x test

Clean

# Clean specific project
./​gradlew :sdks:java:core:clean

# Clean everything
./​gradlew clean

Formatting

# Java formatting (Spotless)
./​gradlew spotlessApply

# Check formatting
./​gradlew spotlessCheck

# Format CHANGES.md
./​gradlew formatChanges

Publishing

# Publish to Maven Local
./​gradlew -Ppublishing :sdks:java:core:publishToMavenLocal

# Publish all Java artifacts
./​gradlew -Ppublishing publishToMavenLocal

Pre-commit Tasks

Java

./​gradlew javaPreCommit

Python

./​gradlew pythonPreCommit

Combined

./​gradlew :checkSetup  # Validates Go, Java, Python environments

Useful Flags

FlagDescription
-p <path>Run task in specific project directory
-x <task>Exclude task
--tests <pattern>Filter tests
-PpublishingEnable publishing tasks
-PdisableSpotlessCheck=trueDisable formatting check
-PdisableCheckStyle=trueDisable checkstyle
-PskipCheckerFrameworkSkip Checker Framework
--continueContinue after failures
--infoVerbose output
--debugDebug output
--scanGenerate build scan
--parallelParallel execution

GCP-related Properties

-PgcpProject=my-project
-PgcpRegion=us-central1
-PgcpTempRoot=gs://bucket/​temp
-PgcsTempRoot=gs://bucket/​temp

Docker Tasks

# Build Java SDK container
./​gradlew :sdks:java:container:java11:docker

# Build Python SDK container
./​gradlew :sdks:python:container:py39:docker

# With custom repository
./​gradlew :sdks:java:container:java11:docker \
  -Pdocker-repository-root=gcr.io/​project \
  -Pdocker-tag=custom

Dependency Management

View Dependencies

./​gradlew :sdks:java:core:dependencies
./​gradlew :sdks:java:core:dependencies --configuration runtimeClasspath

Force Dependency Version

In build.gradle:

configurations.all {
    resolutionStrategy.force 'com.google.guava:guava:32.0.0-jre'
}

Troubleshooting

Clean Gradle Cache

rm -rf ~/​.gradle/​caches
rm -rf .gradle
rm -rf build

Common Errors

NoClassDefFoundError
  • Run ./​gradlew clean
  • Delete gradle cache
Proto-related Errors
  • Regenerate protos: ./​gradlew generateProtos
Dependency Conflicts
  • Check dependencies: ./​gradlew dependencies
  • Use --scan for detailed analysis

Useful Tasks

# List all tasks
./​gradlew tasks

# List tasks for a project
./​gradlew :sdks:java:core:tasks

# Show project structure
./​gradlew projects

IDE Integration

IntelliJ

  1. Open repository root as Gradle project
  2. Wait for indexing
  3. Gradle tool window shows all tasks

VS Code

Install Gradle extension for task discovery