gradle-build

Licensed to the Apache Software Foundation (ASF) under one

or more contributor license agreements. See the NOTICE file

distributed with this work for additional information

regarding copyright ownership. The ASF licenses this file

to you under the Apache License, Version 2.0 (the

"License"); you may not use this file except in compliance

with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,

software distributed under the License is distributed on an

"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

KIND, either express or implied. See the License for the

specific language governing permissions and limitations

under the License.

name: gradle-build description: Guides understanding and using the Gradle build system in Apache Beam. Use when building projects, understanding dependencies, or troubleshooting build issues.

Gradle Build System in Apache Beam

Overview

Apache Beam is a mono-repo using Gradle as its build system. The entire project (Java, Python, Go, website) is managed as a single Gradle project.

Key Files

• build.gradle.kts - Root build configuration
• settings.gradle.kts - Project structure and module definitions
• gradle.properties - Global properties and versions
• buildSrc/ - Custom Gradle plugins including BeamModulePlugin

BeamModulePlugin

Located at buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Purpose

• Manages Java dependencies centrally
• Configures project types (Java, Python, Go, Proto, Docker, etc.)
• Defines common custom tasks

Java Project Configuration

apply plugin: 'org.apache.beam.module'
applyJavaNature(
    automaticModuleName: 'org.apache.beam.sdk.io.kafka'
)

Common Commands

Build

# Build entire project
./gradlew build

# Build specific project
./gradlew :sdks:java:core:build
./gradlew -p sdks/java/core build

# Compile only (no tests)
./gradlew :sdks:java:core:compileJava

Test

# Run tests
./gradlew :sdks:java:core:test

# Run specific test
./gradlew :sdks:java:core:test --tests *MyTest

# Skip tests
./gradlew build -x test

Clean

# Clean specific project
./gradlew :sdks:java:core:clean

# Clean everything
./gradlew clean

Formatting

# Java formatting (Spotless)
./gradlew spotlessApply

# Check formatting
./gradlew spotlessCheck

# Format CHANGES.md
./gradlew formatChanges

Publishing

# Publish to Maven Local
./gradlew -Ppublishing :sdks:java:core:publishToMavenLocal

# Publish all Java artifacts
./gradlew -Ppublishing publishToMavenLocal

Pre-commit Tasks

Java

./gradlew javaPreCommit

Python

./gradlew pythonPreCommit

Combined

./gradlew :checkSetup  # Validates Go, Java, Python environments

Useful Flags

GCP-related Properties

-PgcpProject=my-project
-PgcpRegion=us-central1
-PgcpTempRoot=gs://bucket/temp
-PgcsTempRoot=gs://bucket/temp

Docker Tasks

# Build Java SDK container
./gradlew :sdks:java:container:java11:docker

# Build Python SDK container
./gradlew :sdks:python:container:py312:docker

# Build and push a container into a custom repository
./gradlew :sdks:java:container:java11:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers

If a :docker task produces logs that contain the following:

WARNING: No output specified with docker-container driver.
Build result will only remain in the build cache.

then you must use -PuseDockerBuildx when running :docker tasks in this environment. For example:

# Build and push a go container into a custom repository
./gradlew :sdks:go:container:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers \
  -PuseDockerBuildx

Dependency Management

View Dependencies

./gradlew :sdks:java:core:dependencies
./gradlew :sdks:java:core:dependencies --configuration runtimeClasspath

Force Dependency Version

In build.gradle:

configurations.all {
    resolutionStrategy.force 'com.google.guava:guava:32.0.0-jre'
}

Troubleshooting

Clean Gradle Cache

rm -rf ~/.gradle/caches
rm -rf .gradle
rm -rf build

Common Errors

NoClassDefFoundError

• Run ./gradlew clean
• Delete gradle cache

Proto-related Errors

• Regenerate protos: ./gradlew generateProtos

Dependency Conflicts

• Check dependencies: ./gradlew dependencies
• Use --scan for detailed analysis

Useful Tasks

# List all tasks
./gradlew tasks

# List tasks for a project
./gradlew :sdks:java:core:tasks

# Show project structure
./gradlew projects

IDE Integration

IntelliJ

1. Open repository root as Gradle project
2. Wait for indexing
3. Gradle tool window shows all tasks

VS Code

Install Gradle extension for task discovery