Apache Spark

Apache Spark

Spark is the king of Big Data. v4.0 (2024/​2025) makes Spark Connect the default, allowing thin clients (like VS Code) to connect to massive clusters easily.

When to Use

• Data Engineering: ETL at Petabyte scale.
• Streaming: Structured Streaming for real-time analytics.
• Legacy ML: spark.ml (though mostly replaced by XGBoost/​Torch).

Core Concepts

Spark Connect

Decouples client (your laptop) from server (the cluster). Allows using Spark from Go/​Rust/​TypeScript.

Catalyst Optimizer

Optimizes your SQL/​DataFrame queries before execution.

RDD

The low-level API. Almost never used directly in modern Spark.

Best Practices (2025)

Do:

• Use PySpark: It is now a first-class citizen with Python UDF profiling.
• Use Delta Lake / Iceberg: Spark works best with modern table formats.
• Use pandas_udf: For vectorized Python UDFs.

Don't:

• Don't use rdd.map: It is slow (Python serialization). Use DataFrames.

References

• Apache Spark