How to Start Building a Custom ETL in Scala

When Scala Makes Sense for ETL

Scala is most valuable when your ETL logic is complex enough that strong abstractions, reusable libraries, and compile-time checks save you real maintenance effort. It is especially attractive when your runtime is Apache Spark, because Spark's core APIs are native to the JVM and first-class in Scala.

Start with the Right Shape

Project Scaffold

A minimal Spark-based ETL project can start with build.sbt, a single main class, and a conventional src/main/scala layout. Replace the example Spark and Scala versions below so they match your target cluster runtime.

name := "custom-etl"

version := "0.1.0"

scalaVersion := "2.13.17"

val sparkVersion = "4.1.2" // Replace to match your target cluster

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.typesafe" % "config" % "1.4.3",
  "org.scalatest" %% "scalatest" % "3.2.19" % Test
)

A Minimal Job Skeleton

Even a small starter job should accept input and output locations as arguments, construct a SparkSession, apply a narrow transformation chain, and write to a deterministic target format such as Parquet.

package dev.ryware.etl

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.{col, to_timestamp}

object CustomEtlJob {
  def main(args: Array[String]): Unit = {
    val inputPath = args(0)
    val outputPath = args(1)

    val spark = SparkSession.builder()
      .appName("custom-etl-job")
      .getOrCreate()

    val curated = spark.read
      .option("header", "true")
      .csv(inputPath)
      .filter(col("transaction_id").isNotNull)
      .withColumn("amount", col("amount").cast("decimal(18,2)"))
      .withColumn("updated_at", to_timestamp(col("updated_at")))

    curated.write
      .mode("overwrite")
      .parquet(outputPath)

    spark.stop()
  }
}

Production Practices to Add Next