Engineering Guide

How to Start Building a Custom ETL in Scala

Scala is a strong ETL choice when you want Spark-native development, type safety, and an implementation that can grow from a single batch job into a larger data platform.

Build a Custom ETL View ETL Hub

When Scala Makes Sense for ETL

Scala is most valuable when your ETL logic is complex enough that strong abstractions, reusable libraries, and compile-time checks save real maintenance effort. It is especially attractive when your runtime is Apache Spark, because Spark's core APIs are native to the JVM and first-class in Scala.

Use Scala when you expect the pipeline codebase to behave like software, not just like a collection of scripts.

What Spark, Hadoop, and HDFS Actually Are

These technologies are related, but they are not the same layer of the stack.

Spark

Spark is a distributed compute engine. It parallelizes reads, joins, aggregations, and writes across many workers. For ETL, think of Spark as the execution layer rather than the long-term storage layer.

Hadoop

Hadoop is a broader ecosystem that historically bundled distributed storage, cluster resource management, and batch processing. Modern teams often borrow concepts from Hadoop without running a classic Hadoop cluster end to end.

HDFS

HDFS is Hadoop Distributed File System. It stores files across many machines and was the standard storage layer for self-hosted big data clusters before object storage became the default in many cloud environments.

Catalog and table metadata

As soon as multiple jobs and analysts rely on the same curated datasets, you also need a catalog or metastore. It keeps schemas, partitions, and table definitions consistent across your ETL and query tools.

Should You Use HDFS?

Usually not as a default. HDFS still has a place, but many modern ETL stacks use object storage instead.

- Choose HDFS mainly when you operate your own cluster, want data locality, and already accept the operational cost of managing distributed storage nodes.
- Prefer S3, ADLS, or GCS when your compute can be ephemeral, your storage should scale independently, or you want simpler disaster recovery and cross-service integration.
- Do not adopt Hadoop just because you use Spark. Spark can run on Kubernetes, YARN, Databricks, EMR, Synapse, or other managed runtimes without HDFS being the center of the design.

Start with the Right Shape

Core layers

- Configuration: input paths, secrets, environment flags, target tables
- Readers: JDBC, files, message queues, or API ingestion adapters
- Transforms: normalization, enrichment, deduplication, and business rules
- Validators: schema checks, row-count checks, and domain assertions
- Writers: lake, warehouse, or service-specific output adapters

Early design goals

- Idempotent loads so reruns do not duplicate data
- Structured logging with batch or partition identifiers
- Clear failure boundaries between extract, transform, and load stages
- Small, testable transformations instead of giant monolithic jobs
- A deployment model that matches the runtime from day one

Do You Have to Use Spark?

No. Spark is powerful, but it is not the mandatory starting point for every ETL pipeline.

Use Spark when

You need distributed joins, large historical scans, partitioned batch outputs, or multi-hundred-gigabyte to multi-terabyte processing windows that no longer fit comfortably on one machine.

Skip Spark when

Your ETL is mostly SQL against Postgres, the data volume is still modest, one server can finish the workload inside the SLA, and operational simplicity matters more than horizontal scale.

Common alternatives

Start with Postgres SQL, dbt, Airflow plus Python, DuckDB, or Polars when the pipeline is small or medium. Move to Spark when the workload proves it needs a distributed compute engine.

Project Scaffold

A minimal Spark-based ETL project can start with build.sbt, a single main class, and a conventional src/main/scala layout. Replace the example Spark and Scala versions below so they match your target cluster runtime.

name := "custom-etl"

version := "0.1.0"

scalaVersion := "2.13.17"

val sparkVersion = "4.1.2" // Replace to match your target cluster

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.typesafe" % "config" % "1.4.3",
  "org.scalatest" %% "scalatest" % "3.2.19" % Test
)

A Minimal Job Skeleton

Even a small starter job should accept input and output locations as arguments, construct a SparkSession, apply a narrow transformation chain, and write to a deterministic target format such as Parquet.

package dev.ryware.etl

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.{col, to_timestamp}

object CustomEtlJob {
  def main(args: Array[String]): Unit = {
    val inputPath = args(0)
    val outputPath = args(1)

    val spark = SparkSession.builder()
      .appName("custom-etl-job")
      .getOrCreate()

    val curated = spark.read
      .option("header", "true")
      .csv(inputPath)
      .filter(col("transaction_id").isNotNull)
      .withColumn("amount", col("amount").cast("decimal(18,2)"))
      .withColumn("updated_at", to_timestamp(col("updated_at")))

    curated.write
      .mode("overwrite")
      .parquet(outputPath)

    spark.stop()
  }
}

When Postgres Is Enough and When to Move to a Warehouse

Raw database size matters, but query concurrency, data shape, retention, and the cost of long analytical scans matter more.

Approximate decision guide for analytics workloads. These are planning bands, not hard limits.
Workload stage	When Postgres is still reasonable	When you should plan the move	Typical target
Early analytics	Up to roughly 50-100 GB of curated analytical data, a few internal dashboards, light concurrency, and daily batch updates.	Stay put unless queries already compete with transactional traffic or refresh windows are missing the SLA.	Postgres plus SQL, dbt, and simple orchestration.
Growing reporting stack	Roughly 100-300 GB, moderate joins, fewer than 10-20 active analysts, and mostly structured tables.	Plan a warehouse when refresh jobs grow brittle, vacuum and index tuning turns into constant work, or semi-structured data starts piling up.	Postgres can still work, but start evaluating BigQuery, Snowflake, Redshift, Synapse, or a lakehouse pattern.
Heavy analytics	Possible only with careful tuning, but the tradeoff gets worse once scans span hundreds of gigabytes, concurrency climbs, and retention grows fast.	Move when BI workloads, historical backfills, and ML feature extraction all want the same data at the same time.	Columnar warehouse or lakehouse with separated storage and compute.
Platform scale	Rarely the right long-term shape once you are in multi-terabyte territory, ingesting many domains, or supporting self-service analytics.	At this point the question is usually not whether to move, but which warehouse, lakehouse, governance model, and cost controls fit the business.	Warehouse or lakehouse platform with catalog, partitioning, and workload isolation.

What If the Source Files Are XML and You Need XPath?

Spark can process XML, but the best approach depends on how irregular the documents are and how much XPath logic you need.

Spark can read XML

On open-source Spark, teams commonly add the spark-xml package and define an explicit row tag and schema. That works well when the XML documents are repetitive enough to map into tabular records.

XPath is possible, but expensive

XPath-style extraction is reasonable for a few known fields, but deeply nested or dynamic XPath rules can become CPU-heavy and awkward to test at scale. Flatten the documents as early as you can.

Sometimes pre-processing is better

If the XML is highly irregular, namespace-heavy, or validated against complex XSD rules, it can be cleaner to parse it first with a dedicated XML library in Scala or Java and then land normalized JSON or Parquet for downstream Spark work.

Operational advice

Keep the raw XML, version the mapping rules, capture malformed files separately, and build test fixtures from real samples. Special formats fail at the edges, so sample coverage matters more than happy-path demos.

Production Practices to Add Next

Validation and quality

- Add schema assertions before writing curated outputs
- Store row counts and null rates per run for anomaly detection
- Reject or quarantine malformed partitions instead of silently coercing everything

Testing and packaging

- Unit test transformation functions with small in-memory DataFrames
- Add integration tests for source and sink adapters
- Package with sbt and run with spark-submit or the cluster's native launcher

A Sensible First Milestone

Your first goal should not be to build the whole data platform. It should be to read one source reliably, apply one clean transformation path, write one curated output, and make the run observable.

Milestone 1

Single batch job with deterministic outputs.

Milestone 2

Configuration-driven environments and validation rules.

Milestone 3

Orchestration, alerts, lineage, and quality baselines.

Related ETL Guides

Continue with platform selection, anomaly detection, or AWS Glue-specific implementation options.

Back to ETL hub

Platform Comparison