Engineering Guide

So starten Sie ein benutzerdefiniertes ETL in Scala

Scala ist eine starke ETL-Wahl, wenn Sie Spark-native Entwicklung, Typsicherheit und einen Codepfad wollen, der von einem einzelnen Batch-Job zu einer größeren Datenplattform wachsen kann.

Custom ETL entwickeln ETL-Hub ansehen

Wann Scala für ETL sinnvoll ist

Scala ist besonders wertvoll, wenn ETL-Logik komplex genug ist, dass starke Abstraktionen, wiederverwendbare Bibliotheken und Compile-Time-Sicherheit echten Wartungsaufwand sparen. Vor allem mit Apache Spark passt Scala technisch sehr gut.

Verwenden Sie Scala, wenn sich die Pipeline wie Software und nicht wie eine Sammlung von Skripten verhalten soll.

What Spark, Hadoop, and HDFS Actually Are

These technologies are related, but they are not the same layer of the stack.

Spark

Spark is a distributed compute engine. It parallelizes reads, joins, aggregations, and writes across many workers. For ETL, think of Spark as the execution layer rather than the long-term storage layer.

Hadoop

Hadoop is a broader ecosystem that historically bundled distributed storage, cluster resource management, and batch processing. Modern teams often borrow concepts from Hadoop without running a classic Hadoop cluster end to end.

HDFS

HDFS is Hadoop Distributed File System. It stores files across many machines and was the standard storage layer for self-hosted big data clusters before object storage became the default in many cloud environments.

Catalog and table metadata

As soon as multiple jobs and analysts rely on the same curated datasets, you also need a catalog or metastore. It keeps schemas, partitions, and table definitions consistent across your ETL and query tools.

Should You Use HDFS?

Usually not as a default. HDFS still has a place, but many modern ETL stacks use object storage instead.

- Choose HDFS mainly when you operate your own cluster, want data locality, and already accept the operational cost of managing distributed storage nodes.
- Prefer S3, ADLS, or GCS when your compute can be ephemeral, your storage should scale independently, or you want simpler disaster recovery and cross-service integration.
- Do not adopt Hadoop just because you use Spark. Spark can run on Kubernetes, YARN, Databricks, EMR, Synapse, or other managed runtimes without HDFS being the center of the design.

Mit der richtigen Form starten

Kernschichten

- Konfiguration: Pfade, Secrets, Umgebungen, Zieltables
- Reader: JDBC, Dateien, Queues oder API-Adapter
- Transformationen: Normalisierung, Enrichment, Deduplizierung, Business Rules
- Validierung: Schema-, Count- und Domänenprüfungen
- Writer: Lake-, Warehouse- oder service-spezifische Ausgaben

Frühe Designziele

- Idempotente Loads ohne Duplikate bei Re-Runs
- Strukturiertes Logging mit Batch- oder Partition-Kontext
- Klare Fehlergrenzen zwischen Extract, Transform und Load
- Kleine testbare Transformationen statt Monolithen
- Ein Deployment-Modell, das von Anfang an zur Runtime passt

Do You Have to Use Spark?

No. Spark is powerful, but it is not the mandatory starting point for every ETL pipeline.

Use Spark when

You need distributed joins, large historical scans, partitioned batch outputs, or multi-hundred-gigabyte to multi-terabyte processing windows that no longer fit comfortably on one machine.

Skip Spark when

Your ETL is mostly SQL against Postgres, the data volume is still modest, one server can finish the workload inside the SLA, and operational simplicity matters more than horizontal scale.

Common alternatives

Start with Postgres SQL, dbt, Airflow plus Python, DuckDB, or Polars when the pipeline is small or medium. Move to Spark when the workload proves it needs a distributed compute engine.

Projektgerüst

Ein minimales Spark-ETL-Projekt kann mit build.sbt, einer Main-Klasse und einem konventionellen src/main/scala-Layout starten. Passen Sie die Beispielversionen an Ihre Zielumgebung an.

name := "custom-etl"

version := "0.1.0"

scalaVersion := "2.13.17"

val sparkVersion = "4.1.2" // Replace to match your target cluster

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.typesafe" % "config" % "1.4.3",
  "org.scalatest" %% "scalatest" % "3.2.19" % Test
)

Minimales Job-Skelett

Auch ein kleiner Starter-Job sollte Input- und Output-Pfade als Argumente annehmen, eine SparkSession erstellen, eine schlanke Transformationskette ausführen und in ein deterministisches Format wie Parquet schreiben.

package dev.ryware.etl

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.{col, to_timestamp}

object CustomEtlJob {
  def main(args: Array[String]): Unit = {
    val inputPath = args(0)
    val outputPath = args(1)

    val spark = SparkSession.builder()
      .appName("custom-etl-job")
      .getOrCreate()

    val curated = spark.read
      .option("header", "true")
      .csv(inputPath)
      .filter(col("transaction_id").isNotNull)
      .withColumn("amount", col("amount").cast("decimal(18,2)"))
      .withColumn("updated_at", to_timestamp(col("updated_at")))

    curated.write
      .mode("overwrite")
      .parquet(outputPath)

    spark.stop()
  }
}

When Postgres Is Enough and When to Move to a Warehouse

Raw database size matters, but query concurrency, data shape, retention, and the cost of long analytical scans matter more.

Approximate decision guide for analytics workloads. These are planning bands, not hard limits.
Workload stage	When Postgres is still reasonable	When you should plan the move	Typical target
Early analytics	Up to roughly 50-100 GB of curated analytical data, a few internal dashboards, light concurrency, and daily batch updates.	Stay put unless queries already compete with transactional traffic or refresh windows are missing the SLA.	Postgres plus SQL, dbt, and simple orchestration.
Growing reporting stack	Roughly 100-300 GB, moderate joins, fewer than 10-20 active analysts, and mostly structured tables.	Plan a warehouse when refresh jobs grow brittle, vacuum and index tuning turns into constant work, or semi-structured data starts piling up.	Postgres can still work, but start evaluating BigQuery, Snowflake, Redshift, Synapse, or a lakehouse pattern.
Heavy analytics	Possible only with careful tuning, but the tradeoff gets worse once scans span hundreds of gigabytes, concurrency climbs, and retention grows fast.	Move when BI workloads, historical backfills, and ML feature extraction all want the same data at the same time.	Columnar warehouse or lakehouse with separated storage and compute.
Platform scale	Rarely the right long-term shape once you are in multi-terabyte territory, ingesting many domains, or supporting self-service analytics.	At this point the question is usually not whether to move, but which warehouse, lakehouse, governance model, and cost controls fit the business.	Warehouse or lakehouse platform with catalog, partitioning, and workload isolation.

What If the Source Files Are XML and You Need XPath?

Spark can process XML, but the best approach depends on how irregular the documents are and how much XPath logic you need.

Spark can read XML

On open-source Spark, teams commonly add the spark-xml package and define an explicit row tag and schema. That works well when the XML documents are repetitive enough to map into tabular records.

XPath is possible, but expensive

XPath-style extraction is reasonable for a few known fields, but deeply nested or dynamic XPath rules can become CPU-heavy and awkward to test at scale. Flatten the documents as early as you can.

Sometimes pre-processing is better

If the XML is highly irregular, namespace-heavy, or validated against complex XSD rules, it can be cleaner to parse it first with a dedicated XML library in Scala or Java and then land normalized JSON or Parquet for downstream Spark work.

Operational advice

Keep the raw XML, version the mapping rules, capture malformed files separately, and build test fixtures from real samples. Special formats fail at the edges, so sample coverage matters more than happy-path demos.

Was als Nächstes in Produktion ergänzt werden sollte

Validierung und Qualität

- Schema-Prüfungen vor dem Schreiben der Curated Outputs
- Speicherung von Row Counts und Null-Raten je Lauf
- Fehlerhafte Partitionen ablehnen oder quarantänisieren statt stillschweigend zu korrigieren

Tests und Packaging

- Unit-Tests für Transformationen mit kleinen In-Memory-DataFrames
- Integrationstests für Source- und Sink-Adapter
- Packaging mit sbt und Ausführung via spark-submit oder Cluster-Launcher

Ein sinnvoller erster Meilenstein

Das erste Ziel sollte nicht die gesamte Datenplattform sein. Ziel ist: eine Quelle zuverlässig lesen, einen sauberen Transformationspfad bauen, einen kuratierten Output schreiben und den Lauf beobachtbar machen.

Meilenstein 1

Ein einzelner Batch-Job mit deterministischen Outputs.

Meilenstein 2

Konfigurationsgetriebene Umgebungen und Validierungsregeln.

Meilenstein 3

Orchestrierung, Alerts, Lineage und Qualitäts-Baselines.