Introduction
As datasets develop, processing them effectively on a single machine can turn into difficult because of limitations in CPU capability, reminiscence, and storage throughput.
Apache Spark addresses these challenges by offering a distributed information processing engine that may divide workloads into partitions and execute duties in parallel throughout CPU cores and a number of machines.
Nevertheless, Spark is just not merely about including extra machines.
Understanding how its parts work collectively—Drivers, Executors, RDDs, DataFrames, Partitions, Jobs, Levels, and Duties—is crucial to designing environment friendly and dependable information pipelines.
On this article, I’ll stroll by Spark’s structure, execution mechanics, and key efficiency issues utilizing a sensible e-commerce gross sales aggregation instance.
1. Why Apache Spark – Parallel Knowledge Processing
Apache Spark is a distributed computing engine designed for large-scale information processing and information engineering.
Spark processes huge information leveraging parallel information processing:
Determine A. Excessive-level Apache Spark distributed cluster structure (cluster, driver, and employee nodes), evaluating a single node execution (Created by Kuriko IWAI)
In contrast with a single node processing (proper, Determine A), Spark divides work into duties utilizing a number of nodes, executing them throughout sources, when obtainable, throughout machines.
Right here, distributed does not imply a number of machines are obligatory. Spark can run in native mode on a single machine. Even there, information may be partitioned and processed in parallel.
2. Core Apache Spark Structure
Spark consists of a number of key parts:
- Spark driver (driver): The central coordination technique of Spark software.
- Cluster supervisor: An exterior service answerable for allocating {hardware} sources like CPU cores and reminiscence.
- Employee nodes (employee): A bodily / digital machine or compute node within the cluster.
- Executors: Distributed employee processes launched on employee nodes.
- Java Digital Machine (JVM): The runtime surroundings required to execute Spark’s bytecode (since Spark is written in Scala/Java).
First, the driver coordinates a Spark software and rework it right into a directed acyclic graph (DAG) of bodily execution duties, and schedules them throughout the executors.
Then, cluster supervisor allocates sources below the deployment’s resource-management system.
Employee nodes provide compute sources to executors, which execute duties and maintain information throughout the job execution.
Some managed environments like Databricks use one executor per employee, however this isn’t a common Spark rule. A employee can host a number of executors relying on cluster configuration.
2-1. E-commerce Use Case – Enter Knowledge Construction
Allow us to think about a boutique e-commerce firm.
It has 10,000 gross sales data, every with a inventory retaining unit (SKU) and amount offered. Assume three doable SKUs: A, B, and C.
| SKU | Data | Amount | Anticipated Sum |
|---|---|---|---|
| A | 4,000 | 2 | 8,000 |
| B | 3,500 | 3 | 10,500 |
| C | 2,300 | 4 | 9,200 |
| Invalid / Returns | 200 | -1 | -200 |
| Grand Whole | 10,000 |
Desk 1. E-Commerce gross sales report breakdown & anticipated aggregations instance.
Our objective is to calculate whole legitimate items offered per SKU:
- Filter out data whose
qty , then - Combination samples by SKU.
Within the illustrative full dataset, suppose 9,800 of the ten,000 data stay after filtering. The ultimate totals anticipated from Spark are:
[("A", 8000), ("B", 10500), ("C", 9200)]
2-2. Spark Cluster
I’ll outline a simplified cluster for our use case.
Employee Nodes (Employees)
I’ll set 3 employee nodes, every of which has an occasion kind of 8 vCPU with 32 GiB RAM:
Determine B. Useful resource allocation throughout Employee Nodes (Created by Kuriko IWAI)
In whole, 24 vCPU can be found within the system.
Spark Executors and CPU/Reminiscence Allocation
Let’s say, I’ll safe:
- 7 executors.
- 2 vCPU cores for every executor.
- 3GiB JVM heap reminiscence for every executor.
Spark software requests the cluster supervisor to allocate sources to those 7 executors.
Then, the cluster supervisor decides CPUs and reminiscence allocations based mostly on the useful resource utilization of every employee.
As an example, it decides to create 14 duties slots – 2 vCPU core per executor x 7 executors:
Determine C-1. vCPU cores, reminiscence, and Executor allocation throughout Employees with useful resource imbalance in Employee A
Word that the three GiB reminiscence allotted to Executor is just not its whole reminiscence as a result of it has to safe reminiscence overhead for Employees and different executions.
Assuming {that a} single job requires 1 vCPU (spark.job.cpus = 1), every employee allocates its vCPU:
Determine C-2. Parallel vCPU allocation throughout Employees
This exhibits two key factors:
- Every employee has its distinctive CPU utilization price (In Determine C-2, Employee A’s utilization is 75%, whereas Employees B and C stay 50%).
- Regardless that the system has 24 vCPU cores (8 vCPU/employee x 3 Employees), it may solely use 14 vCPU cores; the remaining 10 stay unused for these duties.
2-3. Java Digital Machine (JVM) Behind Spark
Apache Spark is constructed on the Java Virtual Machine (JVM).
JVM is a runtime surroundings required to execute Spark’s bytecode.
Since Spark is written in Scala/Java:
- When utilizing Scala or Java in software: The code runs natively contained in the JVM.
- When utilizing PySpark (Python): A Python course of runs the code, but it surely communicates with a JVM-based driver by way of a bridge corresponding to Py4J. The Spark executors on the employee nodes nonetheless run inside a JVM, which then manages Python staff to execute Python code and Python User-Defined Functions (Python UDFs).
In both case, the JVM manages execution threads, reminiscence allocation, and rubbish assortment for each the motive force and executors.
Driver & Executor JVMs in PySpark Software
In a PySpark software, the Python driver communicates with the Spark driver operating within the JVM;
- Python driver: The place application-level PySpark code is written and executed.
- Driver JVM: Coordinates Spark execution and scheduling.
- Executor JVM: Runs Spark duties and manages execution sources. Python employee processes are concerned when duties require Python execution.
- Shopper JVM (shopper course of or submission machine): JVM operating on the native machine or an edge/gateway node.
As proven within the determine, the shopper JVM first connects to the driver JVM, which subsequently coordinates with the executor JVMs:
Determine D. General relation of shopper, driver, and executor JVMs
From right here, the motive force JVM hosts the Spark Driver to handle the applying, whereas the executor JVMs host the person executors answerable for executing these duties.
Every JVM driver has JVM heap to deal with information processing, caching, and person objects effectively.
2-4. Resilient Distributed Datasets (RDDs) Deep Dive
A Resilient Distributed Dataset (RDD) is an immutable, partitioned assortment of parts that Spark can course of in parallel.
RDDs monitor dependencies between transformations, permitting Spark to recompute misplaced partitions in lots of failure eventualities.
The identify captures three properties:
- Resilient: Spark data lineage—which RDD is determined by which earlier RDD—and may usually reconstruct lacking partitions as a substitute of sustaining replicas of each intermediate outcome.
- Distributed: The dataset is represented as partitions that Spark might course of on totally different executors. A number of bodily machines are not required; native mode additionally helps RDDs.
- Dataset: An RDD can comprise key-value pairs, integers, strings, tuples, and different supported parts—not solely relational rows. RDD APIs embrace operations corresponding to
map,filter,reduceByKey,be part of, andaggregateByKey.
An RDD is a logical assortment and computation abstraction, not a declaration that exact partitions are completely assigned to specific staff.
Partitions
A partition is a logical chunk of information and usually corresponds to 1 job inside a stage.
Let’s say we create 100 partitions (These 100 preliminary partitions are an express instructing alternative).
With 10,000 supply parts, it offers roughly 100 parts per partition if the enter is distributed roughly evenly:
Determine E. Parallel partition allocation throughout 100 preliminary information splits
3. RDD API vs DataFrame API: Inner Catalyst Optimization
The unique RDD instance helps perceive lineage, partitions, and shuffle.
In typical structured ETL, nevertheless, the DataFrame API is the popular start line as a result of it offers columns, schema info, SQL-like expressions, and optimizer-visible operations.
Beneath diagram illustrates the connection:
Determine F. How Spark’s Graph ideas relate; RDD linage, DataFrame, and Spark Execution DAG
Identical to RDD has RDD lineage to outline the transformation, DataFrame question plan represents its structured question.
Spark constructs execution work from dependencies and bodily operators.
DataFrame execution might use RDD-based mechanisms internally, however that does not imply customers must manually construct RDD lineage when writing DataFrame code.
3-1. DataFrame Question Plan
A DataFrame question plan is the sequence of logical and bodily steps an execution engine generates (purple field in Determine L).
A DataFrame question strikes by a number of representations:
- Logical plan: An outline of the requested operations (scan, filter, group, mixture).
- Analyzed and optimized logical plan: Spark resolves references and applies relevant logical optimizations, corresponding to column pruning or predicate pushdown.
- Bodily plan: Spark chooses executable operators, e.g. scans, partial/remaining aggregates, exchanges, joins, and kinds.
- Execution: The bodily plan is run utilizing Spark’s execution mechanisms, in the end involving duties on executors.
In our use case, the bodily plan can embrace the next steps:
- Partial hash aggregation,
- Change by SKU, and
- Remaining hash aggregation.
3-2. Evaluating Actions: RDD API vs DataFrame API
Allow us to evaluate RDD API and DataFrame API, taking the SKU aggregation for an instance.
RDD API
The weather are (sku, qty) pairs. Our Python aggregation logic is express, and Spark follows RDD dependencies:
result_rdd = (
raw_rdd
.filter(lambda row: row[1] > 0)
.reduceByKey(lambda a, b: a + b, numPartitions=4)
)
rdd_results = result_rdd.acquire()
# Instance:
# [('A', 8000), ('B', 10500), ('C', 9200)]
DataFrame API
Assume sales_df has columns sku (string) and qty (numeric):
from pyspark.sql import capabilities as F
result_df = (
sales_df
.filter(F.col("qty") > 0)
.groupBy("sku")
.agg(F.sum("qty").alias("total_qty"))
)
df_results = result_df.acquire()
# Instance:
# [Row(sku='A', total_qty=8000),
# Row(sku='B', total_qty=10500),
# Row(sku='C', total_qty=9200)]
Right here, each examples resolve the identical enterprise drawback:
Filtering invalid gross sales data and aggregating legitimate portions by SKU.
And in each circumstances, acquire() triggers Spark execution and returns all ensuing data to the Driver.
The important thing distinction right here is within the outcome illustration:
- RDD API: Returns Python parts, corresponding to
(sku, total_qty)tuples. - DataFrame API: Returns structured
Rowobjects with named fields, corresponding toskuandtotal_qty.
Right here’s an total comparability of RDD and DataFrame APIs:
| RDD API | DataFrame / SQL API | |
|---|---|---|
| Knowledge Illustration | Common parts, together with arbitrary supported objects | Rows with named columns and schema |
| Programming Model | Decrease-level transformations | Declarative relational transformations |
| Aggregations | Sure: reduceByKey, aggregateByKey, and so forth. |
Sure: groupBy, agg, SQL |
| Question Optimization | Much less relational semantic info uncovered | Catalyst and bodily execution optimization |
| Typical Use | Specialised low-level processing / instructing execution mechanics | Structured ETL, analytics, characteristic engineering |
| Execution | Spark jobs, levels, duties | Spark jobs, levels, duties |
Desk 2. RDD API vs. DataFrame API – characteristic & abstraction comparability:(Created by Kuriko IWAI)
4. Wrapping Up
The objective of Spark tuning is to not maximize partition rely or CPU utilization independently.
It’s to cut back end-to-end execution value and latency with out introducing instability.
On this article, we be taught the core relationships in its system:
- Sources: Employees host executors; executors run duties; configured CPU and reminiscence restrict concurrency.
- RDD programming: Transformations produce immutable RDD definitions and lineage; actions set off computation.
- DataFrame programming: Transformations produce logical question operations; optimization chooses executable bodily operators; actions set off computation.
- Execution: Spark organizes work into jobs, levels, and duties, and duties course of partitions on executors. Shuffles and different dependencies create coordination boundaries amongst levels.
Understanding how Spark executes distributed workloads is simply the inspiration.
You’ll be able to be taught a seven-step computation flow using RDD API.

