Apache Spark with Java & Spring Boot - Complete Tutorial

Overview

This project demonstrates Apache Spark concepts from basic to advanced, integrated with Spring Boot.

Project Structure

src/main/java/com/sparklearning/
├── basic/              # Basic Spark concepts
├── intermediate/       # Intermediate operations
├── advanced/          # Advanced features
├── config/            # Spring configuration
└── controller/        # REST endpoints

Prerequisites

• Java 17+
• Maven 3.6+
• Apache Spark 3.5.0

Getting Started

mvn clean install
mvn spring-boot:run

Learning Path

BASIC Level

1. RDD Operations (BasicRDDOperations.java)

   • Creating RDDs
   • Transformations (map, filter, flatMap)
   • Actions (collect, count, reduce)
   • Word Count example

2. DataFrame Operations (BasicDataFrameOperations.java)

   • Creating DataFrames
   • Select, filter, groupBy
   • SQL queries

INTERMEDIATE Level

3. Data Sources (DataSourceOperations.java)

   • Reading CSV, JSON, Parquet
   • JDBC connections
   • Writing data with partitioning

4. Advanced Transformations (IntermediateTransformations.java)

   • Window functions
   • Joins (inner, outer, left, right)
   • Aggregations
   • User Defined Functions (UDF)

ADVANCED Level

5. Performance Optimization (AdvancedOptimization.java)

   • Caching and persistence
   • Broadcast joins
   • Partitioning strategies
   • Adaptive Query Execution

6. Machine Learning (MachineLearningPipeline.java)

   • Feature engineering
   • Classification models
   • Cross-validation
   • Hyperparameter tuning

7. Structured Streaming (AdvancedStreaming.java)

   • Real-time data processing
   • Windowing operations
   • Stateful streaming
   • Stream-stream joins

API Endpoints

Basic Examples

GET  /api/spark/basic/transformations
GET  /api/spark/basic/actions
POST /api/spark/basic/wordcount
GET  /api/spark/basic/dataframe

Exercise Solutions

GET  /api/solutions/basic/exercise1
POST /api/solutions/basic/exercise2
GET  /api/solutions/basic/exercise3
GET  /api/solutions/intermediate/exercise4
GET  /api/solutions/intermediate/exercise5
GET  /api/solutions/intermediate/exercise6
GET  /api/solutions/advanced/exercise7
GET  /api/solutions/advanced/exercise9
GET  /api/solutions/projects/log-analytics
GET  /api/solutions/projects/recommendations
GET  /api/solutions/projects/etl-pipeline

Key Concepts

RDD vs DataFrame vs Dataset

• RDD: Low-level API, full control
• DataFrame: High-level API, optimized
• Dataset: Type-safe, best of both

Transformations vs Actions

• Transformations: Lazy (map, filter, join)
• Actions: Trigger execution (collect, count, save)

Performance Tips

1. Use DataFrames over RDDs
2. Cache frequently used data
3. Avoid shuffles when possible
4. Use broadcast for small tables
5. Partition data appropriately

Next Steps

1. Run basic examples
2. Experiment with your own data
3. Try streaming examples with Kafka
4. Build ML pipelines
5. Optimize for production

Resources

• Apache Spark Documentation
• Spark SQL Guide
• MLlib Guide ============================================= Project Structure Basic Level (Start here):

BasicRDDOperations.java - RDD fundamentals, transformations, actions, word count BasicDataFrameOperations.java - DataFrames, SQL queries, basic operations Intermediate Level:

DataSourceOperations.java - Reading/writing CSV, JSON, Parquet, JDBC IntermediateTransformations.java - Joins, window functions, aggregations, UDFs Advanced Level:

AdvancedOptimization.java - Caching, broadcast joins, partitioning, AQE MachineLearningPipeline.java - MLlib, feature engineering, model training AdvancedStreaming.java - Structured streaming, windowing, stateful operations Spring Boot Integration:

SparkConfig.java - Spark configuration beans SparkController.java - REST endpoints to test Spark operations application.yml - Application configuration Learning Resources README.md - Quick start guide and project overview LEARNING_GUIDE.md - Comprehensive concepts, best practices, interview questions EXERCISES.md - Hands-on exercises from basic to advanced To Get Started Build the project: mvn clean install Run: mvn spring-boot:run Test endpoints: http://localhost:8080/api/spark/basic/transformations Learning Path Start with RDD basics (transformations, actions) Move to DataFrames (more optimized) Learn data sources and I/O operations Master joins and aggregations Explore performance optimization Try machine learning pipelines Build streaming applications Each file is heavily commented with explanations. Work through them sequentially, and use the exercises to practice!

========================================================

Sample Data Files

The data/ directory contains sample CSV, JSON, and text files for all exercises:

Available Data Files

• employees.csv - Employee records with salary and department
• products.csv - Product catalog
• customers.json - Customer information
• orders.json - Order transactions
• sales.json - Sales data with dates
• departments.json - Department details
• people.json - People data for age categorization
• ratings.csv - User ratings for recommendations
• transactions.csv - Transaction data for fraud detection
• messy_data.csv - Intentionally messy data for cleaning
• ml_dataset.csv - Machine learning training data
• sample_text.txt - Text for word count exercises
• access.log - Apache web server logs
• small_lookup.csv - Small lookup table for broadcast joins
• large_dataset_sample.csv - Sample large dataset structure

Generate Custom Data

You can generate larger datasets for performance testing:

# Generate 1 million rows
curl -X POST "http://localhost:8080/api/data/generate/large?numRows=1000000"

# Generate time series data
curl -X POST "http://localhost:8080/api/data/generate/timeseries?numRows=10000"

# Generate transactions
curl -X POST "http://localhost:8080/api/data/generate/transactions?numRows=50000"

# Generate ML training data
curl -X POST "http://localhost:8080/api/data/generate/ml?numRows=10000"

# Generate web logs
curl -X POST "http://localhost:8080/api/data/generate/logs?numRows=100000"

Generated files will be saved in data/generated/ directory.

learn-apache-spark