Audience
Developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner.
Description
The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataSets/DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL, DataFrames, and DataSets, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization. The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data and integrating with the Kafka server. Labs are supported in both Python and Scala
Objectives
Upon successful completion of this course, the student will be able to:
- Understand the need for Spark in data processing
- Understand the Spark architecture and how it distributes computations to cluster nodes
- Be familiar with basic installation/setup / layout of Spark
- Use the Spark for interactive and ad-hoc operations
- Use Dataset/DataFrame/Spark SQL to efficiently process structured data
- Understand basics of RDDs (Resilient Distributed Datasets), and data partitioning, pipelining, and computations
- Understand Spark's data caching and its usage
- Understand performance implications and optimizations when using Spark
- Be familiar with Spark Graph Processing and SparkML machine learning
Prerequisites
Students should have an introductory knowledge of Python or Scala. An overview of Scala is provided if needed.