In this PySpark course, you will discover how to utilize Spark from Python. This PySpark course is created to help you master skills that are required to become a successful Spark developer using Python. Python Spark Training Course is designed to provide you with the knowledge and skills to become a successful Big Data & Spark Developer. This Training would help you to clear the CCA Spark and Hadoop Developer (CCA175) Examination. You will understand the basics of Big Data and Hadoop. You will learn how Spark enables in-memory data processing and runs much faster than Hadoop MapReduce. You will also learn about RDDs, Spark SQL for structured processing, different APIs offered by Spark such as Spark Streaming, Spark MLlib. This course is an integral part of a Big Data Developer’s Career path. It will also encompass the fundamental concepts such as data capturing using Flume, data loading using Sqoop, a messaging system like Kafka, etc. The training will show you how to build and implement data-intensive applications after you know about machine learning, leveraging Spark RDD, Spark SQL, Spark MLlib, Spark Streaming, HDFS, Flume, Spark GraphX, and Kafka.

Course Objectives:

After completing this course attendees should be able to:

• Overview of Big Data & Hadoop including HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator)
• Comprehensive knowledge of various tools that falls in Spark Ecosystem like Spark SQL, Spark MlLib, Sqoop, Kafka, Flume and Spark Streaming
• The capability to ingest data in HDFS using Sqoop & Flume, and analyse those large datasets stored in the HDFS
• The power of handling real-time data feeds through a publish-subscribe messaging system like Kafka
• The exposure to many real-life industry-based projects which will be executed
• Projects which are diverse in nature covering banking, telecommunication, social media, and government domains
• Rigorous involvement of an SME throughout the Spark Training to learn industry standards and best practices

Course content

Introduction to Big Data Hadoop and Spark

• What is Big Data?
• Big Data Customer Scenarios
• Limitations and Solutions of Existing Data Analytics Architecture with Uber Use Case
• How Hadoop Solves the Big Data Problem?
• What is Hadoop?
• Hadoop’s Key Characteristics
• Hadoop Ecosystem and HDFS
• Hadoop Core Components
• Rack Awareness and Block Replication
• YARN and its Advantage
• Hadoop Cluster and its Architecture
• Hadoop: Different Cluster Modes
• Big Data Analytics with Batch & Real-Time Processing
• Why Spark is Needed?
• What is Spark?
• How Spark Differs from its Competitors?
• Spark at eBay
• Spark’s Place in Hadoop Ecosystem

Introduction to Python for Apache Spark

• Overview of Python
• Different Applications where Python is Used
• Values, Types, Variables
• Operands and Expressions
• Conditional Statements
• Loops
• Command Line Arguments
• Writing to the Screen
• Python files I/O Functions
• Numbers
• Strings and related operations
• Tuples and related operations
• Lists and related operations
• Dictionaries and related operations
• Sets and related operations

Functions, OOPs, and Modules in Python

• Functions
• Function Parameters
• Global Variables
• Variable Scope and Returning Values
• Lambda Functions
• Object-Oriented Concepts
• Standard Libraries
• Modules Used in Python
• The Import Statements
• Module Search Path
• Package Installation Ways

Deep Dive into Apache Spark Framework

• Spark Components & its Architecture
• Spark Deployment Modes
• Introduction to PySpark Shell
• Submitting PySpark Job
• Spark Web UI
• Writing your first PySpark Job Using Jupyter Notebook
• Data Ingestion using Sqoop

Playing with Spark RDDs

• Challenges in Existing Computing Methods
• Probable Solution & How RDD Solves the Problem
• What is RDD, It’s Operations, Transformations & Actions
• Data Loading and Saving Through RDDs
• Key-Value Pair RDDs
• Other Pair RDDs, Two Pair RDDs
• RDD Lineage
• RDD Persistence
• WordCount Program Using RDD Concepts
• RDD Partitioning & How it Helps Achieve Parallelization
• Passing Functions to Spark

DataFrames and Spark SQL

• Need for Spark SQL
• What is Spark SQL
• Spark SQL Architecture
• SQL Context in Spark SQL
• Schema RDDs
• User Defined Functions
• Data Frames & Datasets
• Interoperating with RDDs
• JSON and Parquet File Formats
• Loading Data through Different Sources
• Spark-Hive Integration

Machine Learning using Spark MLlib

• Why Machine Learning
• What is Machine Learning
• Where Machine Learning is used
• Face Detection: USE CASE
• Different Types of Machine Learning Techniques
• Introduction to MLlib
• Features of MLlib and MLlib Tools
• Various ML algorithms supported by MLlib

Deep Dive into Spark MLlib

• Supervised Learning: Linear Regression, Logistic Regression, Decision Tree, Random Forest
• Unsupervised Learning: K-Means Clustering & How It Works with MLlib
• Analysis of US Election Data using MLlib (K-Means)

Understanding Apache Kafka and Apache Flume

• Need for Kafka
• What is Kafka
• Core Concepts of Kafka
• Kafka Architecture
• Where is Kafka Used
• Understanding the Components of Kafka Cluster
• Configuring Kafka Cluster
• Kafka Producer and Consumer Java API
• Need of Apache Flume
• What is Apache Flume
• Basic Flume Architecture
• Flume Sources
• Flume Sinks
• Flume Channels
• Flume Configuration
• Integrating Apache Flume and Apache Kafka

Apache Spark Streaming – Processing Multiple Batches

• Drawbacks in Existing Computing Methods
• Why Streaming is Necessary
• What is Spark Streaming
• Spark Streaming Features
• Spark Streaming Workflow
• How Uber Uses Streaming Data
• Streaming Context & DStreams
• Transformations on DStreams
• Describe Windowed Operators and Why it is Useful
• Important Windowed Operators
• Slice, Window and ReduceByWindow Operators
• Stateful Operators

Apache Spark Streaming – Data Sources

• Apache Spark Streaming: Data Sources
• Streaming Data Source Overview
• Apache Flume and Apache Kafka Data Sources
• Example: Using a Kafka Direct Data Source

Spark GraphX

• Introduction to Spark GraphX
• Information about a Graph
• GraphX Basic APIs and Operations
• Spark GraphX Algorithm – PageRank, Personalized PageRank, Triangle Count, Shortest Paths, Connected Components, Strongly Connected Components, Label Propagation

To see the full course content Download now