Big Data Engineer Master’s Program makes you proficient in tools and systems used by Big Data experts. Big Data Master’s Program to professionals who seek to dependant on their knowledge in the field of Big Data. This Big Data Engineer Course  includes training on Hadoop and Spark stack, Cassandra, Talend and Apache Kafka messaging system. DASVM's comprehensive Big Data Engineer training course is designed by Industry top experts with real time project experience. It makes you proficient in tools and systems used by Big Data experts. It includes training on Hadoop and Spark, Java Essentials, and SQL. The program is customized based on current industry standards that comprise of major sub-modules as a part of the training process. This program is designed by the industry experts to provide hands-on training with tools that are used to speed up the training process.

Learning Path:

1. Java Essentials
2. Big Data Hadoop
3. Apache Cassandra
4. Talend for Data Integration and Big Data
5. Apache Spark and Scala
6. Apache Kafka

Course content

Java Essentials

Introduction to Java

• Introduction to Java
• Class Files
• Data types and Operations
• Loops – for, while & do while
• Bytecode Compilation Process If Conditions

Data Handling and Functions

• Arrays – Single Dimensional and Multidimensional arrays
• Functions with Arguments
• Concept of Static Polymorphism
• String buffer Classes
• Functions
• Function Overloading
• If Conditions
• String Handling -String

Object Oriented Programming in Java

• Concept of Object Orientation
• Classes and Objects
• Methods and Constructors:
• Default Constructors Inheritance
• Final and Static
• Attributes and Methods
• Constructors with Arguments
• Abstract

Data Handling and Functions

• Packages and Interfaces
• Package
• Multi-threading
• Access Specifiers
• Exception Handling

Collections

• Wrapper Classes and Inner Classes: Integer, Character, Boolean, Float etc.
• Applet Programs: How to write UI programs with Applet, Java.lang, Java.io, Java.util.
• Collections: ArrayList, Vector, HashSet, TreeSet, HashMap, HashTable.

Big Data & Hadoop

Understanding Big Data and Hadoop

• Introduction to Big Data & Big Data Challenges
• Limitations & Solutions of Big Data Architecture
• Data types and Operations
• Hadoop Processing: MapReduce Framework
• Hadoop Storage: HDFS (Hadoop Distributed File System)
• Hadoop & its Features
• Hadoop Ecosystem
• Hadoop 2.x Core Components
• Different Hadoop Distributions

Hadoop Architecture and HDFS

• Hadoop 2.x Cluster Architecture
• Federation and High Availability Architecture
• Typical Production Hadoop Cluster
• Common Hadoop Shell Commands
• Hadoop Cluster Modes
• Hadoop 2.x Configuration Files
• Single Node Cluster & Multi-Node Cluster set up
• Basic Hadoop Administration

Hadoop MapReduce Framework

• Traditional way vs MapReduce way
• YARN Components
• YARN Architecture
• YARN Workflow
• YARN MapReduce Application Execution Flow
• Anatomy of MapReduce Program
• Why MapReduce
• MapReduce : Combiner & Partitioner
• Input Splits, Relation between Input Splits and HDFS Blocks
• Demo of Health Care Dataset
• Demo of Weather Dataset

Advanced Hadoop MapReduce

• Counters
• MRunit
• Custom Input Format
• XML file Parsing using MapReduce
• Distributed Cache
• Reduce Join
• Sequence Input Format

Apache Pig

• Introduction to Apache Pig
• Pig Components & Pig Execution
• Pig Data Types & Data Models in Pig
• Pig UDF & Pig Streaming
• Aviation use-case in PIG
• MapReduce vs Pig
• Pig Latin Programs
• Shell and Utility Commands
• Testing Pig scripts with Punit
• Pig Demo of Healthcare Dataset

Apache Hive

• Introduction to Apache Hive
• Hive Architecture and Components
• Limitations of Hive
• Comparison with Traditional Database
• Hive Tables (Managed Tables & External Tables)
• Importing Data
• Querying Data & Managing Outputs
• Hive Demo on Healthcare Dataset
• Hive vs Pig
• Hive Metastore
• Hive Partition
• Hive Data Types and Data Models
• Hive Bucketing
• Hive Script & Hive UDF
• Retail use case in Hive

Advanced Apache Hive and HBase

• Hive QL: Joining Tables, Dynamic Partitioning
• Hive Indexes and views
• Hive Thrift Server
• HBase v/s RDBMS
• HBase Architecture
• HBase Configuration
• Apache HBase: Introduction to NoSQL Databases and HBase
• Custom MapReduce Scripts
• Hive Query Optimizers
• Hive UDF
• HBase Components
• HBase Run Modes
• HBase Cluster Deployment

Advanced Apache HBase

• HBase Data Model
• HBase Client API
• Apache Zookeeper Introduction
• Zookeeper Service
• Getting and Inserting Data
• Hive Data Loading Techniques
• ZooKeeper Data Model
• HBase Bulk Loading
• HBase Filters

Processing Distributed Data with Apache Spark

• What is Spark
• Spark Components
• Why Scala
• Spark RDD
• Spark Ecosystem
• What is Scala
• SparkContext

Oozie and Hadoop Project

• Oozie
• Oozie Workflow
• Scheduling Jobs with Oozie Scheduler
• Oozie Commands
• Oozie for MapReduce
• Combining flow of MapReduce Jobs
• Hadoop Talend Integration
• Oozie Components
• Demo of Oozie Workflow
• Oozie Coordinator
• Oozie Web Console
• Hive in Oozie

Apache Cassandra

Introduction to Big Data and Cassandra

• Introduction to Big Data and Problems caused by it
• 5V – Volume, Variety, Velocity, Veracity and Value
• Traditional Database Management System
• Limitations of RDMS
• NOSQL databases
• Common characteristics of NoSQL databases
• CAP theorem
• How Cassandra solves the Limitations?
• History of Cassandra
• Features of Cassandra

Cassandra Data Model

• Introduction to Database Model
• Understand the analogy between RDBMS and Cassandra Data Model
• Understand following Database Elements: Cluster, Keyspace, Column Family/Table, Cloumn Column Family Options
• Columns
• Wide Rows, Skinny Rows
• Static and dynamic tables

Cassandra Architecture

• Cassandra as a Distributed Database
• Replication Factor
• Data Replication in Cassandra
• Gossip protocol – Detecting failures
• Staged Event-Driven Architecture (SEDA)
• Managers and Services
• Consistency level
• Repair
• Key Cassandra Elements: Memtable, Commit log, SSTables
• Gossip: Uses
• Snitch: Uses Data Distribution
• Virtual Nodes: Write path and Read path
• Incremental repair

Deep Dive into Cassandra Database

• Replication Factor
• Defining columns and data types
• Recognizing a partition key
• Specifying a descending clustering order
• Deleting data
• Updating a TTL
• Replication Strategy
• Defining a partition key
• Updating data
• Tombstones
• Using TTL

Node Operations in a Cluster

• Cassandra nodes
• Bootstrapping a node
• Adding a node (Commissioning) in Cluste
• Removing (Decommissioning) a node
• What’s new in incremental repair
• Cassandra and Spark Implementation
• Specifying seed nodes
• Defining a partition key
• Removing a dead node
• Repair
• Read Repair
• Run a Repair Operation

Managing and Monitoring the Cluster

• Cassandra monitoring tools
• Tailing
• Using JConsole
• Runtime Analysis Tools
• Logging
• Using Nodetool Utility
• Learning about OpsCenter

Backup & Restore and Performance Tuning

• Creating a Snapshot
• Restoring from a Snapshot
• RAM and CPU recommendations
• Hardware choices
• Cluster connectivity, security and the factors that affect distributed system performance
• End-to-end performance tuning of Cassandra clusters against very large data sets
• Selecting storage
• Types of storage to Avoid
• Load balance and streams

Hosting Cassandra Database on Cloud

• Security
• Ongoing Support of Cassandra Operational Data
• Hosting a Cassandra Database on Cloud

Talend for Data Integration & Big data

Talend – A Revolution in Big Data

• Working with ETL
• Rise of Big Data
• Role of Open Source ETL Technologies in Big Data
• Comparison with other market leader tools in ETL domain
• Importance of Talend (Why Talend)
• Talend and its Products
• Introduction of Talend Open Studio
• TOS for Data Integration
• GUI of TOS with Demo

Working with Talend Open Studio for DI

• Launching Talend Studio
• Working with projects
• Working with different workspace directories
• Creating and executing jobs
• Connection types and triggers
• Most frequently used Talend components [tJava, tLogRow, tMap]
• Read & Write Various Types of Source/Target Systems
• Working with files [CSV, XLS, XML, Positional]
• Working with databases [MySQL DB]
• Metadata management

Basic Transformations in Talend

• Context Variables
• tJoin
• tSortRow
• tReplicate
• Lookup
• Using Talend components
• tFilter
• tAggregateRow
• tSplit
• tRowGenerator
• Accessing job level/ component level information within the job
• SubJob (using tRunJob, tPreJob, tPostJob)

Advance Transformations & Executing Jobs remotely

• Various components of file management (like tFileList, tFileAchive, tFileTouch, tFileDelete)
• Error Handling [tWarn, tDie]
• Type Casting (convert datatypes among source-target platforms)
• Looping components (like tLoop, tForeach)
• Using FTP components (like tFTPFileList, tFTPFileExists, tFTPGet, tFTPPut)
• Exporting and Importing Talend jobs
• How to schedule and run Talend DI jobs externally (using Command line)
• Parameterizing a Talend job from command line

Big Data and Hadoop with Talend

• Big Data and Hadoop
• HDFS and MapReduce
• Benefits of using Talend with Big Data
• Integration of Talend with Big Data
• HDFS commands Vs Talend HDFS utility
• Big Data setup using Hortonworks Sandbox in your personal computer
• Explaining the TOS for Big Data Environment

Hive in Talend

• Hive and It’s Architecture
• Connecting to Hive Shell
• Set connection to Hive database using Talend
• Create Hive Managed and external tables through Talend
• Load and Process Hive data using Talend
• Transform data from Hive using Talend

Pig and Kafka in Talend

• Pig Environment in Talend
• Pig Data Connectors
• Integrate Personalized Pig Code into a Talend job
• Apache Kafka
• Kafka Components in TOS for Big data

Apache Spark & Scala

Introduction to Big Data Hadoop and Spark

• What is Big Data?
• Limitations and Solutions of Existing Data Analytics Architecture with Uber Use Case
• What is Hadoop?
• Big Data Customer Scenarios
• How Hadoop Solves the Big Data Problem?
• Hadoop’s Key Characteristics
• Hadoop Ecosystem and HDFS
• Rack Awareness and Block Replication
• Hadoop Cluster and its Architecture
• Big Data Analytics with Batch & Real-time Processing
• What is Spark? Spark at Yahoo!
• Hadoop Core Components
• YARN and its Advantage
• Hadoop: Different Cluster Modes
• Why Spark is needed?
• How Spark differs from other frameworks?

Introduction to Scala and Apache Spark

• What is Scala?
• Scala in other Frameworks
• Basic Scala Operations Control Structures in Scala
• Collections in Scala- Array
• Why Scala for Spark?
• Introduction to Scala REPL
• Variable Types in Scala
• Foreach loop, Functions and Procedures
• ArrayBuffer, Map, Tuples, Lists, and more

Functional Programming and OOPs Concepts in Scala

• Functional Programming
• Anonymous Functions
• Getters and Setters
• Properties with only Getters
• Singletons
• Overriding Methods
• Higher Order Functions
• Class in Scala
• Custom Getters and Setters
• Auxiliary Constructor and Primary Constructor
• Extending a Class
• Traits as Interfaces and Layered Traits

Deep Dive into Apache Spark Framework

• Spark’s Place in Hadoop Ecosystem
• Spark Deployment Modes
• Writing your first Spark Job Using SBT
• Spark Web UI
• Spark Components & its Architecture
• Introduction to Spark Shell
• Submitting Spark Job
• Data Ingestion using Sqoop

Playing with Spark RDDs

• Challenges in Existing Computing Methods
• Probable Solution & How RDD Solves the Problem
• What is RDD, It’s Functions, Transformations & Actions?
• Data Loading and Saving Through RDDs
• Key-Value Pair RDDs
• Other Pair RDDs o RDD Lineage
• RDD Lineage
• RDD Persistence
• WordCount Program Using RDD Concepts
• RDD Partitioning & How It Helps Achieve Parallelization
• Passing Functions to Spark

DataFrames and Spark SQL

• Need for Spark SQL
• Spark SQL Architecture
• User Defined Functions
• Interoperating with RDDs
• What is Spark SQL?
• SQL Context in Spark SQL
• Data Frames & Datasets
• JSON and Parquet File Formats
• Spark – Hive Integration
• Loading Data through Different Sources

Machine Learning using Spark MLlib

• Why Machine Learning?
• What is Machine Learning?
• Where Machine Learning is Used?
• Face Detection: USE CASE
• Different Types of Machine Learning Techniques
• Introduction to MLlib
• Features of MLlib and MLlib Tools
• Various ML algorithms supported by MLlib

Deep Dive into Spark MLlib

• Supervised Learning – Linear Regression, Logistic Regression, DecisionmTree, Random Forest
• Unsupervised Learning – K-Means Clustering & How It Works with MLlib
• Analysis on US Election Data using MLlib (K-Means)

Understanding Apache Kafka & Apache Flume

• Need for Kafka
• Core Concepts of Kafka
• Where is Kafka Used?
• Configuring Kafka Cluster
• What is Kafka?
• Kafka Architecture
• Understanding the Components of Kafka Cluster
• Need of Apache Flume
• What is Apache Flume?
• Flume Sources
• Flume Channels
• Integrating Apache Flume and Apache Kafka
• Basic Flume Architecture
• Flume Sinks
• Flume Configuration

Apache Spark Streaming- Processing Multiple Batches

• Drawbacks in Existing Computing Methods
• Why Streaming is Necessary?
• What is Spark Streaming?
• Spark Streaming Features
• Spark Streaming Workflow
• How Uber Uses Streaming Data
• Streaming Context & DStreams
• Transformations on DStreams
• Describe Windowed Operators and Why it is Useful
• Important Windowed Operators
• Slice, Window and Reduce By Window Operators
• Stateful Operators

Apache Spark Streaming- Data Sources

• Apache Spark Streaming: Data Sources
• Streaming Data Source Overview
• Apache Flume and Apache Kafka Data Sources
• Example: Using a Kafka Direct Data Source
• Perform Twitter Sentimental Analysis Using Spark Streaming

Apache Kafka

Introduction to Big Data and Apache Kafka

• Introduction to Big Data
• Need for Kafka
• Kafka Features
• Kafka Architecture
• ZooKeeper
• Kafka Installation
• Configuring Single Node Single Broker Cluster
• Big Data Analytics
• What is Kafka?
• Kafka Concepts
• Kafka Components
• Where is Kafka Used?
• Kafka Cluster
• Types of Kafka Clusters

Kafka Producer

• Configuring Single Node Multi Broker Cluster
• Constructing a Kafka Producer
• Sending a Message to Kafka
• Producing Keyed and Non-Keyed Messages
• Sending a Message Synchronously & Asynchronously
• Configuring Producers
• Serializers
• Serializing Using Apache Avro
• Partitions

Kafka Consumer

• Consumers and Consumer Groups
• Consumer Groups and Partition Rebalance
• Subscribing to Topics
• Configuring Consumers
• Rebalance Listeners
• Consuming Records with Specific Offsets
• Standalone Consumer
• Creating a Kafka Consumer
• The Poll Loop
• Commits and Offsets
• Deserializers

Kafka Internals

• Cluster Membership
• Replication
• Physical Storage
• Broker Configuration
• Using Producers in a Reliable System
• Using Consumers in a Reliable System
• The Controller
• Request Processing
• Reliability
• Validating System Reliability
• Performance Tuning in Kafka

Kafka Cluster Architectures & Administering Kafka

• Use Cases – Cross-Cluster Mirroring
• Other Cross-Cluster Mirroring Solutions
• Topic Operations
• Dynamic Configuration Changes
• Consuming and Producing
• Multi-Cluster Architectures
• Apache Kafka’s MirrorMaker
• Consumer Groups
• Partition Management
• Unsafe Operations

Kafka Monitoring and Kafka Connect

• Considerations When Building Data Pipelines
• Kafka Broker Metrics
• Lag Monitoring
• Kafka Connect
• Kafka Connect Properties
• Metric Basics
• Client Monitoring
• End-to-End Monitoring
• When to Use Kafka Connect?

Kafka Stream Processing

• Stream Processing
• Stream-Processing Design Patterns
• Kafka Streams: Architecture Overview
• Stream-Processing Concepts
• Kafka Streams by Example

Integration of Kafka With Hadoop, Storm and Spark

• Apache Hadoop Basics
• Kafka Integration with Hadoop
• Configuration of Storm
• Hadoop Configuration
• Apache Storm Basics
• Integration of Kafka with Storm
• Apache Spark Basics
• Kafka Integration with Spark
• Spark Configuration

Integration of Kafka With Talend and Cassandra

• Flume Basics
• Integration of Kafka with Flume
• Integration of Kafka with Cassandra
• Talend Basics
• Integration of Kafka with Talend
• Cassandra Basics such as and KeySpace and Table Creation

To see the full course content Download now