Azure Data Engineer is the most lucrative job profile and those who possess the skill sets require to become Azure Data Engineer are always in great demand and easily can get a 6 figure salary in USD as per the market trend.

But, it can be overwhelming for a beginner to start his/her journey to become a Azure Data Engineer since there are so many different technologies and services offered by Azure.

This Course measures your ability to accomplish the following technical tasks: design and implement data storage; design and develop data processing; design and implement data security; and monitor and optimize data storage and data processing.

Learning Objectives:

This course helps you to understand solutions in Azure for Data Engineering purpose

• Azure Fundamentals and Data Fundamentals
• Azure Storage Account and Data Lake Store Gen 2
• Azure Databricks
• Azure Data Factory
• Azure Containers
• Azure EventGrid, EventHub, Service Bus
• Azure Synapse Analytics
• Azure Data Warehouse (Serverless/Dedicated SQL Pool)
• Synapse SQL, Link and Pipeline Services
• Azure Synapse Spark
• Microsoft Power BI

Course content

Azure Fundamentals

• Cloud Concepts
  • Key concepts and terms
  • Economics of Scale
  • Types of Cloud Models
  • Types of Cloud Services
  • Management Responsibilities
• Core Azure Services
  • Terminologies: Data Centers, Regions, Pairs, Geographies, Availability Sets and Zones.
  • Provisioning concepts: Resources, Groups, Portal, Resource Manager, Market Place
  • Compute Services: Virtual Machines, Containers, Serverless Compute
  • Networking Services: Virtual Network, Load Balancers, VPN and Appl Gateway
  • Storage Services: Storage Account, Data Lake Store

• Core Azure Services
  • Database Services: SQL Database, Cosmos DB
  • IoT Services: IoT Central, IoT Hub
  • Big Data and Analytics Services: Synapse Analytics, Data bricks, Stream Analytics.
  • ML and AI Services: ML Service, Cognitive Services
• Security, Privacy and Compliances
  • Azure Network Security Solutions
  • Shared Responsibilities
  • Azure Active Directory
  • Security Center
  • Azure Key Vault
  • Azure Firewall
  • Network Security Group
  • Application Security Group
  • Azure Policies
• Azure Pricing and Support
  • Azure Subscription
  • Management Groups
  • Azure Products and Services
  • Azure Free Account
  • Pricing Calculator
  • Knowledge Centers and Forums
  • Service Level Agreements

Azure Storage Account and Data Lake Store Gen 2 for Big Data Store

• Overview of Storages, Object and Hierarchical Storages
• Azure Blob Storage Vs. Data Lake Store Gen 2
• Provisioning Azure Storage account
• The Blob, File, Table Storage
• File System using Azure Storage Explore
• Provisioning Azure Data Lake Store Gen 2
• Uploading/Downloading files
• The Account Key and SAS Token

Azure Database Solutions

• Azure Data Services
• SQL Servers on Azure VMs and its use cases
• Azure Database PaaS solutions
  • Elastic Pool
  • Single SQL Database
  • Azure SQL Managed Instance
  • PostgreSQL, MariaDB and MySQL services
  • Provisioning Azure SQL Database,
  • Configuring connectivity and firewall

Working with Database through Management Studio.

• Understanding Structured Query Language
• Overview of SQL Management Studio
• SQL Dialect
• DML and DDL

Working with Database through Management Studio.

• Understanding Structured Query Language
• Overview of SQL Management Studio
• SQL Dialect
• DML and DDL
• Joins and other clauses
• Stored Procedures and Functions
• Overview of SQL Data Studio
• Data Modelling using concepts of Normalization

Understanding Azure Data Factory

• Introduction to Azure Data Factory as Orchestration tool
• Copy activity to migrate data from Azure Blob Storage to Azure Data Lake store
• Azure Data Factory Components
  • Connectors/Link Service
  • Data Sets
  • Pipeline
  • Activities

Working with Azure Data Factory

• Integration with Azure SQL Database
• Monitoring and Troubleshooting
• Scheduling the Data Pipeline
• Working with Parameters

Working with Azure Data Factory

• Different types of Transformations.
• Working with Incremental Data
• Working with Bulk Copy
• Working with Mapping Flows.
• Monitoring and Troubleshooting

Azure Container Instances

• Azure Container Registry
• Run Containers with ACI
• Assigning public DNS for ACI

Azure Service Bus

• Creating Messaging Service
• Understanding Topics and Queues
• Creating topic Subscriptions
• Creating topic Filters

Azure Event Grids

• Understanding Event based communication
• Event publishers and subscribers
• Filtering events
• Create custom Event Grid Topics
• Publishing events from Applications

Azure Event Hubs

• Creating Event Hub
• Getting connection String – SAS tokens
• Event Hub vs IoT Hub
• Creating partitions
• Send, Receive and Capture events

Azure Databricks for Analytical Processing

• Big Data Analytics with few use-cases
  • Extract, Load and Transform
  • Batch Processing
  • Realtime and Stream Processing for Data Engineer
  • Interactive processing for Data Scientist
  • Predictive Analytics with Machine Learning
  • Cognitive analytics with Deep Learning
• The Azure Databricks as unified solution
• Azure Databricks Features, Components
  • Databricks Workspace
  • Databricks Runtime
  • Databricks Clusters and Jobs
  • Notebook and Code Governance
  • Databricks Workflows
  • Databricks File System
• The processing engine- Apache Spark and its features
• Spark Context, Spark Session
• Working with Databricks File System

Azure Databricks for Analytical Processing

• Concept of RDDs and Data Frames
• ELT operations in Spark
• Working with RDDs
• Maps, FlatMaps, PairedRDDs and operations on RDDs
• Accumulators and Broadcasting

Azure Databricks for Analytical Processing on Data Frames

• Spark Data Frames
• Maps, Filters, Group By, Order By operations
• Analytics operation on Data Frames: Data Cleaning, Structural Changes
• Working with CSV, Parquet, JDBC, Text files.
• Different Transformations and operations on Data Frame
• Temp and Global Temp Views, Structured SQL
• Applying Schema, dealing with headers, Inferring Schema

Azure Databricks: Stream Processing and Structured Streaming

• Stream Analytics on Spark.
• Lambda and Kappa Architectures
• Delta Lake on Spark

Azure Databricks: More insight and Scheduling

• Understanding Spark UI
• Troubleshooting and Monitoring
• Working with Job Cluster and Scheduling Jobs.
• Azure Databricks with Azure Data Factory

Overview of Data Warehousing and Azure Synapse Analytics

• Introduction to SQL Data Warehouse
• Features of SQL Data
• Benefits of SQL Data Warehouse
• Architecture of SQL Data Warehouse
• Massive Parallel Processing
• Components of Azure Synapse
• SQL Analytics: Complete T-SQL based analytics
• SQL pool (pay per DWU provisioned) – Generally Available
• SQL on-demand (pay per TB processed) – (Preview)
• Spark: Deeply integrated Apache Spark (Preview)
• Data Integration: Hybrid data integration (Preview)
• Studio: unified user experience. (Preview)
• Provisioning SQL Data Warehouse
• Working with the Server Settings
• Working with the Database Settings
• Scalable Storage and Compute
• Pause and Scale on Demand
• Data Warehouse Units
• Hybrid Data Source Scenarios
• Data Warehouse Workload
• Pricing Details

 Working With SQL Data Warehouse-I

• Creating a Database
• Connecting to a Database
• Creating a table
• Creating a table from a SELECT Statement
• Limitations of options while creating a table
• Working with Tables
• Creating Views and Stored Procedures
• Querying SQL Data Warehouse Tables
• T-SQL in SQL Data Warehouse

Data warehousing using Azure Synapse Analytics-II

• Introduction to Synapse Pool: Serverless and Dedicated SQL Pool
• Connecting from SQL Server Management Studio
• Connecting from SQL Server Data Tools
• Sharding Patterns and Table geometries
• Working with Dimensions and Fact tables
• Choosing appropriate partition keys
• The Star and Snowflake schema
• Designing and Implementing Serving Layer

Working with Bulk data in Azure Synapse Pipelines

• Preparing the Data Warehouse
• Setting up Connections
• Retrieving the Data
• Transforming the Data
• Loading the Data
• Loading Data Using PolyBase

Transform data with Azure Data Factory or Azure Synapse Pipelines

• Data integration with Azure Data Factory or Azure Synapse Pipelines
• Code-free transformation at scale with Azure Data Factory or Azure Synapse Pipelines
• Create data pipeline to import poorly formatted CSV files
• Create Mapping Data Flows

Manage Databases in Azure SQL Data Warehouse

• Management Tools
• Query Monitoring
• Pause and Scale
• Tables and Indexes

Design and implement a Modern Data Warehouse

• Design a Modern Data Warehouse using Azure Synapse Analytics
• Design a multidimensional schema to optimize analytical workloads
• Code-free transformation at scale with Azure Data Factory
• Populate slowly changing dimensions in Azure Synapse Analytics pipelines

Integration of Azure SQL Data Warehouse with Azure Services

• Power BI
• Azure Data Factory
  • Loading the Data with Azure Data Factory
  • Move Data To/From SQL Data Warehouse with Data Factory

 Introduction to Microsoft PowerBI

• What is Business Intelligence
• What is Power BI
• Different Products of Power BI
  • Power BI Desktop
  • Power BI Pro
  • Power BI Premium
  • Power BI Service
  • Power BI Mobile
  • Power BI Embedded
  • Power BI Reports

Introduction to Power Query

• Overview of Power Query
• Importing data into Power Query from Excel file, Csv, Text Files
• Import data into Power Query from MS Access file/ SQL Server
• Import data into Power Query from Web, PDF
• Loading data from Power Query back to Excel sheet, Power View
• Configuring Cache, Refreshing changes in Data of Source

Power Query: Data Transformation

• Text Functions
• UPPER, LOWER, TRIM, PROPER, CLEAN
• ADD PREFIX, ADD SUFFIX
• LEFT, RIGHT, MID, LENGTH
• TEXT BEFORE/AFTER/BETWEEN DELIMITERS
• MERGE, SPLIT BY DELIMITER, SPLIT BY POSITION
• Number Functions
• ADDITION, SUBTRACTION, MULTIPLICATION, DIVSION
• MODULO, PERCENTAGE
• SUM, AVERAGE, MIN, MAX, COUNT
• ROUNDUP, ROUNDDOWN, ROUND
• Date Functions
• FIND AGE, DATE ONLY
• YEAR, START OF YEAR, END OF YEAR
• MONTH, START OF MONTH, END OF MONTH, DAYS IN MONTH, NAME OF MONTH
• QUARTER, START OF QUARTER, END OF QUARTER
• WEEK OF MONTH, WEEK OF YEAR, START AND END OF WEEK
• DAY OF WEEK, DAY OF YEAR, NAME OF DAY, START AND END OF DAY
• SUBTRACT DAYS, EARLIEST, LATEST

Power Query: Data Cleaning, Transformation

• Row Operations
• Remove TOP, BOTTOM, ALTERNATE rows
• Remove DUPLICATE, BLANK ROWS, ERRORS
• Column Operations
• Index (from 0,1, Custom)
• Custom Column
• Column from Examples
• Duplicate column

Power Query: Data Cleaning, Transformation

• Miscellaneous Operations
• Count Rows
• Reverse
• Transpose
• Pivot Columns
• Unpivot Columns, Unpivot other columns
• Fill up and Fill down
• First Row as Headers and Headers as first row
• Group By
• Data Type
• Parameters

Data Modelling

• Designing a Data Model
• Define and Configuring Tables and Relationships
• Configuring column properties
• Define relationship cardinalities
• Define cross-filter direction
• Create Hierarchies

Introduction to Power View

• The Power View in Power BI
• Filters Pane, Visualizations Pane, Fields Pane
• Report Visuals, Fields, Pages and Filters
• Report, Data and Relationship Options
• Working with Visualizations and PBI Canvas
• Working with Enter Data, Get Data Options
• Enter Data from Excel Files, Basic Reports
• PBIX and PBIT File Formats. Differences
• Designing Simple / Basics Reports in PBI
• Visual Interactions in Power BI – Options
• Spotlight Options with Visuals, Real-time Use
• VIEW Options with Power BI Desktop Tool

Power View: Visualizations

• Power BI Visualizations and Types
• Fields, Formats and Analytics Options
• Table Visuals and Properties, Filters
• Data Bar and Data Scaling Options
• Divergent Colors and Data Labelling
• Conditional Formatting, Grid Lines
• Matrix : Sub Totals, Drilldown Options
• Row Groups and Column Groups in Matrix
• Slicer – Single Select and Show All
• Chart Reports – Types, Common Properties
• Axis, Legend, Value and Tooltip Options
• Stacked Bar Chart, Clustered Column Chart
• Stacked Bar Chart, Stacked Column Chart
• Clustered Bar Chart : Properties, Use
• Clustered Column Chart : Properties
• 100% Stacked Bar and Column Charts
• Data Label Properties, Legend Properties
• Line, Pie and Donut Chart: Properties and Usage
• Cards, Text Box, Map, Tree Map, Scatters: Properties and Usage

Power BI Services on Cloud

• POWER BI CLOUD (OPS)
• Power BI Service (Cloud) Architecture
• App Workspace Creation in Real-time
• Publish Reports from Power BI Desktop
• Reports, Datasets in Power BI Cloud
• Creating Dashboards and Realtime Use
• Pin Visuals and Pin LIVE Report Pages
• Dashboard Creation with Tiles, Media
• Tiles – Images, Web Content, Chapters
• Dataset Actions – Report Creation Options

To see the full course content Download now