Agenda

• Introduction to data engineering on Azure
• What is data engineering
• Important data engineering concepts
• Data engineering in Microsoft Azure
• Introduction to Azure Data Lake Storage Gen2
• Understand Azure Data Lake Storage Gen
• Enable Azure Data Lake Storage Gen2 in Azure Storage
• Compare Azure Data Lake Store to Azure Blob storage
• Understand the stages for processing big data
• Use Azure Data Lake Storage Gen2 in data analytics workloads
• Introduction to Azure Synapse Analytics
• What is Azure Synapse Analytics
• How Azure Synapse Analytics works
• When to use Azure Synapse Analytics
• Use Azure Synapse serverless SQL pool to query files in a data lake
• Understand Azure Synapse serverless SQL pool capabilities and use cases
• Query files using a serverless SQL pool
• Create external database objects
• Use Azure Synapse serverless SQL pools to transform data in a data lake
• Transform data files with the CREATE EXTERNAL TABLE AS SELECT statement
• Encapsulate data transformations in a stored procedure
• Include a data transformation stored procedure in a pipeline
• Create a lake database in Azure Synapse Analytics
• Understand lake database concepts
• Explore database templates
• Create a lake database
• Use a lake database
• Analyze data with Apache Spark in Azure Synapse Analytics
• Get to know Apache Spark
• Use Spark in Azure Synapse Analytics
• Analyze data with Spark
• Visualize data with Spark
• Transform data with Spark in Azure Synapse Analytics
• Modify and save dataframes
• Partition data files
• Transform data with SQL
• Use Delta Lake in Azure Synapse Analytics
• Understand Delta Lake
• Create Delta Lake tables
• Create catalog tables
• Use Delta Lake with streaming data
• Use Delta Lake in a SQL pool
• Analyze data in a relational data warehouse
• Design a data warehouse schema
• Create data warehouse tables
• Load data warehouse tables
• Query a data warehouse
• Load data into a relational data warehouse
• Load staging tables
• Load dimension tables
• Load time dimension tables
• Load slowly changing dimensions
• Load fact tables
• Perform post load optimization
• Build a data pipeline in Azure Synapse Analytics
• Understand pipelines in Azure Synapse Analytics
• Create a pipeline in Azure Synapse Studio
• Define data flows
• Run a pipeline
• Use Spark Notebooks in an Azure Synapse Pipeline
• Understand Synapse Notebooks and Pipelines
• Use a Synapse notebook activity in a pipeline
• Use parameters in a notebook
• Plan hybrid transactional and analytical processing using Azure Synapse Analytics
• Understand hybrid transactional and analytical processing patterns
• Describe Azure Synapse Link
• Implement Azure Synapse Link with Azure Cosmos DB
• Enable Cosmos DB account to use Azure Synapse Link
• Create an analytical store enabled container
• Create a linked service for Cosmos DB
• Query Cosmos DB data with Spark
• Query Cosmos DB with Synapse SQL
• Implement Azure Synapse Link for SQL
• What is Azure Synapse Link for SQL?
• Configure Azure Synapse Link for Azure SQL Database
• Configure Azure Synapse Link for SQL Server 2022
• Get started with Azure Stream Analytics
• Understand data streams
• Understand event processing
• Understand window functions
• Ingest streaming data using Azure Stream Analytics and Azure Synapse Analytics
• Stream ingestion scenarios
• Configure inputs and outputs
• Define a query to select, filter, and aggregate data
• Run a job to ingest data
• Visualize real-time data with Azure Stream Analytics and Power BI
• Use a Power BI output in Azure Stream Analytics
• Create a query for real-time visualization
• Create real-time data visualizations in Power BI
• Introduction to Microsoft Purview
• What is Microsoft Purview?
• How Microsoft Purview works
• When to use Microsoft Purview
• Integrate Microsoft Purview and Azure Synapse Analytics
• Catalog Azure Synapse Analytics data assets in Microsoft Purview
• Connect Microsoft Purview to an Azure Synapse Analytics workspace
• Search a Purview catalog in Synapse Studio
• Track data lineage in pipelines
• Explore Azure Databricks
• Get started with Azure Databricks
• Identify Azure Databricks workloads
• Understand key concepts
• Use Apache Spark in Azure Databricks
• Get to know Spark
• Create a Spark cluster
• Use Spark in notebooks
• Use Spark to work with data files
• Visualize data
• Run Azure Databricks Notebooks with Azure Data Factory
• Understand Azure Databricks notebooks and pipelines
• Create a linked service for Azure Databricks
• Use a Notebook activity in a pipeline
• Use parameters in a notebook