Introduction to Data Engineering

 1 Week
Skills Gained

• Data structures
• Methods for Data storage
• Data Pipelines definition
• DE general architecture
• Big Data definition
• Distributed systems

Learning Outcomes

You will discover the Data Lifecycle and get familiar with Data Engineering related terminologies. You will also see the big picture of the process and get introduced to Big Data architectures and platforms and what defines data to be “big”.

Programming skills for Data Engineers

1 Week
Skills Gained

• Dealing with files/dataframes in Python
• Get familiar with OOP with Python
• Linux as an operating system
• Shell scripting with Linux
• Deploying containers using Docker
• Docker main commands

Learning Outcomes

You will hone your skills in Python by understanding its basics and how data engineers use it to extract, transform, and load data. You will also learn how to deal with Linux environments, and run shell scripts in Linux. And for application’s fast deployment, you will learn how to use Docker to run different tools and applications .

Introduction to Databases

1 Week
Skills Gained

• Data relationships and data architectures
• Normalization and denormalization for SQL databases
• Writing Python scripts to access databases
• NoSQL database categories

Learning Outcomes

You will learn the different data models and the basics of SQL and NoSQL Databases. You will get familiar with common database management systems and build client sessions to access databases programmatically with Python.

Data Pipelines: Extract, Transform, & Load

1 Week
Skills Gained

• Extract data from different sources
• Data parsing, filtering, and validation
• Loading data in filing systems, databases, cloud, and data lakes
• Building ETL pipelines scripts

Learning Outcomes

You will learn the common data stacks and get introduced to some tools for each part of the pipeline. You will also explore each step of the process starting from extracting data from different sources, passing through data transformation and ending with the different options for loading data. At the end of this week, you will combine all of the previous steps to build a full pipeline.

Data Warehousing & Pipeline Automation

1 Week
Skills Gained

• Data Warehouses Architecting
• Star and Snowflake schemas for data modeling
• OLTP vs. OLAP data operations
• Automating ETL scripts with Linux shell scripting
• Data pipelines automation and orchestration with Apache Airflow

Learning Outcomes

This week, you will learn how to construct Data Warehouses, query data and create views in them. However, now that you have become familiar with the big picture of data engineering and data warehousing, and you know how to build data pipelines, you will learn how to automate and orchestrate its execution.

Big data: Hadoop & Spark ecosystem

• Distributed systems for storage and processing of data
• Hadoop Cluster Architecture and User Web Interface
• HDFS for big data storage
• Spark for processing big data

Data Warehouses in Hadoop ecosystem

1 Week
Skills Gained

• Understanding Apache Hive as data warehouse
• Grasp the concept of querying engines
• Basics of Machine Learning with Spark MLlib

Learning Outcomes

You will learn how to deal with big data warehouses and how to set up Apache Hive on a Hadoop cluster. You will dive deep into how to load data into Hive from local file systems and HDFS programmatically, in addition to the concept of querying engines. At the end, you will learn about the basics of Machine Learning, and how data engineers can leverage that with SparkML.

Big Data Streaming & Automation

1 Week
Skills Gained

• Apache Kafka for messaging systems
• Apache Flink for streaming data
• Apache Sqoop for databases migration
• Preparing data in warehouses for BI engineers

Learning Outcomes

This week, you will learn about new integrated tools within the Hadoop ecosystem, starting with dealing with messaging systems with Apache Kafka, and how to use Kafka to produce/consume data. Then you will go through streaming data sources with Apache Flink. In addition, you will get exposed to the migration of relational data into your data warehouse, and other tools to include such as Apache Oozie/Airflow used to automate and orchestrate data pipelines. At the end of this week, you will be able to understand the relationships between data and business intelligence engineers.

Data Engineering on AWS

1 Week
Skills Gained

• Accessing AWS through the Console vs. CLI
• AWS Kinesis for data collection and streaming
• AWS S3 for cloud Data Lake
• Accessing S3 from CLI and programmatically

Learning Outcomes

Beside working with data on-premises, you will dive into collecting and storing data in the cloud. This week, you will explore different services offered by cloud providers such as AWS for big data processing.

Data Engineering on Azure

1 Week
Skills Gained

• Azure Data Factory as a data ingestion tool
• Relational and non-relational data services in Azure
• Data Processing and Analysis on Azure using Databricks, Synapse Analytics, HDInsight, and much more
• Power BI for dashboards

Learning Outcomes

After exploring how to collect and store data in the cloud, you will learn how to process it, analyze it, and use Azure tools to create visuals.