Source Systems, Data Ingestion, and Pipelines

About this course

In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipelines as code tools. You’ll also explore AWS and open source tools for monitoring your data systems and data quality.

What you will learn

• Gather stakeholder needs and translate them into system requirements.
• Implement a batch and a streaming ingestion process on AWS to ingest data from various source systems.
• Integrate aspects of security, data management, DataOps and orchestration into the data systems you build.