Analyzing and Securing AI System Performance

About this course

This long course develops skills for operational analytics, secure data practices, and governance essential to building trustworthy, auditable agentic systems. You will aggregate and analyze operational metrics, design A/B experiments and statistical tests to validate agent improvements, and craft clear visualizations and alerting rules for stakeholders. The course covers end-to-end data hygiene: cleaning, schema validation, reproducible notebooks with data versioning, and trade-offs between sample size and noise for experimental design. It also addresses security and governance: securing API endpoints per OWASP ASVS, dependency vulnerability analysis, secret-management trade-offs (on-prem vs managed), and threat modeling (STRIDE). Practical tasks include building DBT models for telemetry, configuring alerts, producing reproducible analytic notebooks, and creating STRIDE diagrams with documented mitigations to reduce operational and supply-chain risk.

What you will learn

• Use data aggregation and A/B testing to analyze metrics, create clear visualizations, and build automated KPI alerts.
• Clean raw data, evaluate quality trade-offs, and create reproducible, versioned notebooks for peer replication.
• Secure APIs using OWASP guidelines, analyze vulnerability scans, and evaluate secret management solutions.
• Create structured threat models to analyze, document, and prioritize system security risks and vulnerabilities.