AI & Agentic Testing for QA Engineers

Test LLM, RAG and AI agents with practical evals, security checks, CI/CD gates and a runnable QA lab.
What you'll learn

---

Design eval strategies for LLM, RAG, and agentic systems using deterministic, model-graded, and human-review checks.
Test AI outputs for correctness, safety, grounding, relevance, consistency, and regression risk.
Validate RAG retrieval, citations, context quality, and failure behavior through practical QA scenarios.
Test agent tool use, permissions, traces, security controls, and CI/CD quality gates using a working sample app.
Requirements

---

Basic software-testing knowledge is helpful. No prior AI or LLM automation experience is required. A computer is needed for the local lab.
Description

---

AI systems do not behave like traditional software. Their outputs can vary, retrieval can fail silently, agents can choose the wrong tool, and a passing functional check can still hide serious quality or safety risk. This course gives QA professionals a practical framework for testing those systems with confidence.
You will move from familiar software-testing concepts into modern AI evaluation. The course begins by translating assertions, test data, acceptance criteria, and regression testing into an eval-driven approach. You will then test LLM outputs for correctness, relevance, consistency, safety, and non-deterministic behavior using deterministic checks, model-graded evaluation, and human review.
Using a runnable QA Shop Assistant sample application, you will investigate real failure modes instead of relying only on theory. You will build evaluation datasets, validate Retrieval-Augmented Generation (RAG), measure retrieval quality, test faithfulness and grounding, and catch hallucinations. You will also test agent behavior, including tool selection, tool arguments, returned results, multi-step trajectories, and permission boundaries.
The security section covers the expanded attack surface of AI applications, prompt injection, red-team thinking, Promptfoo-based checks, and safety and policy evaluations. You will then bring these practices into delivery pipelines by running evals in GitHub Actions, defining thresholds and quality gates, identifying regressions, and planning production monitoring.
By the end of the course, you will be able to design a complete evaluation strategy for an AI-enabled product and assemble a practical test suite that combines functional quality, RAG validation, agent verification, security checks, and release criteria.
This course is designed for manual testers, QA engineers, SDETs, test leads, and QE architects who want to expand into AI and agentic-system quality engineering. Basic software-testing knowledge is helpful, but prior AI testing or LLM automation experience is not required. The downloadable lab package includes the sample application and setup instructions used throughout the practical exercises.
Who this course is for

---

Manual testers, QA engineers, SDETs, test leads, and QE architects moving into AI and agentic-system quality engineering.
Homepage

---