Testing, validation, and troubleshooting cheat sheet
All 14 AIP-C01 flashcards for Testing, validation, and troubleshooting on one page, grouped by exam objective. Print it or save it as a PDF for a last-minute read.
Implement evaluation systems for GenAI
- What is a golden dataset?
- A curated set of prompts with expected answers or rubrics, replayed after every change to catch regressions.
- What is LLM-as-a-judge?
- Using a model to score other outputs against criteria such as correctness, completeness, helpfulness or harmfulness.
- Which evaluation types does Bedrock Model Evaluation cover?
- Automatic (programmatic) evaluation, human evaluation, and model-based (LLM-as-a-judge) evaluation, plus RAG evaluation.
- Why evaluate retrieval separately in RAG?
- A bad answer can come from poor retrieval or poor generation. Separate scores show which one to fix.
- What does canary testing of a model look like?
- Send a small share of traffic to the new model or prompt, compare metrics, and expand only if quality holds.
- What should you measure for agents?
- Task completion rate, tool selection and argument accuracy, reasoning quality across steps, and cost per task.
- Why collect user feedback?
- Ratings and annotations show real failures and feed new cases into the golden dataset.
Troubleshoot GenAI applications
- First check when a RAG answer is wrong?
- The retrieved chunks. Wrong or missing context is a more common cause than a weak model.
- What causes ValidationException on long input?
- The prompt plus history exceeds the model's context window. Trim, summarise or retrieve less.
- What does stop reason max_tokens tell you?
- Generation ended because it hit the output cap, so the answer may be truncated.
- How do you fix ThrottlingException?
- Backoff with jitter, request a quota increase, spread load with Cross-Region inference, or buy Provisioned Throughput.
- How do you diagnose embedding problems?
- Check that query and documents use the same embedding model and dimension, then inspect nearest neighbours for sample queries.
- Which tools help trace a slow GenAI request?
- AWS X-Ray for the call path and CloudWatch Logs Insights over invocation logs for prompt and response details.
- How do you keep prompt changes safe?
- Version prompts, run them against the test set, compare with the previous version, and roll back if scores drop.