Skip to content
calwm

Red pen: Applications of foundation models

Exam answers on Applications of foundation models are often almost right. Each round hands you a short document with a few mistakes hidden in it. Mark the faulty lines, say why each one is wrong, and see what a good version would say.

Find the mistake before the auditor does.

Grab the red pen2 documents, about 4 min

What you will practise

You will practise Applications of foundation models, with questions touching FM evaluation and prompt engineering.

  • FM evaluation
  • Prompt engineering

It is part of the AWS Certified AI Practitioner track (AIF-C01), where Applications of foundation models is about 28% of the exam.

A taste of what you will get

  • Hannah wants every summary in the firm's 5-bullet format, and lawyers must be able to trust it.

    Sample document 1: Omar's prompt template for the contract summariser. Which lines are wrong, and why?

    • Role: you are a legal assistant for Calder & Wright
    • Task: summarise the contract in the firm's 5-bullet format
    • Format: no examples; the model will work out the format
    • Paste the client's own message into these instructions, word for word
    • Cite the clause number for every point

    Answer: Line 3 (Format: no examples; the model will work out the format): Worked examples are the quickest way to pin down a format. Better: Add two or three example contracts with their ideal summaries (few-shot). Line 4 (Paste the client's own message into these instructions, word for word): The client's text could override these instructions. Better: Keep the client's text separate from the instructions, and screen it with a guardrail.

  • Hannah wants proof the summaries are good before every lawyer uses them.

    Sample document 2: Luis's evaluation plan for the contract summariser. Which lines are wrong, and why?

    • Test set: 200 recent contracts the assistant hasn't seen
    • Accuracy: ROUGE scores, with no reference summaries written
    • Quality: lawyers spot-check a sample of summaries each week
    • Success measure: the number of prompts lawyers send
    • Report the results to Hannah before rollout

    Answer: Line 2 (Accuracy: ROUGE scores, with no reference summaries written): ROUGE needs reference summaries to compare against. Better: Have lawyers write reference summaries for a sample, then score against them. Line 4 (Success measure: the number of prompts lawyers send): Usage isn't value: track cost per interaction and satisfaction. Better: Track cost per interaction and lawyer satisfaction alongside the quality scores.

Advertise here

How scoring works

  • Every correct answer pays 10 XP for each difficulty level, so an easy item is worth 10, a medium one 20 and a hard one 30.
  • Answer with time to spare and you can earn up to 5 bonus XP on top.
  • A wrong answer pays nothing. It does not cost you anything either.
  • Flashcards pay 2 XP each time you grade yourself 3 or higher.
  • Using a hint costs 2 XP, so use it when it helps, not when it is comfortable.
  • Finish a session on consecutive days to build a streak. Your progress lives in your browser, and signing in syncs it.

Quick answers

How long does the red pen on Applications of foundation models take?
About 4 minutes for 2 documents. Pause whenever you like.
Do I need an account to play the red pen?
No. Progress is saved in your browser, and signing in syncs it across devices.