Pathwise

Manuals / Prompt Engineering / Chapter 10

C · Production · advanced · ~45 min · Chapter 10 of 12

Evals, golden sets & failure modes

Ship prompts like code: version them, test on golden cases, track regressions. Know the failure modes: hallucination, sycophancy, prompt injection.

Path progress
79%

Step 1 of 2

Build a 10-case eval set

Mix easy, medium, adversarial. Run before/after prompt changes. No vibes-only shipping.

Build a 10-case eval setDrag stickies · tap for tips
Study mapDrag stickies · tap for tipsKeep it shortdrag · tap →Name the waitdrag · tap →Scope locatorsdrag · tap →Trace when stuckdrag · tap →One browser firstdrag · tap →Isolate statedrag · tap →Assert the UIdrag · tap →Retry wiselydrag · tap →Seed datadrag · tap →Close the loopdrag · tap →Keep it shortdrag · tap →Name the waitdrag · tap →Scope locatorsdrag · tap →Trace when stuckdrag · tap →One browser firstdrag · tap →Isolate statedrag · tap →Pathwise hackdrag · tap →Build a 10-case evdrag · tap →Try thisdrag · tap →Follow the dashed drag · tap →

Do this now

10 cases for one prompt. Baseline score. Improve prompt. Re-score.

Was this step clear?
Chapter learning outcomes
  • Eval harnesses
  • Golden sets
  • Failure modes

Clear these before you leave

Side quest

Regression test

Change a prompt. Run eval set. Ensure score did not drop.