Testing & Evaluation
AI evaluation & failure diagnosis
Define what good looks like, test failures and make model behavior inspectable.
- Choose this when
- Choose this for AI behavior, failure taxonomies and test sets; choose Functional & UX testing for conventional app QA.
- Scope unit
- One behavior/failure class and agreed evaluation volume
- Possible delivery format
- Agreed diagnosis/Eval plan, test set and rubric; results and limits when execution is included
You send · You receive
What to send
- AI feature, current outputs and failure examples
- Expected behavior and cost of important mistakes
- Test dataset, runtime access and evaluation budget
- Reporting structure, sample size and measurement rules
Start by describing the materials you already have; missing details can be added when we agree the scope. Before work begins, we will confirm the materials, permissions and working conditions needed for your specific task.
Two stages
Stage 1 · Initial assessment
First assess current outputs, failure examples, test data and success criteria, delivering a failure taxonomy, evaluation scope/plan and data or ground-truth gaps. Evaluation/diagnosis can be the complete agreed deliverable; executed results and sample evidence are supplied when runs are included in scope.
Possible outputs
- Failure taxonomy, evaluation scope/plan and data/ground-truth gaps
An initial assessment can be the complete order.
Stage 2 · Agreed evaluation runs
Agree dataset size, scoring, accessible runtime, run count and budget before execution. Additional runs or model/workflow changes need their own scope; completing an evaluation does not require purchasing development.
Possible outputs
- Test set and rubric; executed results, sample evidence and proposals when runs are in scope
Scoped and quoted separately.
Both stagesEvaluation/diagnosis can be the complete deliverable; later model or workflow changes are not included by default
Third-party costsModel/API runs, third-party platforms, specialist labeling and domain review are separate costs to agree in advance. No cost-free or unlimited runs are promised.
Key checkThe client can inspect rubric, sample size, automated/human judgments and limits. When execution is included, results map to actual cases/evidence and unexecuted items are labeled.
Verifiable acceptance · 3
- Both stagesThe client receives agreed diagnosis/Eval artifacts explaining failure classes, operational pass/fail criteria, required data and unknowns
- Stage 2 · Agreed evaluation runsWhen runs are included: results trace to cases, actual outputs and failure evidence; unexecuted cases are marked
- Both stagesSeparate automated from human scoring, disclose sample size/generalization limits and do not substitute accuracy figures for missing ground truth
Hypothetical brief & output structure
Illustration · hypothetical
Illustrative example — not a completed client order
The request
An agent fabricates citations; define a targeted evaluation to surface the failure pattern.
Sample artifacts
eval-plan.mdrubric and failure classestest-set.csvbounded cases and expected evidenceeval-results.mdobserved evidence, proposals and gaps when execution is agreed
File names are illustrative
Other tasks of this kind
- In a sandbox, test whether an agent with external actions writes data without required confirmation; evaluate its approval boundary.
Output structure / test path
Test columns: input | expected_evidence | source_valid | claim_supported | reviewer_note
Scope boundaries
Agree test data and success criteria; separate automated from human judgments; quote model run costs separately.
Not included by default
- Accuracy guarantees without ground truth
- Guaranteeing a probabilistic model never fails
- A permanent evaluation platform unless commissioned
What affects the quote
Existing eval assets, labeling, runs and human review complexity.
Actual project references
Asterline ↗
Self-directed AI workflow and eval with synthetic data
Evaluation on synthetic test data is not real-client business accuracy.
Score Simplifier ↗
Published Agent Skill (Alpha)
Published examples verify score-sourced simplification, engraving and PDF/MusicXML/MIDI checks; they do not prove accurate transcription of arbitrary audio recordings.
Self-directed work and published projects demonstrate relevant practice. They are not paid-client delivery records or guarantees for this task.
