How every Audition is scored.
Code checks the facts. An AI reviewer rates the work against levels written for each question. Code computes every score, and your team makes the decision.
Plan · 15%
How did they plan the work before building? 15% of the Audition score by default.
Framing · 25%
From the Plan: how the candidate framed the problem and the decision it serves.
Approach · 30%
From the Plan: the order of the work and what comes first.
Assumptions and trade-offs · 25%
From the Plan: the assumptions the answer depends on, and the trade-offs.
Checks and risks · 20%
From the Plan: the checks and risks planned before building.
Build · 70%
Does what they built give correct results that hold up? 70% of the Audition score by default.
Results correct · 22.5%
Automatic checks on the candidate’s own data: numbers 60% and decisions 40%, or numbers alone when the question has no other decision check.
Works on new inputs · 22.5%
The same automatic checks when the candidate’s main file runs again on fresh data.
Handling the stakeholder update · 15%
The update’s automatic checks: half on the candidate’s own data and half on fresh data. Scored only when the update was sent and reached the candidate.
Decisions and scope · 15%
Choices defended with the candidate’s own results (70%) and the scope and priorities of the build (30%).
Method and validation · 15%
The method as implemented (60%) and whether numbers were checked before use (40%).
Follows the task rules · 5%
Reads data at run time, uses the published shapes, no fixed values.
Deliverables complete · 5%
Every required file and writeup present with real content.
Writeup · 15%
Can the people who rely on the work understand and use it? 15% of the Audition score by default.
For the person who asked for the work · 35%
The stakeholder writeup, read by the person who asked for the work.
For the engineer who takes over · 25%
The technical writeup, so the engineer who takes over can rerun the work.
Consistency · 25%
Whether the Plan, code, outputs and writeups agree with each other.
Caveats and risks · 15%
Task-specific limits and risks, and their effect on the answer.
01 · Code checks the facts
The candidate’s solution runs on their own data and again on fresh data. Within each run, the numbers count for 60% and the decisions for 40%. The checks on the stakeholder update are scored as their own part.
02 · The reviewer reads the work
The question brief with the role, seniority and domain it was written for, the Plan, the code, the outputs, both writeups and the automatic results. A separate review reads the whole AI conversation beside the Plan. Neither sees the recording.
03 · Two runs, a third if they disagree
Fourteen checks are rated on written levels, twice, and code averages the runs. If the two disagree, a third run settles it and each check counts its middle rating. The report shows every run, and the reviewer then writes a summary that agrees with the scores.
Every score shows where it comes from.
From the public example report: two results checked by code and one rated by the reviewer, with the reason written out in the report.
Illustrative results for a fictional candidate. Open the report
Clear about scoring.
- Auditions: scored out of 100 the way the work happens, from fourteen rated checks and the automatic results. By default Plan counts for 15%, Build for 70% and Writeup for 15%.
- Weights per question: a question may set its own weights within fixed limits: Plan 10 to 25%, Build 60 to 80% and Writeup 10 to 25%. Results on the candidate’s own data always count as much as results on fresh data. The weights are fixed when the question is released, the reviewer never sees them, and the report shows the weights it used.
- Stakeholder update: your team chooses whether one is sent during the Audition. When it is not sent, or the interview ends before it arrives, that part is not scored and its share goes to the other Build parts.
- Python and SQL: 100 when every automatic check passes. Otherwise the reviewer places the answer on written levels, scoring 85, 70, 50, 25 or 0, in two runs with a third when they disagree. The latest submission counts; work never submitted is scored as it stands when the interview ends, and an empty or unchanged answer scores 0.
- Multiple choice: the share answered correctly, checked by code.
- AI usage: how the candidate used the assistant against their own Plan, how they checked its answers, and the tokens used against the allocation. Reported beside the score, never inside it.
- The interview score: the plain average of its parts. Each Audition, Python and SQL question is one part, and all multiple-choice items together are one part.
- The interview summary: a few lines at the top of the report covering every section, AI usage included, written by the AI reviewer from the section results. It agrees with the scores and never changes one.
- Not scored: length, polish, tone, speed, programming language, how much AI was used, and work beyond the brief.
- Missing evidence: a capture or recording failure is shown as not scored, never as a zero, and its share goes to the rest of its section. An Audition the candidate never opened scores 0.
- Session signals: help your team review the recording. They are never given to the reviewer and never change a score.
- The decision: the report never says hire or reject. Your team makes the call.