The candidate report your team reviews.
A fictional Mid-level Forward-Deployed AI Engineer on a public demo Audition. Code checks the results and an AI reviewer rates the Plan, build and writeups.
Maya R.
Forward-Deployed AI Engineer · Submitted- AI usageNot in the score98
One Audition question, scored out of 100 under the scoring rules. AI usage is shown beside the score and never counted in it.
Interview summary
Written by the AI reviewerMaya scored 96: strongest in the Build, weakest in the Plan.
The Audition scored 96. A safe order change assistant where code decides every change and reply; the call to go live with review is backed by the numbers. Beside the score, AI usage rated 97.5: used the assistant for one check in code and first drafts of both notes, and checked each answer, using 4.6% of the token allocation.
By section
- Audition · 2 review runsCan the assistant change orders before they ship?A safe order change assistant where code decides every change and reply; the call to go live with review is backed by the numbers.96.3/100
- AI usage, beside the score · 2 review runsHow they worked with the AI assistantUsed the assistant for one check in code and first drafts of both notes, and checked each answer. 3 requests, 11K of 250K tokens (4.6%).97.5/100
What the reviewer read
- The question brief, with the role, seniority and domain it was written for
- Their Plan, code and output files
- Both writeups
- Automatic checks on their data and on fresh data
- The whole conversation with the AI assistant, beside their Plan and run history
The reviewer never sees the session recording or its signals.
Can the assistant change orders before they ship?
A safe order change assistant where code decides every change and reply; the call to go live with review is backed by the numbers.
Two independent review runs scored 96.8 and 95.7. The score is their average.
Maya built the two tools the model calls, applied Wrenbury’s six rules in code and built every customer message from the result, so none of the model’s seven mistakes reached a customer. The automatic checks pass on her data and on fresh data. The review page shows the support team what the model tried. The note to Ines leads with the call and the 58% behind it, and handles the update in numbers. The handover could list the checks run by hand.
Did well
- Built every customer message from the result, so the $64.00 refund promise and the “tomorrow” delivery never reached a customer.
- Applied the six rules in order in one function, with all three settings declared.
- Recommended going live with review, backed by 7 of 12 corrected chats.
Missed
- The handover does not list the checks run by hand.
- The Plan does not say how the 30-minute packing cutoff was checked with the warehouse.
- The Plan does not say how many change requests the team handles today.
Plan
84 /100 · 15% of the scoreFraming 25%75
From the Plan: how the candidate framed the problem and the decision it serves.
Framed the decision in the Plan
AI review · rated 3 and 3 out of 4 in the two runs
The Plan names Ines as the person who decides and the three costly mistakes, but not how many change requests her team handles today.
Approach 30%88
From the Plan: the order of the work and what comes first.
Ordered the work in the Plan
AI review · rated 4 and 3 out of 4 in the two runs
The Plan puts the two tools and the ordered rules first, then the loop that calls the model, the messages and the review page.
Assumptions and trade-offs 25%75
From the Plan: the assumptions the answer depends on, and the trade-offs.
Named assumptions and tradeoffs in the Plan
AI review · rated 3 and 3 out of 4 in the two runs
The Plan chooses all three settings with a reason each, but states the 30-minute packing cutoff without saying how it was checked with the warehouse.
Checks and risks 20%100
From the Plan: the checks and risks planned before building.
Planned checks and risks
AI review · rated 4 and 4 out of 4 in the two runs
The Plan names concrete checks: every result read against the rules, refunds against item prices, and replies scanned for dates and amounts.
Build
100 /100 · 70% of the scoreCode written by the assistant: less than 1% of the submitted code (1 of 174 lines). Shown beside the Build score and never counted in it; the AI usage section says how it is counted.
Results correct 22.5%100
Automatic checks on the candidate’s own data: numbers 60% and decisions 40%, or numbers alone when the question has no other decision check.
Automatic checks on the candidate’s own data
Checked by code · 100 of 100
Numbers: 8 of 8 points. Decisions: 4 of 4 points.
Works on new inputs 22.5%100
The same automatic checks when the candidate’s main file runs again on fresh data.
Automatic re-run on fresh data
Checked by code · 100 of 100
Numbers: 8 of 8 points. Decisions: 4 of 4 points.
Handling the stakeholder update 15%100
The update’s automatic checks: half on the candidate’s own data and half on fresh data. Scored only when the update was sent and reached the candidate.
The update’s automatic checks
Checked by code · 100 of 100
On the candidate’s own data: 3 of 3 points. On fresh data: 3 of 3 points.
Decisions and scope 15%100
Choices defended with the candidate’s own results (70%) and the scope and priorities of the build (30%).
Choices defended with own results
AI review · rated 4 and 4 out of 4 in the two runs
Going live with review follows the published 20% rule from the run’s own 58%, and each setting is chosen with a reason.
Scope and priorities in the build
AI review · rated 4 and 4 out of 4 in the two runs
The build stays on the brief: two tools, the rules, one review page and the two notes.
Method and validation 15%100
The method as implemented (60%) and whether numbers were checked before use (40%).
Method as implemented
AI review · rated 4 and 4 out of 4 in the two runs
change_order applies the six rules in order in one function, get_order never returns the address, email or total, and message() builds every reply from the result.
Verified numbers before using them
AI review · rated 4 and 4 out of 4 in the two runs
Every corrected chat appears on the review page with what the model tried, and the run was repeated under each other setting to see each rule change the results.
Follows the task rules 5%100
Reads data at run time, uses the published shapes, no fixed values.
Reads data at run time, published shapes, no fixed values
AI review · rated 4 and 4 out of 4 in the two runs
The main file reads the four tables at run time, treats a model error as a handoff and stops if any chat does not get exactly one result.
Deliverables complete 5%100
Every required file and writeup present with real content.
Every required file and writeup present
AI review · rated 4 and 4 out of 4 in the two runs
Every output, the review page and both notes are present with real content.
Writeup
91 /100 · 15% of the scoreFor the person who asked for the work 35%75
The stakeholder writeup, read by the person who asked for the work.
Stakeholder writeup
AI review · rated 3 and 3 out of 4 in the two runs
The note leads with the call, gives the 7 of 12 behind it, names each model mistake in plain words and gives Ines a next step after 500 live chats.
For the engineer who takes over 25%100
The technical writeup, so the engineer who takes over can rerun the work.
Technical writeup
AI review · rated 4 and 4 out of 4 in the two runs
The handover explains the tools, the loop, how to run it and its limits, but does not list the checks run by hand.
Consistency 25%100
Whether the Plan, code, outputs and writeups agree with each other.
Plan, code, outputs and writeups agree
AI review · rated 4 and 4 out of 4 in the two runs
The Plan, the code, the outputs and both notes agree on the rules, the settings, the 58% and the call.
Caveats and risks 15%100
Task-specific limits and risks, and their effect on the answer.
Task-specific limits and their effect
AI review · rated 4 and 4 out of 4 in the two runs
Both notes name the small scripted sample and the weak point of the corrected check, and say what production needs.
How they worked with the AI assistant
Used the assistant for one check in code and first drafts of both notes, and checked each answer.
Two independent review runs scored 100 and 95. The score is their average.
- Requests
- 3
- 1 in Build, 2 in Writeup. AI is off during Plan.
- Tokens used
- 11K
- of 250K allocated, 4.6%
- Input / output
- 9.4K / 2K
- tokens on Claude Sonnet 5.5
- Code written by the assistant
- less than 1%
- 1 of 174 lines of submitted code; code in 1 reply, largest 1 line
Code written by the assistant: less than 1% of the submitted code (1 of 174 lines). Counted by code from the changes the assistant suggested in the chat that are in the submitted code files; a line the candidate changed counts as theirs, so the figure leans low.
Maya wrote the tools, the rules and the loop without help, then asked the assistant for a check that every chat gets exactly one result and ran the main file again. In Writeup, Maya asked for first drafts of the handover and the note to Ines from her own numbers, and checked each one against the summary before keeping it. Maya used 4.6% of the token allocation.
- Wrote the tools, the six rules and the loop without the assistant.
- Asked for one check in code: that every chat gets exactly one result.
- Used the assistant for first drafts of both notes, from Maya’s own numbers.
- Ran the main file after the check went in, and checked each draft against the summary before keeping it.
Plan alignment 20%100
Used the AI to carry out their own Plan
AI review · rated 4 and 4 out of 4 in the two runs
Each request carries out a step the Plan names: the check that every chat gets one result, then first drafts of the two notes from her own numbers.
Evidence: Plan, AI request 1, AI request 3
Request quality 20%88
Clear, scoped requests with the right context
AI review · rated 4 and 3 out of 4 in the two runs
Requests name the goal and the facts to use, one step at a time; the first could have said where in the file the check belongs.
Evidence: AI request 1, AI request 2
Verification 25%100
Checked what the AI produced before relying on it
AI review · rated 4 and 4 out of 4 in the two runs
Ran the main file again after the check went in, and read each draft against summary.csv before keeping it.
Evidence: AI request 1, Main output
Judgment and ownership 20%100
Kept the decisions their own
AI review · rated 4 and 4 out of 4 in the two runs
The tools, the six rules, the settings and the call are her own; the assistant added one check and drafted two notes she then checked.
Evidence: AI request 3, Stakeholder memo
Efficiency 15%100
Got value from the AI without waste
AI review · rated 4 and 4 out of 4 in the two runs
Three requests, each one moving the work forward, with no repeated asks.
Evidence: AI request 2
Submitted work
Every file exactly as submitted. Read-only.
Plan
- Locked before Build
Audition workspace
- The assistant, its tools and its rules
- One result per chat
- The three settings she chose
- Counts, refunds and the go-live call
- The review page for the support team
- For the next engineer
- For Ines
Order changes: can the assistant go live?
The headline
Not on its own yet. My code had to step in on 7 of the 12 chats (58%), above your 20% limit. I recommend going live with your team approving each change.
The evidence
- 6 changes made: 3 new addresses, 2 canceled items and 1 size swap. Refunds total $30.00.
- 1 order was already being packed, 1 large order went to your team, 2 changes were not allowed (an address in Paris and a size out of stock) and 2 customers were asked for their details.
- The model said "done" twice when nothing changed, followed an instruction hidden in a message, misread an order number, quoted $64.00 instead of an $18.00 refund, promised delivery "tomorrow" and once answered without the tools. Each time the customer saw my code's message instead.
What I recommend
Go live with review: the assistant prepares each change and a person approves it on the review page. After 500 live chats, switch cancellations on alone if their corrected share is at or below 20%.
What changed since your Plan and why
I added "a reply that promises a date" to what counts as corrected, after C-110.
How you handled the update
With packing an hour earlier, 2 of the 6 changes come too late (the Columbus and Dallas orders), so changes drop from 6 to 4. Tell customers that changes close 30 minutes before packing, which is now an hour earlier.
The biggest caveat
Twelve chats from a scripted model is a small sample. The live rate will differ, so the review page should be checked daily in week one.
How this was scoredThe AI model and how this result was scored
- Model
- Fictional example, no model was run
- Scored
- 17 minutes after submitting
- Review runs
- Every review runs twice. When the two runs disagree, two levels apart on any check or more than 5 points apart overall, a third run settles it and each check counts its middle rating. No result waits for a person.
- Audition, Can the assistant change orders before they ship?: 2 runs: 96.8 and 95.7, averaged.
- AI usage: 2 runs: 100 and 95, averaged.
- AI usage
- Five checks on the whole conversation with the assistant against the Plan, rated in two runs with a third when they disagree, shown beside the interview score and never counted in it.
- Evidence cited
- 5 of 6 gradable items
Session replay
Sample
2:07 condensed3 review markers3 AI requests
Maya’s session, condensed: her screen, camera and microphone together, with the Plan, Build and Writeup, each AI request, each code run and the review markers on one timeline.
Watch the replayWork captured
- PlanCaptured
- Work files7 files
- WriteupCaptured
- AI prompts3
Team notes
1 note- JS
Jordan S., hiring manager
Code decides every change, and the call to go live with review is honest. In the call I would ask what it would take to switch cancellations on alone.
Timeline
- Invited
- Started
- Submitted104 min of 110 min used
- Scored17 min after submitting
- Your team’s decisionWaiting