The candidate report your team reviews.
A fictional Senior Data Engineer, who had to find the data’s problems. Code checks each issue on new test data and an AI reviewer rates the Plan and writeups.
Sam R.
Senior Data Engineer · Submitted- AI usageNot in the score98
One Audition question, scored out of 100 under the scoring rules. AI usage is shown beside the score and never counted in it.
Interview summary
Written by the AI reviewerSam scored 93: strongest in the Plan, weakest in the Build.
The Audition scored 93, with 8 of 10 issues solved, each checked by code on its own test data. The issues not solved: “Two events of one dispute in the same second make one history row” and “The processor’s corrected amounts from 9 October are applied under the chosen policy”. Built the pipeline, the page and the board-day call; events in the same second keep the wrong amount, and the update rewrites the history. Beside the score, AI usage rated 97.5: used the assistant for the page’s filters and download, and checked them in Preview, using 11.8% of the token allocation.
By section
- Audition · 8 of 10 issues solved · 2 review runsCan finance and the disputes team trust one chargebacks page?Built the pipeline, the page and the board-day call; events in the same second keep the wrong amount, and the update rewrites the history.92.6/100
- AI usage, beside the score · 2 review runsHow they worked with the AI assistantUsed the assistant for the page’s filters and download, and checked them in Preview. 1 request, 31K of 262K tokens (11.8%).97.5/100
What the reviewer read
- The question brief, with the role, seniority and domain it was written for
- Their Plan, code and output files
- Both writeups
- Automatic checks on their data and on fresh data
- For each issue code did not prove solved: which of its test sets failed, which outputs were wrong there and their own outputs, never the expected values
- The whole conversation with the AI assistant, beside their Plan and run history
The reviewer never sees the session recording or its signals.
Can finance and the disputes team trust one chargebacks page?
Built the pipeline, the page and the board-day call; events in the same second keep the wrong amount, and the update rewrites the history.
Two independent review runs scored 93.1 and 92.2. The score is their average.
Sam found the problems in the processor’s files without being told them and built an Airflow DAG with SQL tasks and a dbt project: repeats count once, gift card and cents events are set aside, late events land where they happened, and corrections and won appeals restate a final month. Code proved 8 of the 10 issues solved on test data Sam never saw, the final call and the page among them. Two events in the same second keep the opening’s amount, and the 9 October corrections overwrite old history rows. The Plan and both notes are clear.
Did well
- September stayed provisional on board day, became final when it matched the statement, and was restated when an appeal was won.
- One page serves both teams: the months beside the board day’s figures, every dispute by deadline, filters in the address and a download.
- Before the DAG, explore.sql found the files’ problems: repeats, the gift card shop, amounts in cents, late events and changes to August.
Missed
- When two events of a dispute share a second, the history row keeps the opening’s amount, not the correction’s.
- The 9 October corrections are written onto the old lost rows, so the history loses the amounts reported before the update.
- Neither the Plan nor the runbook says how events in the same second, or a late correction, are checked.
Plan
95 /100 · 15% of the scoreFraming 25%100
From the Plan: how the candidate framed the problem and the decision it serves.
Framed the decision in the Plan
AI review · rated 4 and 4 out of 4 in the two runs
The Plan names who uses the page and what a wrong number costs each: a board figure that moves later for Grace, a missed evidence deadline and a lost dispute for Leo.
Approach 30%100
From the Plan: the order of the work and what comes first.
Ordered the work in the Plan
AI review · rated 4 and 4 out of 4 in the two runs
The Plan lays out the run task by task, from locate to export, says where state lives between runs, and makes event time, never the file, the history’s only clock.
Assumptions and trade-offs 25%100
From the Plan: the assumptions the answer depends on, and the trade-offs.
Named assumptions and tradeoffs in the Plan
AI review · rated 4 and 4 out of 4 in the two runs
The Plan chooses restate, names what it costs Grace after the board pack, and says what adjust would hide from Leo until the next month.
Checks and risks 20%75
From the Plan: the checks and risks planned before building.
Planned checks and risks
AI review · rated 3 and 3 out of 4 in the two runs
The Plan names checks that can fail: September provisional on the 5th, the replay of the 6th, dbt’s tests and the run counts. It does not say how a history row is checked when two events share a second, or how a late correction shows in the history.
Build
92 /100 · 70% of the score8 of 10 issues solved
Code written by the assistant: about 5% of the submitted code (21 of 412 lines). Shown beside the Build score and never counted in it; the AI usage section says how it is counted.
Build is the real problems in the brief, the call it asks for and the stakeholder update. Code checks each one on its own test data the candidate never saw. A solved issue scores 100, and only code gives 100. An issue that is not solved gets partial credit from the AI reviewer: 85, 70, 50, 25 or 0. Every issue is the same share of Build.
An event the processor sends again is counted once, even one that was set asideSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
Disputes of the gift card shop, a separate account, are set asideSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
An amount the correction tool wrote in cents is set asideSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
An event that arrives late or out of order counts at the time it happenedSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
Two events of one dispute in the same second make one history rowNot solved, partial credit from the AI reviewer70
Not solved: some outputs were wrong on every one of its 2 test sets.
AI review · rated 3 and 3 out of 4 in the two runs, 70 and 70 points, averaged
dispute_history_now.sql keeps one row for the events of a dispute in the same second, as the rules ask, but orders them by event_seq ascending, so the row takes the opening’s amount instead of the correction’s. The history’s amounts were wrong on both test sets.
A corrected amount counts from the time of the correction, also after a month is finalSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
A loss won back on appeal leaves its month’s losses, also after the month is finalSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
The page gives finance its months and the disputes team its disputes, with working filtersSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
A month is called final only when it matches the processor’s statementSolved on both test sets100
Checked by code on its own test data: every output was right on every set. Only code gives 100.
The processor’s corrected amounts from 9 October are applied under the chosen policyNot solved, partial credit from the AI reviewer50
Not solved: some outputs were wrong on every one of its 2 test sets.
AI review · rated 2 and 2 out of 4 in the two runs, 50 and 50 points, averaged
The 9 October run applies the processor’s corrected amounts, and September under restate is right at 22 disputes and 1,840.63. But stage.sql writes each correction onto the dispute’s old lost event and drops the correction, so no history row starts on 9 October. The history was wrong on both test sets.
The full runsShown and read by the reviewer, not scored
The candidate’s solution also ran on their own full data and on fresh full data. These runs are shown and read by the reviewer, and they add no points, so one mistake never costs twice.
Automatic checks on the candidate’s own data
Checked by code · 88 of 100
Numbers: 5 of 6.25 points. Decisions: 4 of 4 points.
Automatic re-run on fresh data
Checked by code · 91 of 100
Numbers: 5.5 of 6.5 points. Decisions: 0.5 of 0.5 points.
Writeup
93 /100 · 15% of the scoreFor the person who asked for the work 35%100
The stakeholder writeup, read by the person who asked for the work.
Stakeholder writeup
AI review · rated 4 and 4 out of 4 in the two runs
The note answers whether Grace and Leo can trust the page, gives each month’s figures and status after the 9 October run, and defends restate to both of them.
For the engineer who takes over 25%100
The technical writeup, so the engineer who takes over can rerun the work.
Technical writeup
AI review · rated 4 and 4 out of 4 in the two runs
The runbook covers running a day, rerunning it, backfilling and recovering, and says what a failed check_final_months means and where to look first.
Consistency 25%88
Whether the Plan, code, outputs and writeups agree with each other.
Plan, code, outputs and writeups agree
AI review · rated 4 and 3 out of 4 in the two runs
The Plan, code, outputs and notes agree on the months, the replay and the update; neither note mentions the events that share a second, which explore.sql found.
Caveats and risks 15%75
Task-specific limits and risks, and their effect on the answer.
Task-specific limits and their effect
AI review · rated 3 and 3 out of 4 in the two runs
The note names what restate costs Grace, but no limit of the pipeline itself, such as what the history keeps when the processor corrects an amount.
How they worked with the AI assistant
Used the assistant for the page’s filters and download, and checked them in Preview.
Two independent review runs scored 100 and 95. The score is their average.
- Requests
- 1
- 1 in Build. AI is off during Plan.
- Tokens used
- 31K
- of 262K allocated, 11.8%
- Input / output
- 30K / 512
- tokens on Claude Haiku 5.5
- Code written by the assistant
- 5%
- 21 of 412 lines of submitted code; code in 1 reply, largest 21 lines
Code written by the assistant: about 5% of the submitted code (21 of 412 lines). Counted by code from the changes the assistant suggested in the chat that are in the submitted code files; a line the candidate changed counts as theirs, so the figure leans low.
Sam built the pipeline, the page, the update and both notes without help. Sam asked the assistant for one file, the page’s filters and download, applied it and opened the filtered views before relying on it. Sam used 11.8% of the token allocation.
- Asked for one part, the page’s filters and download, in Sam’s own words.
- Took one short piece of code: the filters file, applied as written.
- Wrote the DAG, the SQL tasks, the dbt project, the rest of the page and both notes without the assistant.
- Ran the page in Preview and opened its filtered views before relying on the file.
Plan alignment 20%100
Used the AI to carry out their own Plan
AI review · rated 4 and 4 out of 4 in the two runs
The one request builds a part the Plan names: the page both teams use, its filters and its download.
Evidence: Plan, AI request 1
Request quality 20%88
Clear, scoped requests with the right context
AI review · rated 4 and 3 out of 4 in the two runs
A scoped request in Sam’s own words: one file, what each function does, and the address format.
Evidence: AI request 1
Verification 25%100
Checked what the AI produced before relying on it
AI review · rated 4 and 4 out of 4 in the two runs
Sam applied the file, ran the page in Preview and opened its filtered views before relying on it.
Evidence: AI request 1, Run history
Judgment and ownership 20%100
Kept the decisions their own
AI review · rated 4 and 4 out of 4 in the two runs
The DAG, the SQL tasks, the dbt project, the month policy and the update are Sam’s own; the assistant wrote one short file.
Evidence: AI request 1, Code
Efficiency 15%100
Got value from the AI without waste
AI review · rated 4 and 4 out of 4 in the two runs
One request, applied and checked, with nothing asked twice.
Evidence: AI request 1
Submitted work
Every file exactly as submitted. Read-only.
Plan
- Locked before Build
Writeups
- For Grace and Leo
- The runbook, for the next engineer
Outputs
- Each month’s losses and status, for every run
- Every dispute’s history, for every run
- Events set aside, with the reason
- One row per run, with its counts and policy
- What the last run left for the page
Airflow DAG and SQL tasks
- The DAG: six tasks for one day
- Repeats, set-asides, accepted events and the run log
- This run’s history, and each month under the policy
- The four files and the page data
dbt project
- Staging as views, marts as tables
- The warehouse the DAG uses
- The start file, each row as an event
- Every accepted event, with its status
- The processor’s statement
- Sources and staging models
- Every dispute’s history, by event time
- Each month beside the statement
- The two marts
- Each dispute’s rows chain with no gap
- Every status is one of the five
- Every run log row adds up
Page
- The page for both teams
- Written with AI Assist
Other files
- Scratch queries: what the files really hold
Can you trust the chargebacks page?
Yes, for every run the pipeline has published. Each night's run reads the processor's file once. An event it has seen before is skipped; an event for the gift card shop, or with an amount above its order, is set aside with its reason. Every dispute's history is rebuilt in the order events happened, so a late event lands where it belongs.
The months, after the 9 October run
- August: restated, 19 disputes, 1,058.72. It matched the statement on 3 October; corrections and won appeals since have lowered it.
- September: restated, 22 disputes, 1,840.63. On 5 October it was provisional: 21 and 1,711.16 against the statement's 23 and 2,078.69, because two losses had not arrived. It was final on 7 October at exactly 2,078.69, then restated on 8 October when an appeal was won.
- October: provisional, 12 disputes, 1,082.30. There is no statement yet.
What the processor's corrections changed
Anika's 9 October file corrected four disputes lost in late September, from the order total to the amount disputed. September fell by 177.25, to 1,840.63. The history now shows the corrected amounts in place of the old ones.
Why restate
Leo, you see every change the morning it happens. Grace, a month you reported stays marked restated, and the page shows the figures from the board day's run beside today's, so you can show the board both. The cost is a September figure that moved after the board pack. Under adjust it would have stayed, with the change booked in October.
Using the page
The months come first, then every dispute, open ones with the soonest evidence_due first. Filter by loss month and status; the total and Download CSV follow the filters.
How this was scoredThe AI model and how this result was scored
- Model
- Fictional example, no model was run
- Scored
- 19 minutes after submitting
- Issues
- Build is the real problems in the brief, the call it asks for and the stakeholder update. Code checks each one on its own test data the candidate never saw. A solved issue scores 100, and only code gives 100. An issue that is not solved gets partial credit from the AI reviewer: 85, 70, 50, 25 or 0. Every issue is the same share of Build. Each issue has one set made from the candidate’s data and one fresh set; the call, and an update that asks for a decision, have one set for each outcome. An issue that is not solved is placed by the AI reviewer on the Python and SQL levels, never above them. The full runs on the candidate’s data and on fresh data are shown and add no points.
- Review runs
- Every review runs twice. When the two runs disagree, two levels apart on any check or more than 5 points apart overall, a third run settles it and each check counts its middle rating. No result waits for a person.
- Audition, Can finance and the disputes team trust one chargebacks page?: 2 runs: 93.1 and 92.2, averaged.
- AI usage: 2 runs: 100 and 95, averaged.
- AI usage
- Five checks on the whole conversation with the assistant against the Plan, rated in two runs with a third when they disagree, shown beside the interview score and never counted in it.
- Evidence cited
- 6 of 7 gradable items
Session replay
Sample- Plan
- Build
- Writeup
Work captured
- PlanCaptured
- Work files26 files
- WriteupCaptured
- AI prompts1
Team notes
1 note- ML
Marta L., data engineering manager
Eight of ten issues proved by code, and the board-day call is right in every run. In the call I would ask how Sam would keep the history honest when the processor corrects an amount weeks later.
Timeline
- Invited
- Started
- Submitted112 min of 115 min used
- Scored19 min after submitting
- Your team’s decisionWaiting