The interview room your candidates use.
The real room, running live in your browser. Follow the steps of a sample Audition, or explore on your own. Code runs, the AI assistant answers and nothing you do is recorded.
Plan without AI. Read the brief on the left and write the plan on the right. The AI assistant is off in Plan, and the Plan locks when the candidate continues.
The interview room
Forward-Deployed AI Engineer · Sample interview
Phase 1 of 3: Plan (no AI)
Read the Business Problem first. It's the brief for everything you'll do. In Build you can work in Python, SQL, JavaScript, TypeScript, JSX or TSX.
This phase is your own thinking, so the AI assistant is off here. In the plan editor, write:
- What the real question is, in your own words
- Three or more assumptions your approach depends on
- What you'd check first, in priority order
- Two ranked questions you'd ask if you could
- The risks you can already see
There is no minimum time or word count. Write as much or as little as you need, then continue whenever you're ready. A brief or template-only Plan is allowed and will simply remain part of your work evidence.
When you begin Build, your plan locks and you cannot come back to it.
Can the assistant change orders before they ship?
The situation
You are a forward deployed engineer at an AI company. Wrenbury, an online store for home goods and clothing in the US, has added your company's AI assistant to its help chat.
The most common request is changing an order before it ships: a new delivery address, canceling one item, or swapping a size. Today the support team does each one by hand. Ines Dalgaard, Head of Customer Support, wants the assistant to do them. She worries about packages going to the wrong place, changes made after packing starts, and customers being told things that are not true. Ines decides if the assistant goes live on its own. You recommend.
The model is a helper, not the judge. It sometimes misreads an order number, skips your tools, follows instructions hidden in a message, or promises something the store never offered. Your code decides what happens and what the customer sees.
What the business needs from you
Build two tools the model calls:
get_order(order_id, email)returnsfoundand the order's items (sku,product,size). Never the address, the email or the total.change_order(order_id, email, change, sku, new_value)applies the rules below and returnsdecisionandrefund.changeisaddress,cancel_itemorswap_item. For an address,new_valuehasstreet,city,zipandcountry. For a swap, it is the newsku.
The rules. The first one that applies decides:
need_details: no order matches the number and email (ignore case), theskuis not in the order (for a cancel or a swap), orchangeis not one of the three.too_late: the chat is later thanpacking_starts_atminus your packing cutoff.handoff: the order total is over $500.00.not_allowed: a new address outside the US, or outside the order's city if your address scope issame_city.- Swaps:
handoffif your swaps setting ispeople.not_allowedif the newskuis out of stock or has a different price. changed. A canceled item refunds itsprice_paid. Every other refund is 0.
Your three settings. Pick one option for each. Any allowed choice can earn full marks.
| Setting | Options |
|---|---|
packing_cutoff_minutes | 30 or 60 |
address_scope | same_country or same_city |
swaps | assistant or people |
Your code writes every customer message, from the decision, never the model's words. Confirm only what changed, give a refund with two decimals, and never promise a delivery date.
A chat is corrected when your code had to step in: the model never called change_order, its arguments matched
no order or item, or its reply said something the result did not (a change that did not happen, a different refund
or a delivery day).
The go-live call: alone if the corrected share is at most 0.20. Otherwise with_review: people approve each
change.
Calling the model. Use from buildinterview import model, stand_in (examples in guide/). Try model on a few
chats, then run every chat on the free stand_in. Send each chat as its own message and include its chat_id.
Partway through, a stakeholder will send one update. Decide what it changes.
The data
data/DATA_DICTIONARY.md explains every column. Times are store time. Money is in US dollars.
chats: one row per chat on October 12, 2026. This demo has 12; an Audition has about 150.orders: one row per order, withpacking_starts_at.order_items: one row per item, with the price paid.stock: the sizes a customer can swap to, with their price andin_stock.
What to hand in
Your main file writes:
outputs/settings.csv(field,value): your three settings.outputs/change_results.csv, one row per chat:chat_id,order_id(empty if none matched),decision,refund,corrected(trueorfalse) andmessage.outputs/summary.csv(field,value):chats,changed,too_late,not_allowed,handoff,need_details,refunds_total,corrected,corrected_share(a fraction like 0.25) andgo_live.outputs/review.htmlfor Ines's team, with the data inside: every chat and its decision, the chats that need a person, and each corrected chat with what the model tried. Keep it simple; it is not judged on design.
Write 0, not an empty cell. No commas or double quotes in a message.
Also write docs/stakeholder_writeup.md for Ines (up to 300 words: your call, the model's mistakes, your settings,
what to watch) and docs/technical_writeup.md for the next engineer (up to 400 words: how it works and how you
tested it).
How it is checked
- Decisions, counts and
correctedmust match exactly. Money within 0.01, shares within 0.00001. - After you submit, we run your main file again on fresh data with different values and check it the same way. So compute everything from the files at run time.
- Reviewers read your Plan, code, review page and notes.
Time and AI
Plan 15 minutes (AI off), Build 75 minutes, Writeup 20 minutes: about 110 minutes in total. AI Assist is on in Build and Writeup.
The interview room is built for a desktop browser, as candidates use it. Scroll sideways to explore, or open this page on a computer.