Why candidates who pass every visible test still fail
The pattern in the reports is consistent. The candidate types fast, the assistant writes most of the code, every test on screen goes green, and the score comes back too low. One candidate clicked run, saw everything pass, and only found out at the end that other tests existed. Another scored about 650 of 1000 and was rejected the next day.
Our read: the visible tests check that you built a feature. The hidden ones check that you built the feature the product manager described, edge cases included. The spec is not written down anywhere. It exists only in a ten-minute chat with an AI that answers what you ask and nothing more. Candidates who accepted the first answer built something plausible and missed the rules the hidden tests are about. The assessment is grading requirements work as much as coding, and it never says so.
The first feature: search and filters
The opening prompt is a complaint: a page shows too much. The spec you are meant to pull out is a search over item titles, case-insensitive and matching part of a title, plus filters by item type and by availability. Filters combine with AND, no filters means every item, and the filtering happens on the backend, with frontend controls that send the values. The tests one candidate listed: case-insensitive search, item type, availability, combined filters, no filters, and no matches.
The catch is availability. It is not a column you can filter on. The application works it out from loans and holds, so a status filter that guesses from the loans table disagrees with the status the app shows. Find the existing logic and reuse it.
The bug, without the assistant
Claude is switched off for one section. A page shows a wrong number or an unfiltered list, and the cause is in the backend. In the version reported in October, it sat in a metrics file: the query that loads the items ran before the filter was added, so the filter changed nothing. The fix was moving it a few lines down. Small, once found. Finding it means tracing a symptom from the page through the frontend into the Python code on your own.
The second feature: notifications
The second spec is longer and the time is shorter. One candidate had about thirty minutes left for it. The feature: notify a patron when an item they hold becomes available, in hold queue order, only if they are eligible, where overdue items and account status count against them. Then remind borrowers before a loan is due and when it becomes overdue. Never send the same notification twice. One example spec: a reminder 24 hours before the due time and one at the due time.
Every rule here is a question to ask the product manager. Is an ineligible patron skipped, or do they block the queue? How many holders hear about one copy? What happens to a reminder when the due date changes? The candidates' advice is to ask them all in one structured message instead of one at a time, and to copy the answers into the notes area, because the chat is capped.
What is confirmed, and what is not
- Confirmed: CodeSignal, 90 minutes, five sections; chats with an AI product manager capped at ten minutes; a library lending app; search and filters first; a backend bug fix without the assistant; a notification feature with hold and due date notices, idempotency and async processing; hidden tests deciding the score. Each is in more than one report or in both a report and the site's digest.
- Reported once: the metrics file as the bug's location; Claude Haiku 4.5 as the model; the 600 point floor from a recruiter; the guidance-only assistant in September; pasting the product manager chat into Claude being blocked; terminal output not pasteable into the assistant.
- Not known: how the score is weighted between sections, and whether the requirements conversation itself is graded.
How we would prepare
- Practise getting a spec out of someone. Inputs, outputs, edge cases, what happens on a retry, what happens on a conflict. Write the questions before you send any, and send them together.
- Build a search and filter endpoint with tests in twenty minutes. Each parameter alone, all together, mixed case, no parameters, no matches, with availability computed by the app's own rule.
- Trace a wrong number to its line without help. Pick an unfamiliar repo, break a filter, and practise following the value from the response back to the query.
- Know how to make a notification job idempotent. Give each notification an identity: the hold or the loan, the recipient, the kind, and the due date it was sent for. Check it before sending. Then run the job twice and see that nothing doubles.
- Submit after every finished feature. Candidates report CodeSignal losing code and the final test run freezing the page.
Nyrion's Instacart section starts with Library Lending System, built from these reports. Search and filters come first, over a status the app computes. Then the metrics bug, with the item query running before its filter. Then hold notifications with eligibility, and reminders that survive being run again. The domain, the search rules and the metrics bug are what candidates describe. The rest is ours: the exact notification rules, a second planted bug, and jobs in place of a Celery worker. One real difference: you get the whole spec in the statement, with no product manager to question, and our assistant stays available in the bug section, so leave it closed there if you want the real conditions. It runs in plain Python, without a frontend.
How this round compares with other companies' versions, and what every version grades: the AI coding interview guide.
Frequently asked, briefly answered
Who gets this assessment?
Candidates for software engineering roles, often after a referral. The invite is titled the Instacart Full-Stack Engineer Assessment, and backend applicants report receiving the full-stack version too. One candidate had two weeks to take it.
Which AI assistant is in the editor?
Claude, inside CodeSignal. One late-August report names Claude Haiku 4.5, and candidates describe it as slow. It is switched off for the bug fix section. You cannot paste your product manager conversation into it, so you restate the requirements in your own prompt.
Can I choose the tech stack?
The AI product manager may ask for a preference and offer a list. One candidate picked Python with FastAPI and React for the first feature, and Celery with Redis for the notification work. The bug fix still meant reading both the frontend and the backend.
How is it scored?
By tests, including ones you cannot see. Candidates report passing every visible test and still scoring low. Recruiter guidance put a viable score at about 600 out of 1000, and one candidate at about 650 was rejected the next day, so treat 600 as a floor, not a pass.
Does code carry over between sections?
It depends on the version. Some reports say the first feature's code is gone by the last section; an August report describes connected sections in the same application, where later work exposes problems in earlier changes. Check before you start, and submit after every finished feature, since CodeSignal can lose work.
Is there a version where the AI will not write code?
One September report describes an assistant that explained code and helped narrow down bugs but refused to edit files or give answers. Read the assessment's instructions on what the assistant may do before you plan around it.
Practice problems tagged for Instacart
Browse the catalog and open the Instacart section. No login required to look; sign in when you want to solve one.