AI Engineering: What Actually Happened

In July I published a process for taking an idea to a deployed product with AI. It ended with a promise: once I had enough evidence, I would report back on what worked, which parts turned into unnecessary ceremony, where the agents went off track and how much human involvement was actually required.

We have now run the process end to end on a real product, for a real client, through to a real handover.

This is that report.

The project

The product is Daybook: a personal command centre for a founder who was drowning across Notion, Apple Reminders, pen and paper and a wall of sticky notes, and who was actively pricing a personal assistant. Daybook captures tasks automatically from meetings, email and calendars, and puts one board in charge. The product story is in the case study. This post is about the process.

One honesty note before we start. We ran this build as a proving ground, with a client who already knew and trusted us. A stranger would not have extended that trust to a process that can feel like a black box from the outside, and several of the changes below exist for exactly that reason.

The hypothesis

One day of ideation. One day of brand. One day of UI and UX. One day of bootstrap. One day of development. One week of testing.

A week to build, a week to test. That seemed fair.

The scoreboard

Build effort: roughly one week, as planned.

Rework during testing: about five hours of development, total, across the entire testing period.

Elapsed time: six weeks.

The gap between those numbers is the most useful thing this project produced.

What each phase taught us

Ideation: the tech constrains the product anyway

The hypothesis said you no longer need to think about a tech stack up front. The agents can pick one as needed.

Wrong. The technical realities constrained the product almost immediately. The rule we landed on: the product directs the stack, never the reverse. You still have to look at the stack early, precisely so it cannot quietly start directing the product.

Brand: options are nearly free, so generate them

Our first brand concept was a single option, generated too casually. The client rejected it outright and he was right to. We regenerated two proper directions with real differences between them, he picked one, gave pointed feedback, and that became Daybook.

In this model, a second and third option costs almost nothing. Presenting one is laziness wearing a deadline as a disguise.

UI and UX: you cannot skip a step that later steps depend on

Leave overlapped on both sides mid-project, and we let the brand and UI review gate slip rather than lose the days. The first iteration of the app came out looking and feeling like nothing in particular, and we paid for the skipped gate in rework.

The feel of a product influences the interface far more than I would have argued before this build. The gates are now mechanical: booked into calendars at kickoff, named per phase, with a recorded sign-off. Not optional, not vibes.

Bootstrap: an hour, not weeks

Markdown files describing the stack, the environments, the pipelines and the rules, then let the agent build it. In about an hour we had deployed environments, automated tests, pull request checks and working pipelines. This is work we used to quote in weeks.

Nothing went wrong here. Some phases just work now.

Development: coding is solved, divergence is not

The spec said the email integration should pull one day of messages on first sync. One agent decided, in its own capacity, that fifty days made more sense.

Our functional tests passed, because the test account only had one day of email in it. The tests shared the agent’s blind spot. The client’s first demo opened onto roughly five hundred items in triage.

That bug is the whole argument for human QA in one sentence: an agent will make a locally reasonable decision that violates the spec, and it will not know it has done so. Someone has to be checking for divergence, not just for breakage. A different model reviewing the work helps. A human who knows what was meant helps more.

UAT: the bottleneck has moved to the humans

Here is where the six weeks came from.

Three weeks of client testing produced about five hours of development changes. The constraint was never the code. It was a busy founder finding the hours to properly use the product, and a product new enough that he was still discovering what to test three weeks after sign-off.

Compare a different project we ran through the same process: an admin dashboard automating a workflow the client already performed manually every week. She tested it in a fraction of the time, because she was not evaluating a new idea. She was checking that the machine did her job.

That is the qualification question we now ask before quoting a timeline: are we improving a process you already run, or building something you have never had? The second kind needs a testing window measured in weeks, and a client who knows that going in.

What we changed

Four things went straight into how we sell and run this work:

• A four-week MVP, not two. The build still lands in the first week and a half. The rest is the feedback loop that actually makes it a product.
• Gates diarised at kickoff. Every review meeting booked on day one, calendar events named after the phase, sign-off recorded. The client always knows where in the process they are, because the answer is in their calendar.
• Client commitment in writing. At this pace, the client is on the critical path. Minimum an hour a day for testing and decisions, or we schedule the engagement for when they have it. Condensed does not mean optional.
• Right-sized process. Late in the project, a colour change went through the full development gauntlet: planning, guardrails, tests, review. For a one-line change, that is ceremony. The size of the process should match the size of the change. The original post said exactly that, and we still had to learn it by doing.

One more, filed under humbling: we gave the client a live view of the whole delivery process so he could watch it instead of trusting it. He barely opened it. Not because it was not useful, but because we never made it part of his routine. There was no natural entry point pulling him back in. Transparency tooling only earns its keep if the client lands in it without effort, which is why every gate invite now carries the link: the calendar does the pulling.

The handover

The last phase of the process is the one I was least sure about, so we pushed it.

On the closing call we connected the client, who is not a developer, to the repository through Claude Code. He described a change in plain English, watched the code get written, watched the pull request open and the test suite run, and merged his own change to production. Minutes, and a few dollars of tokens.

The guardrails make this sane rather than reckless. Tests must pass before anything merges. We still review every pull request. And one line does not move: the AI can change the code, never the production database. He can restyle and extend his product freely, and his data cannot be harmed by a bad prompt.

So, how much human involvement was actually required? Humans made every product decision, reviewed every pull request, caught the one serious divergence and did all the real testing. The AI did close to everything else. That ratio felt right, and I would distrust anyone selling a version of this with less human in it.

The uncomfortable conclusion

The most honest sentence from our retro is one a dev shop is not supposed to publish: this entire project could probably have been a set of Claude skills.

Not as polished, not as owned, not as extendable. But the core problem, capture and triage of a person’s tasks, is increasingly solvable by wiring the tools someone already pays for into the model they already pay for. Some of the best work we do now is telling people exactly that before they spend money with us.

That is not a threat to this process. It is the first gate of it. Phase 0 exists to kill builds that should not happen, and a process that cannot recommend “do not build” is a sales funnel, not an engineering method.

The rule, updated

The July post ended with: use AI to accelerate the work, but do not use it as an excuse to skip the thinking.

After running it for real, I would add a second line: the coding is solved, so the schedule now belongs to the humans. Plan the people, the gates, the testing hours and the sign-offs, with the same rigour we used to reserve for the code, because they are the critical path now.

We are running the next builds as four-week engagements with everything above baked in. When those produce evidence worth sharing, I will report back again.