AI Engineering: What Actually Happened

In July I published a process for taking an idea to a deployed product with AI. It ended with a promise: once I had enough evidence, I would report back on what worked, which parts turned into unnecessary ceremony, where the agents went off track and how much human involvement was actually required.

We have now run the process end to end on a real product, for a real client, through to a real handover.

The project

The product is Daybook: a personal command centre for a founder who was drowning across Notion, Apple Reminders, pen and paper and a wall of sticky notes, and who was actively pricing a personal assistant. Daybook captures tasks automatically from meetings, email and calendars, and puts one board in charge. The product story is in the case study. This post is about the process.

One honesty note before we start. We ran this build as a proving ground, with a client who already knew and trusted us. A stranger would not have extended that trust to a process that can feel like a black box from the outside, and several of the changes below exist for exactly that reason.

The hypothesis

One day of ideation. One day of brand. One day of UI and UX. One day of bootstrap. One day of development. One week of testing.

A week to build, and a week to test seemed like a fair split if we did our job right.

The scoreboard

Build effort: roughly one week, as planned.

Rework during testing: about five hours of development, total, across the entire testing period.

Elapsed time: six weeks.

What each phase taught us

Ideation: the tech constrains the product anyway

The hypothesis said you no longer need to think about a tech stack up front. The agents can pick one as needed.

The technical realities constrained the product almost immediately.

The rule we landed on: the product directs the stack, never the reverse. You still have to look at the stack early, precisely so it cannot quietly start directing the product.

Brand: options are nearly free, so generate them

Our first brand concept was a single option, generated too casually. The client rejected it outright and he was right to. We regenerated two proper directions with real differences between them, he picked one, gave pointed feedback, and that became Daybook.

In this model, a second and third option costs almost nothing. Presenting one is taking an unnecessary shortcut.

UI and UX: you cannot skip a step that later steps depend on

Leave overlapped on both sides mid-project, and we let the brand and UI review gate slip rather than lose the days. The first iteration of the app came out looking and feeling like nothing in particular, and we paid for the skipped gate in rework.

The feel of a product influences the interface far more than I would have argued before this build. The gates are now mechanical: booked into calendars at kickoff, named per phase, with a recorded sign-off.

Bootstrap: an hour of well directed time

Markdown files describing the stack, the environments, the pipelines and the rules, then let the agent build it. In about an hour we had deployed environments, automated tests, pull request checks and working pipelines. This is work we used to quote in weeks.

The only thing that went wrong here is that when the agents were blocked by needing deployment secrets, rather than stopping the process and asking for human intervention, they got creative and managed to generate some of them through CLI connections.

While this is a bigger argument for AI being able to do everything, this was a massive overstep and required us to purge and rotate the secrets, putting them in secure environments that the agents do not have access to, but know how to instruct deployment pipelines to fetch.

Development: coding is solved, but entropy is accelerated

The spec said the email integration should pull one day of messages on first sync. One agent decided, in its own capacity, that fifty days made more sense.

Our functional tests passed, because the test account only had one day of email in it. The tests shared the agent’s blind spot. The client’s first demo opened onto roughly five hundred items in triage.

That bug is the whole argument for human QA in one sentence: an agent will make a locally reasonable decision that violates the spec, and it will not know it has done so. Someone has to be checking for divergence, not just for breakage. A different model reviewing the work helps. A human who knows what was meant helps more.

UAT: the bottleneck has moved to the humans

Here is where the six weeks came from.

Three weeks of client testing produced about five hours of development changes. The constraint was never the code. It was a busy founder finding the hours to properly use the product, and a product new enough that he was still discovering what to test three weeks after sign-off.

Compare a different project we ran through the same process: an admin dashboard automating a workflow the client already performed manually every week. She tested it in a fraction of the time, because she was not evaluating a new idea. She was checking that the machine did her job.

That is the qualification question we now ask before quoting a timeline: are we improving a process you already run, or building something you have never had? The second kind needs a testing window measured in weeks, and a client who knows that going in.

What we changed

Four things went straight into how we sell and run this work:

• A four-week MVP, not two. The build still lands in the first week and a half. The rest is the feedback loop that actually makes it a product.
• Gates diarised at kickoff. Every review meeting booked on day one, calendar events named after the phase, sign-off recorded. The client always knows where in the process they are, because the answer is in their calendar.
• Client commitment in writing. At this pace, the client is on the critical path. Minimum an hour a day for testing and decisions, or we schedule the engagement for when they have it. Condensed does not mean optional.
• Right-sized process. Late in the project, a colour change went through the full development gauntlet: planning, guardrails, tests, review. For a one-line change, that is ceremony. The size of the process should match the size of the change. The original post said exactly that, and we still had to learn it by doing.

One more, filed under humbling: we gave the client a live view of the whole delivery process so he could watch it instead of trusting it. He barely opened it. Not because it was not useful, but because we never made it part of his routine. There was no natural entry point pulling him back in. Transparency tooling only earns its keep if the client lands in it without effort, which is why every gate invite now carries the link: the calendar does the pulling.

The handover

The last phase of the process is the one I was least sure about, so we pushed it.

On the closing call we connected the client, who is not a developer, to the repository through Claude Code. He described a change in plain English, watched the code get written, watched the pull request open and the test suite run, and merged his own change to production. Minutes, and a few dollars of tokens.

The guardrails make this sane rather than reckless. Tests must pass before anything merges, and one line does not move: the AI can change the code, never the production database. He can restyle and extend his product freely, and his data cannot be harmed by a bad prompt.

So, how much human involvement was actually required? Humans made every product decision, caught the one serious divergence and did all the real testing. The AI did close to everything else. That ratio felt right, and I would distrust anyone selling a version of this with less human in it.

The uncomfortable conclusion

The most honest sentence from our retro is one a dev shop is not supposed to publish: this entire project could probably have been a set of Claude skills.

Not as polished, not as owned, not as extendable. But the core problem, capture and triage of a person’s tasks, is increasingly solvable by wiring the tools someone already pays for into the model they already pay for. Some of the best work we do now is telling people exactly that before they spend money with us.

That is not a threat to this process. It is the first gate of it. Phase 0 exists to kill builds that should not happen, and a process that cannot recommend “do not build” is a sales funnel, not an engineering method.

Hypothesis versus reality

The hypothesis was one day of ideation, one day of brand, one day of UI and UX, one day of bootstrap, one day of development, one week of testing. A week to build, a week to test.

The build half stuck to the plan as well as any software project can. Roughly one week of effort produced a deployed, tested product, and the bootstrap day came in closer to an hour. Ideation, brand and UI needed their day each, but for reasons the hypothesis did not predict. Not because the work was slow, but because each one gates the next and skipping the gate bit us in the behind later.

The testing half did not hold. One week became three, and elapsed time became six. Not because the product was broken, and not because there was much to fix. Three weeks of client testing produced about five hours of development. The week we budgeted was a week of our effort, and testing is not our effort. It is the client’s, spent in the gaps of a working life, on a product new enough that they are still learning what to ask of it.

So the hypothesis was right about the machine and wrong about the calendar. Every estimate in it was an estimate of engineering time, at a point where engineering time had stopped being the thing that determines when a product ships.

The rule, updated

The July post ended with: use AI to accelerate the work, but do not use it as an excuse to skip the thinking.

After running it for real, I would add a second line: the coding is solved, so the schedule now belongs to the humans. Plan the people, the gates, the testing hours and the sign-offs, with the same rigour we used to reserve for the code, because they are the critical path now.

We are running the next builds as four-week engagements with everything above baked in. When those produce evidence worth sharing, I will report back again.