~/rena
← blog
AutomationAgentsWorkflowOperations

Automating the Boring 80%

The AI automation projects that worked weren't the ambitious ones. They were narrow, boring workflows with a human at the end — and that pattern is not an accident.

RRena Thomas·

I've now been part of a decent number of AI automation projects. The pattern in which ones succeeded is stark enough that I've stopped treating it as coincidence.

The ones that worked were narrow, boring, and had a human at the end. The ones that failed were ambitious, end-to-end, and tried to remove the human.

That's not a claim that agents can't do impressive things. It's a claim about where the value currently sits and why.

The failure mode: automating the exception

Almost every failed automation project I've seen made the same mistake. Someone looked at a workflow, found the part that consumed the most human attention, and pointed AI at that part.

Sounds right. It's backwards.

The part consuming the most attention is usually the part that's hard because it's genuinely ambiguous — the exception, the judgment call, the case where policy is unclear and someone has to decide. That's the worst possible first target. It's where models are weakest, where errors are most expensive, and where you have the least ground truth to evaluate against.

Meanwhile the actual volume is in the boring 80%: the requests that are unambiguous, repetitive, and mind-numbing. Nobody nominates those for automation because individually they're trivial. Collectively they're most of the work.

Concrete version from a healthcare engagement: the team wanted an agent to handle clinical triage decisions. Genuinely hard, high stakes, and we'd have been reckless to ship it. What actually shipped was an agent that gathered and normalized patient context from four different EHR systems before a human ever looked at the case. Not glamorous. It cut the time-to-decision on every single case, and when it was wrong a clinician noticed immediately because they were reading the output anyway.

Design for the review, not the decision

The pattern that keeps working: the agent does the gathering, drafting, structuring, and cross-referencing. A human does the approving.

This works because it inverts the error economics. In a fully automated flow, an error is an incident — wrong action taken, customer affected, incident review. In a review flow, an error is a small annoyance — someone reads the draft, notices it's wrong, fixes it. Same model, same error rate, three orders of magnitude difference in blast radius.

It also gives you a free labeled dataset. Every human edit is a training signal and an eval case:

async function recordReviewOutcome(draft: Draft, final: Draft, reviewer: UserId) {
  await db.reviewOutcomes.create({
    draftId: draft.id,
    accepted: deepEqual(draft.content, final.content),
    diff: computeDiff(draft.content, final.content),
    reviewer,
  })
}

Acceptance rate is now your headline metric. It's honest, it's cheap to compute, and it tells you exactly when the system is good enough to widen its autonomy — and in which categories. On one project we could see acceptance was 96% for one request type and 61% for another. That's the whole autonomy roadmap, right there in the data. Automate the first fully, keep the human on the second.

Which is the point: the review step isn't a limitation you're working around. It's the instrument that tells you when you can remove it. Teams that skip straight to full autonomy have no idea whether they should have.

Workflows over agents, until you need an agent

There's a strong pull toward building everything as an autonomous agent. Give it tools, give it a goal, let it figure out the path.

For most automation, that's the wrong tool. If you know the steps, encode the steps. Use the model for the parts that need language understanding and use ordinary code for the control flow:

// Deterministic pipeline; LLM used surgically
async function processIntakeRequest(req: Request) {
  const extracted = await llm.extract(req.body, IntakeSchema)   // language work
  const validated = validateIntake(extracted)                    // plain code
  if (!validated.ok) return routeToHuman(req, validated.errors)  // plain code

  const record = await crm.upsert(validated.data)                // plain code
  const draft = await llm.draftResponse(record, req)             // language work
  return queueForReview(draft)                                   // plain code
}

This is dramatically more debuggable, cheaper, faster, and more testable than an agent loop that reaches the same outcome by reasoning. When it breaks you get a stack trace and a line number, not a transcript to interpret.

Reach for a real agent loop when the path genuinely can't be known in advance — open-ended research, multi-step debugging, anything where step three depends on what step two discovered. That's a real category and agents are good at it. It's just much smaller than the enthusiasm suggests.

The rough test: can you draw the flowchart? If yes, write the flowchart. If the flowchart has a box that says "figure out what to do next," put an agent in that box and only that box.

The unglamorous prerequisites

Things that determined success far more than model choice:

Someone owns the workflow. If no human currently owns the process end to end, automating it produces an orphan system that degrades until it's quietly switched off. I've seen this three times. Ownership is a harder prerequisite than any technical one.

The inputs are reachable. A shocking amount of automation work is discovering the data lives in a system with no API, or an API with a 60-request-per-minute limit, or a spreadsheet someone maintains by hand. Standardized tool protocols helped a lot here, but they don't conjure an API that doesn't exist.

There's a fallback path. Every automated workflow needs an obvious, well-lit route to a human — for the agent when it's uncertain, and for the user when they want out. Systems without an escape hatch generate the worst support experiences I've ever seen, and the reputational damage outlives the efficiency gain.

The metric predates the automation. If you can't state the current baseline — cases per day, time to resolution, error rate — you will not be able to prove the automation helped, and the project will be judged on anecdote. Measure for two weeks before you build anything.

What this looks like a year in

The successful deployments I've watched follow the same trajectory: start with one narrow workflow and a mandatory review step. Get acceptance rate high in a subset of categories. Remove review for those categories only. Keep review everywhere else. Expand to the adjacent workflow.

It's slower than the pitch deck. It also still exists a year later, which is more than I can say for the ambitious versions.

The thread running through all of this: automation is a systems and process problem wearing an AI costume. The model is rarely the constraint. The measurement and the ownership almost always are.

← all posts