Five team members planning Medical Diagnostic AI project roles and tasks on whiteboard

I Lead My AI Models Like a Design Team

IT Management Leave a comment

TL;DR: The hard part of working with multiple AI models is not prompting. It is project leadership. And design leads already have the muscle for it.

A recent headline asked whether AI agents are starting to act like employees, and who, exactly, is supposed to manage them.

The question is not rhetorical. For my team, the answer is me.

The team I did not plan to build

I was hired as a design lead at a company in a heavily regulated industry. The brief was large: design a new product from zero and improve the existing one. The infrastructure to do it did not exist. No design system. No research repository. No accessibility practice.

And no team. Just me.

So I started hiring. Not people; that headcount did not exist. I built the team out of software.

The first hire was the design system, and it is still the biggest one: tokens, components, documentation, with accessibility built into it as a working practice rather than bolted on as an audit. Then came the supporting roles, one at a time, each as a skill with a defined job. The most recent hire was a research brain: a persistent memory that holds decisions, findings, and context, both within each project and across all of them, so nothing has to be re-derived from scratch. The newest hire is content, and honestly, it is not strong yet. I am still training it.

Somewhere along the way, I stopped prompting models and started leading them.

Not as a metaphor, but as a daily reality. Once the roles were established, my time shifted from writing prompts to assigning work, reviewing output, and deciding what happens next. Roles, handoffs, review loops, a shared source of truth, a human approval gate. That is a team structure. Mine happens to run on software.

The question the industry cannot answer

The framing everywhere right now is that AI agents are becoming coworkers. They take assignments, use tools, produce work, and hand it back. Some can plan, some can code, some can review, and some can act semi-independently across a workflow.

So the next question is obvious: if they act like a team, who is directing them?

Most organizations do not really have an answer yet because they are still treating this as a technology problem. Buy the platform. Pick the model. Deploy the agents. Improve the prompt. Add a bigger context window.

Then discover that a pile of capable workers with no roles, no brief, no shared memory, and no review process is not a team. It is a liability with good demos.

The engineering world has already named part of the solution: Planner-Executor-Critic. One agent decides what should happen. Another does the work. A third inspects the result and sends back actionable feedback. Frameworks like CrewAI go further: before an agent does anything, you define its role, goal, and backstory. You do not just ask it to perform. You cast it.

Read those patterns as a designer, and the disguise falls away. A planner writes the brief. An executor does the build. A critic runs the review. A human makes the call.

That is not a novel AI architecture. That is a project timeline.

My actual loop

Here is the setup I currently use, though I am constantly reviewing it for optimization. I keep a small standing team of three models in fixed roles.

Fable plans. It analyzes the project, thinks through the strategy, writes the plan, and sets up the skills the work will use.

Opus builds. It executes against the plan and does the heavy lifting.

Codex verifies. It checks the code, logic, and implementation details.

Then Fable reviews what came back.

And I drive the loop with one phrase: next step.

This gives me control without any additional prompting. I am not re-briefing the models every time. I am not copying context back and forth. I say next step, and the loop picks up exactly where it left off. It moves to the following item in the plan. It builds. It verifies. It reviews. Then it waits for the human gate again.

The roles are fixed, but the models are not sacred. Fable is my planner and reviewer right now because its judgment is strongest. Opus is my builder. Codex is my verifier. If that changes, I recast the seat. I do not rebuild the process.

Not perfect, but so much better

It works much better than anything I ran before it, and the difference comes down to two mechanics that keep the loop honest.

First, every review must end with a verdict. Approved or sent back with findings attached. It is a design critique that is required to reach a conclusion.

Second, the state of the work lives in one place where every role can read: the plan, what shipped, what changed, what was deferred, and why. That shared memory is the reason next step works. Nothing has to be re-explained because nothing lives only in someone’s head.

Here are a couple of real-life examples:

The check that lied

A modal component came back marked done. Every automated check passed. The fullscreen variant was supposed to fill the screen, and the checks confirmed the right settings were in place. Then I looked at the actual rendered screenshot. It was an inset box floating in the middle of the page, with the app’s navigation still visible around it.

The work had passed review without anyone looking at it.

The fix took an afternoon. The rule it produced is permanent: no review closes without looking at the rendered result. Checks verify claims. Eyes verify work. Any design lead who has approved something off a Jira ticket instead of the actual screen knows exactly this lesson.

The mistake still got through, but the process caught it; the fix cost an afternoon, and the rule means it cannot happen quietly again. The next time there is a visual mismatch, I ask what in the system let it through before I ask how to fix it. Fixing it is a given. Stopping it from happening again is the goal.

The sixty-one-agent afternoon

And here is the other side: what the team makes possible once the review discipline holds. I recently needed structured documentation for every component in the design system’s catalog: 64 components, one locked template.

The thinking happened once, upstream: a planner model shaped the template, the plan, and the quality gate. Then I dispatched 61 agents in parallel to execute it, one per undocumented component, each writing against the template, each output validated by the gate. They all came back covered and consistent.

The economics follow the org chart. Planning and review deserve the strongest judgment I can afford. The execution seats do not; they can be smaller, cheaper models, or even open-source agents. I have not experimented enough yet to know how far down that goes, but the structure is what makes the experiment safe to run: the gate does not care which model did the work.

Learning the hard way

My early loops overspent tokens embarrassingly. I was paying senior experts to do busy work. The cost was not in reading context back — reading is cheap. The cost was in re-deriving it. A frontier model that starts work without shared memory spends its most expensive resource, reasoning, on reconstructing decisions that were already made: re-checking things nothing had touched, producing polish nobody asked for.

Long sessions taught me the same lesson from the other direction. The longer a single session runs, the more each step costs and the blurrier the judgment gets. The context fills with everything that ever happened instead of what matters now. The fix is the one every team already knows: do not depend on heroic marathon meetings.

This is exactly why that next step command works. Because the state of the work lives outside any one session, I can close a session at any point, open a fresh one, and pick up exactly where we left off. Short sessions, durable state.

The correction was not a better prompt. It was better management, in two directions at once.

The System

First, the system. This is what the research brain and the shared source of truth are for. Decisions get looked up instead of figured out again. Progress survives the end of a session. The senior experts stop doing busy work, and the expensive seats focus their reasoning on new problems only.

Second, my role. I stopped accepting and reviewing every individual decision and started holding the overall execution plan instead. I watch the shape of the work, not every keystroke. I stop and redirect when something drifts, evaluate the result, and approve the next move. The models do not need me in every meeting. They just need me at the gate.

Every design lead who has orchestrated a complex project with other designers goes through this exact transition: the shift from checking every pixel to trusting the process you built. It turns out the transition is the same when the team is software. It just happens faster, and the tuition is billed in tokens.

Why a design lead already knows how to do this

Look back at that loop and notice something: there is almost nothing in it a design lead does not already do every week. The brief, the casting, the critique, the verdict, the gate.

I treat my models like a team of specialists, not generalists. Each seat has a different skill level and a different responsibility. The strongest judgment does the planning and the reviewing, other models do the execution, and every result passes model review before it reaches my approval. I did not learn that from an agent framework. I learned it by leading design projects.

The same instinct shows up in how I built my agentic design system, Lumen. I think of it as a council of role-specialized agents. One adds components. One acts as an accessibility specialist, validating contrast ratios and screen-reader behavior. Another holds the content style, ensuring every label and empty state sound like the brand. One surfaces tradeoffs and risks before anything is committed. One runs the final quality pass.

Each has a narrow role. Each has a clear handoff. The workflow sits on top of an atomic-design foundation. That structure matters because agents, like people, do better work when the system around them is legible. A vague project structure creates vague output. Clear orchestration compounds.

What this changes

For designers, the reassuring part is this: the leverage skill in this new era is not prompt craft. It is judgment. Role clarity. A real brief. A useful critique. A shared source of truth. A disciplined approval gate. Those skills transfer directly.

For leaders, the pattern is bigger than one person’s setup, and the data says the gap is already here. A new IBM study of 2,000 technology executives found that two-thirds of CIOs and CTOs are already accountable for AI systems they do not fully control. Seventy percent say teams across the business are deploying technology faster than IT can track, and 77% say adoption is already outpacing governance. Surveyed organizations averaged 54 AI-agent incidents last year that required human correction.

Read those numbers as a project lead rather than a technologist, and they describe something very familiar: unclear roles, missing handoffs, unmanaged quality, and no one owning the final decision. We know how to solve that. We just have to apply the same leadership muscle to software that argues back.

The models will keep changing. Mine changed twice this year. The orchestration did not.

We spent the last few years trying to get better at talking to models. The more durable skill turned out to be organizing them. Designers who have led complex projects have been training for this the whole time. We just did not know the team would show up as software.

References

 

 

View at Medium.com

Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.

Leave a Reply