A raw AI model (the base GPT-5 or Claude Sonnet) is a prediction engine. When you drop 20 interview transcripts into their blank chat box and ask for a presentation, you get a wall of text and a few invented quotes, without the nuances of your context. That blank chat box experience is AI without a harness.
The harness is the software wrapper we build around the model to make it behave like a thinking partner who knows your context and constraints. It sets the rules the model works under, what it remembers, what happens when something breaks, and which buttons you get in the UI. Harness is an engineering word, but shaping one could be design work. This third part of the AI Experience Design series will cover the six affordances that help shape it.
There are six layers that define the shape of an AI harness. To see how they work together, imagine setting up an agent to synthesize twenty raw user interview transcripts into a final presentation deck.
The Context Layer: This is where the AI model makes sense of the situation you provide. Before the agent starts, it needs to assemble a reality that more closely matches your project. In practice, this is the hidden system prompt, a markdown file containing your user research protocols, project files, and the raw transcripts. You are establishing the real-world constraints: “You are a senior design strategist operating strictly within these parameters.”
Loop Control: This sets the agent’s autonomy. Rather than asking for your permission after every single file, you set a continuous loop for the agent to read, tag, and cluster all 20 transcripts on its own before it comes back to you.
Gates: These are the intentional human-in-the-loop moments. Before the agent starts generating the actual slides, it stops and asks: “I found 3 emerging themes. Which one should we highlight for leadership?”
State and Memory: At the deeper layers, the harness tracks both the immediate health of the context and long-term continuity. The state is the indicator that says “Synthesizing transcript 12 of 20...”, so you know the model is working. Memory is the agent recalling that, from a project 3 months ago, your team prefers bulleted lists over complex spider charts and automatically applying that formatting.
Failure Handling: This is the safety net for the task. If the agent tries to pull live quantitative data to support the research and the API times out, the session doesn’t crash. Instead, it asks: “I can’t access the analytics tool right now. Should I proceed with qualitative data only?”
Tools Access: This is what the model can reach for to do the work. Once the outline is approved, the agent calls a slide-generation tool to actually build the deck, and a Slack integration to message you the final link.
None of these mechanisms are traditionally design’s territory, which is probably why they’ve stayed on the engineering side of the room. However, how they are designed shapes the user experience.
With that user experience in mind, I mapped the 6 against two questions: what's already within design's reach and what we can achieve by working with engineering.
Context assembly could be entirely design’s call, since deciding what goes into the model’s window is a judgment about what matters to the user. The other four could be a collaboration between design, engineering, and data teams. With loop control, design can argue about where the agent reports in, though not about how many times it iterates. Gates work the same way: placement could be the designer's call, similar to how it appears in the user's UI. On state, the question of what gets remembered belongs to design, not to storage. And failure handling gives us what the person sees when it breaks, with the retry logic staying on the other side. Those dashed ones are where it gets interesting. They’re the places where we have something to say and no obvious way to say it. We can call them our sphere of influence.
Looking at these more closely, each one contains two distinct types of design work.
One kind is written—the system prompt, the checkpoints, the approval map, the memory rules, the failure cases, the tool descriptions. They live in md files and sit below the line of visibility where the agent reads it.
The other kind is shown in the UI layer: what the agent was told, the wait and stop buttons, the confirm moment, a memory the person can actually edit, the recovery screen, which tool fired. All of that lives above the line, where somebody can see it and act on it. This is where we have UI patterns like stream-of-thought cards and human-in-the-loop confirmations.
The relationship between them matters more than I realized when I started drawing. The deeper written layer defines what the shown layer would be. If the agent was never instructed to hold onto a disagreement between two steps, there’s nothing left to surface by the time we get to the screen. So, in a way, the writing comes first, and it determines exactly how much of the showing is available to us at all. In that sense, the written system rules and prompts are the upstream design layers we need to pay closer attention to.
Key Takeaways for Design Leaders
If you are taking these concepts back to your team, here is where to focus:
The UI is downstream of the “written” layer; writing is a prerequisite for the UI. If we aren’t shaping the system prompts, memory rules, and failure cases below the line, we severely limit what we can design above it.
A new operational framework: The 6 layers of the harness give us a concrete vocabulary to use with engineering and product partners. It moves the conversation from an abstract “make the AI feel magical” to specific, assignable architectural components.
Clarifying our sphere of influence: We might not build the backend database, but we must define the State and Memory requirements. We might not write the API retry logic, but we must design the failure-handling fallback UI.
Designers must become system writers: The skill set of an AI experience designer is expanding deeply into prose. Shaping the context layer with markdown files and behavioral constraints is now a core design task.
And that’s a wrap for this issue. Until next time, take good care of yourself and your loved ones.