Voice Agents: How and Why

Event / Source

SixFactors / Channel · Voice Agents Workshop · September 8, 2026

What it was about

A hands-on look at how voice agents actually work, from prompts, knowledge and memory to tools, human handoffs and testing before they’re put in front of customers.

What I’m keeping

Building the agent is the easy part. Building one you can trust is the real work.

The most interesting part for me wasn’t the voice technology itself. It was everything surrounding it: defining what the agent should not do, deciding when it should hand off to a human, and testing how it behaves across situations you didn’t anticipate.

3–5 things I learned

• Start with a very small job. Give the agent a defined role, boundaries and specific things it should not do.

• Knowledge, memory and tools are different. Knowledge gives the agent information about the business; memory carries useful information across interactions; tools let it actually take actions.

• Human handoff has to be designed into the system. The AI doesn’t automatically know when a person should take over. That escalation path needs to be deliberately built.

• Conversation design is UX design. With a voice agent, you’re not designing buttons and screens. You’re designing the conversation a customer experiences.

• Testing isn’t a final step. It’s part of the product. Dean showed how scenarios, personas, scorecards and evaluators can test an agent across many different customer situations before and after launch.

How this changes my work

This reinforced something I’m increasingly interested in: the valuable skill isn’t simply knowing how to build an AI agent. It’s knowing how to test one. What happens when the customer goes off-script? When should the AI stop? When should a human step in? And how do we know the system is actually behaving the way we intended?