Context
Formee's student support was bottlenecked on async email + chat. Agents had no way to reach a student live, and when they did, they'd spend half the call hunting through the database for context. The CRM was a second-class surface — not the primary way anyone worked.
What I built
A real-time agent desktop with three layers stitched together:
- Voice + chat transport: WebRTC for browser-to-browser calls, Asterisk for PBX routing, and a Twilio trunk for outbound to mobile numbers. A single unified session object on the backend tied a chat thread and a voice leg to the same customer record.
- Live agentic layer: LlamaIndex over the active conversation transcript (ASR + chat history) plus the student DB. Every few seconds the agent desktop refreshed a panel of suggested replies, grounded in whatever was on the screen and in the database — with explicit safety guardrails on anything PII-adjacent.
- Post-call analysis: After the call ended, an agentic pipeline scored the conversation, pulled action items, and attached them to the student record. Leads got automatic priority tags.
Architecture highlights
What I learned
The integration boundary — Asterisk ↔ Twilio ↔ LlamaIndex — is where latency and failures hide. If I were building this again, the very first thing I'd add is a probe on every edge with an independent health check. "The voice leg is up" and "the dialplan is up" are not the same statement.
The other lesson: live LLM suggestions work only when they're fast and non-intrusive. A panel that refreshes with a spinner, or takes longer than the agent's own typing speed, gets ignored. Latency is the UX.
This is the project I point to when people ask what "applied AI" actually looks like in production. It's not a model — it's a pipeline of models glued across three external systems with guardrails on each edge.
Stack
Next.js · WebRTC · Express.js · Asterisk · Twilio · Python · LlamaIndex · OpenAI API · PostgreSQL