AI Avatar Research

Stack

Next.js, OpenAI, whisper.cpp (transcription)

What It Does

Doing exploratory work for a potential client, I created a testbed and functioning avatars using a variety of AI audio and video generation services including ElevenLabs, HeyGen and Tavus. I was able to train and get up and running a functioning custom-trained LLM in my own style of speech, my own voices and AI-generated video using several versions of the HeyGen and Tavus APIs. I learned a lot on this project and hope in the future to have an opportunity to test adjacent open-source technologies to compare performance and cost.

How it works

It's a website and backend connected to several interchangeable AI layers under the hood.

The knowledge base isn't a vector store. It's six numbered markdown files plus three short scripts, loaded into the system prompt at request time. The structure forces editorial discipline: there's no automatic embedding to lean on, so what's in the knowledge has to actually be the right thing. The first substantial pass at the knowledge files came from a 61-minute recorded voice training session transcribed locally with whisper.cpp and the large-v3-turbo model. The transcript got distilled into the markdown rather than dumped in raw.

Built with

• Web app: Next.js, deployed to a small private URL during the POC phase.
• Model: OpenAI (provider chosen for streaming + tool-use latency at the time).
• Knowledge: Markdown files in data/knowledge/, plus a few scripted opener / closer / pivot scripts in data/scripts/.
• Voice (Phase 2): ElevenLabs voice cloning from the same training audio that produced the knowledge.
• Video (Phase 3): HeyGen for scripted talking-head segments. A state machine connects scripted moments to live generated ones.
• Transcription: whisper.cpp with large-v3-turbo, run locally.