Auto-Generating VRM Character Animation Videos with Three.js + Puppeteer
Introduction What if we could automatically convert tech blog posts into VTuber-style explainer videos? Starting from that idea, I built a pipeline that renders VRM characters frame-by-frame using Three.js + Puppeteer, syncs them with VOICEVOX speech, and produces finished videos. In this post, I’ll share the lessons learned and pitfalls encountered during implementation. Overall Pipeline The processing flow is as follows: Load a Markdown article → Generate a section-divided script using an LLM (OpenRouter API) VOICEVOX generates speech audio (WAV) and phoneme timing for each section Three.js + @pixiv/three-vrm renders a VRM model on headless Chrome, outputting lip-synced animation as sequential PNG frames based on phoneme data Auto-generate slide images (HTML → headless Chrome → PNG) FFmpeg composites the slide background + VRM animation + audio into an MP4 video A Python script serves as the orchestrator, invoking the Node.js VRM rendering script as a child process. ...