AI Live Music Visualizations

Electron desktop app for AI-assisted live music visualizations. The app captures live audio, extracts normalized audio features, renders reactive visuals with Butterchurn or a bundled custom shader, and includes the first pieces of an Ollama-based scene planning flow.

Current State

This repository is a working TypeScript/Electron prototype with tests and production build support.

Implemented:

• Electron main, preload, and React renderer processes.
• Secure preload API for app shell data and .milk preset import.
• Live microphone capture through the Web Audio API.
• Audio feature extraction for volume, bass, mid, treble, spectral centroid, and beat detection.
• Butterchurn visual engine with bundled default preset support.
• .milk preset import with file validation.
• Custom shader visual engine.
• Deterministic scene orchestration from audio features.
• Ollama planning service implementation and tests.

Prototype gaps:

• The renderer currently uses the deterministic/stub AI planning path; the Ollama controls are present but not wired into live planning requests.
• E2E tests use mocked audio, fullscreen, and preset import flows.
• Platform package scripts exist, but platform packaging should be verified on each target OS.

Requirements

• Node.js 24 or compatible modern Node.js runtime.
• Corepack enabled for the pinned package manager (pnpm@9.15.4).
• Desktop runtime dependencies required by Electron for your OS.

If pnpm is not available directly, use corepack pnpm ... for the commands below.

Setup

corepack enable
corepack pnpm install

Development

Start the Electron/Vite development app:

corepack pnpm dev

The app opens an Electron window with the visualization panel and audio controls. Browser microphone permissions and available input devices depend on the host OS and Electron runtime.

Validation

Run the container-safe checks:

corepack pnpm lint
corepack pnpm typecheck
corepack pnpm test
corepack pnpm build

Run Electron E2E tests after building:

corepack pnpm build
corepack pnpm e2e

Linux containers may need additional Electron desktop libraries for E2E execution. If Electron fails before the app loads because a system library such as libgtk-3.so.0 is missing, treat that as host setup rather than an app assertion failure.

Project Structure

src/main/       Electron main process and preset import IPC
src/preload/    Secure contextBridge API exposed to the renderer
src/renderer/   React UI, audio capture, AI planning, and visual engines
src/shared/     Shared contracts, defaults, parsers, and audio feature utilities

Useful Scripts

corepack pnpm dev             # Start Electron/Vite development mode
corepack pnpm lint            # Run ESLint
corepack pnpm typecheck       # Run TypeScript checks for node and web configs
corepack pnpm test            # Run Vitest unit tests
corepack pnpm build           # Build main, preload, and renderer output
corepack pnpm e2e             # Run Playwright Electron E2E tests
corepack pnpm package:linux   # Build and package for Linux
corepack pnpm package:mac     # Build and package for macOS
corepack pnpm package:win     # Build and package for Windows

Preset Import

The app supports importing Milkdrop .milk presets through Electron IPC. Imported files are validated for extension, size, and empty content before being loaded into the Butterchurn engine.

AI Planning

The codebase includes two planning paths:

• createStubAiPlanningService(): deterministic local behavior used by the current renderer flow.
• createOllamaAiPlanningService(): requests JSON scene plans from an Ollama /api/generate endpoint and degrades back to the current plan when unavailable.

The next implementation step is wiring the renderer controls to the Ollama service and deciding when live audio features should trigger planning requests.