Neon Water - Developer Documentation

This document provides a deep dive into the architecture, codebase, and technical details for developers who want to understand, modify, or extend Neon Water.

Overview
Architecture
Project Structure
Core Components
Audio Pipeline
Rendering Pipeline
Web Interface
Themes
Adding Features
Performance Considerations

Overview

Neon Water is an audio-reactive video generator that creates fluid simulation visuals synchronized to music. It analyzes audio files to extract musical features (beats, energy, frequency content) and uses those features to drive a WebGL-based fluid simulation rendered via headless browser.

Key Technologies

TypeScript - Core application logic
Three.js - 3D rendering and WebGL abstraction
Playwright - Headless browser for GPU-accelerated rendering
Meyda - Audio feature extraction library
FFmpeg - Audio decoding and video encoding

High-Level Flow

Audio File → Audio Analysis → Musical Features → Headless Renderer → Frame Sequence → Video Encoding → MP4

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                           CLI Entry Point                            │
│                         (src/index.ts)                               │
└─────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
            ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
            │   render    │ │   analyze   │ │   themes    │
            │   command   │ │   command   │ │   command   │
            └─────────────┘ └─────────────┘ └─────────────┘
                    │               │
                    ▼               ▼
            ┌─────────────────────────────────────────┐
            │           Audio Pipeline                 │
            │  ┌─────────┐  ┌──────────┐  ┌────────┐ │
            │  │ Loader  │→ │ Analyzer │→ │  Beat  │ │
            │  │ (FFmpeg)│  │ (Meyda)  │  │Detector│ │
            │  └─────────┘  └──────────┘  └────────┘ │
            │                     │                   │
            │                     ▼                   │
            │            ┌──────────────┐             │
            │            │   Musical    │             │
            │            │   Features   │             │
            │            └──────────────┘             │
            └─────────────────────────────────────────┘
                                    │
                                    ▼
            ┌─────────────────────────────────────────┐
            │         Rendering Pipeline               │
            │  ┌──────────────────────────────────┐   │
            │  │      Headless Browser            │   │
            │  │      (Playwright + Chromium)     │   │
            │  │  ┌────────────────────────────┐  │   │
            │  │  │   render-template.html     │  │   │
            │  │  │   ┌────────────────────┐   │  │   │
            │  │  │   │    Three.js Scene  │   │  │   │
            │  │  │   │  ┌──────────────┐  │   │  │   │
            │  │  │   │  │    Fluid     │  │   │  │   │
            │  │  │   │  │  Simulation  │  │   │  │   │
            │  │  │   │  └──────────────┘  │   │  │   │
            │  │  │   │  ┌──────────────┐  │   │  │   │
            │  │  │   │  │  Particles   │  │   │  │   │
            │  │  │   │  └──────────────┘  │   │  │   │
            │  │  │   └────────────────────┘   │  │   │
            │  │  └────────────────────────────┘  │   │
            │  └──────────────────────────────────┘   │
            │                     │                   │
            │                     ▼                   │
            │            ┌──────────────┐             │
            │            │  PNG Frames  │             │
            │            └──────────────┘             │
            └─────────────────────────────────────────┘
                                    │
                                    ▼
            ┌─────────────────────────────────────────┐
            │           Video Encoding                 │
            │  ┌──────────────────────────────────┐   │
            │  │  FFmpeg: frames + audio → MP4    │   │
            │  └──────────────────────────────────┘   │
            └─────────────────────────────────────────┘

Project Structure

neon-water-2/
├── src/
│   ├── index.ts              # CLI entry point, command definitions
│   ├── preview-render.ts     # Preview rendering script (duration-limited)
│   │
│   ├── audio/
│   │   ├── loader.ts         # FFmpeg-based audio file loading
│   │   ├── analyzer.ts       # Meyda feature extraction
│   │   ├── beat-detector.ts  # BPM and beat position detection
│   │   └── musical-features.ts # Maps raw features to visual params
│   │
│   ├── rendering/
│   │   ├── renderer.ts       # Headless browser renderer (Playwright)
│   │   ├── video-encoder.ts  # FFmpeg video encoding wrapper
│   │   └── frame-exporter.ts # Frame file management
│   │
│   └── config/
│       ├── schema.ts         # TypeScript interfaces
│       └── defaults.ts       # Theme definitions
│
├── web/
│   ├── index.html            # Web interface (song picker + preview)
│   ├── render-template.html  # Three.js scene for headless rendering
│   ├── server.js             # Development server with preview API
│   └── three.min.js          # Three.js library
│
├── audio/                    # Put audio files here
├── output/                   # Rendered videos appear here
│   └── preview/              # Preview videos
│
├── package.json
├── tsconfig.json
├── README.md                 # User documentation
└── ARCHITECTURE.md           # This file

Core Components

CLI Entry Point (`src/index.ts`)

The main entry point uses Commander.js to define three commands:

render - Full video rendering
analyze - Audio analysis without rendering
themes - List available themes

// Key flow for render command:
1. Parse CLI arguments
2. Load and analyze audio
3. Initialize HeadlessRenderer
4. Render frames (loop through features)
5. Encode video with FFmpeg
6. Clean up

Configuration Schema (`src/config/schema.ts`)

Defines TypeScript interfaces for all data structures:

interface MusicalFeatures {
  time: number;
  energy: number;      // 0-1: Overall loudness (RMS)
  pulse: number;       // 0-1: Beat strength
  brightness: number;  // 0-1: High frequency content
  density: number;     // 0-1: Spectral complexity
  impact: number;      // 0-1: Transient/onset strength
  flow: number;        // 0-1: Smoothed tempo feel
  // Raw features also available
}

interface Theme {
  name: string;
  palette: {
    primary: string;
    secondary: string;
    accent: string;
    background: string;
  };
  fluid: {
    dissipation: number;
    pressure: number;
    splatRadius: number;
  };
  particles: {
    count: number;
    size: number;
    speed: number;
  };
}

Audio Pipeline

1. Audio Loading (`src/audio/loader.ts`)

Uses FFmpeg to decode any audio format to raw PCM:

export async function loadAudio(filePath: string, targetSampleRate: number): Promise<AudioData>

Spawns FFmpeg as child process
Decodes to 32-bit float mono PCM
Returns { samples: Float32Array, sampleRate: number, duration: number }

2. Feature Extraction (`src/audio/analyzer.ts`)

Uses Meyda library for per-frame feature extraction:

export function analyzeAudio(audioData: AudioData, fps: number): RawAudioFeatures[]

Extracted features:

RMS - Root mean square (loudness)
Energy - Signal energy
ZCR - Zero crossing rate
Spectral Centroid - Brightness
Spectral Flatness - Noise vs tone
Spectral Rolloff - High frequency cutoff
Loudness - Perceptual loudness
MFCC - Mel-frequency cepstral coefficients
Chroma - Pitch class distribution

Spectral Flux is calculated manually via DFT for onset detection.

3. Beat Detection (`src/audio/beat-detector.ts`)

export function detectBeats(audioData: AudioData): BeatInfo

Computes onset envelope from energy
Finds peaks in onset envelope
Estimates BPM via autocorrelation (60-180 BPM range)
Refines beat positions to align with detected BPM

4. Musical Feature Mapping (`src/audio/musical-features.ts`)

Maps raw audio features to visual-friendly parameters:

export function analyzeFullAudio(audioData: AudioData, fps: number): AudioAnalysis

Mapping logic:

energy = RMS × 0.6 + loudness × 0.4 (smoothed)
pulse = beat strength + onset detection
brightness = spectral centroid (normalized)
density = spectral flatness + rolloff
impact = onset/transient strength
flow = smoothed RMS

Uses exponential moving average for smooth transitions.

Rendering Pipeline

HeadlessRenderer (`src/rendering/renderer.ts`)

Manages Playwright browser instance for GPU-accelerated rendering:

class HeadlessRenderer {
  async initialize(): Promise<void>
  async renderFrame(features: MusicalFeatures, time: number): Promise<Buffer>
  async renderAll(analysis: AudioAnalysis, outputDir: string): Promise<string[]>
  async close(): Promise<void>
}

Browser configuration:

Headless Chromium
WebGL enabled (--use-gl=egl)
Viewport matches output resolution

Rendering loop:

Load render-template.html
Call window.initScene(width, height, theme, seed)
For each frame:
- Call window.renderFrame(features, time)
- Take screenshot
- Save as PNG

Render Template (`web/render-template.html`)

Self-contained Three.js scene with:

Fluid Simulation (Navier-Stokes):

Velocity field advection
Pressure projection
Dye advection
Double-buffer ping-pong rendering

Resolution:

Simulation: 128×72 (low res for performance)
Dye: 512×288 (higher res for visuals)

Shaders:

advectionShader - Moves fluid/dye based on velocity
divergenceShader - Calculates velocity divergence
pressureShader - Iterative pressure solve
gradientShader - Subtracts pressure gradient from velocity
splatShader - Adds force/dye at a point
displayShader - Final compositing with color mapping

Particle System:

500-1000 particles
Velocity-based movement
Additive blending
Colors from theme palette

Audio Response:

impact / pulse triggers splats
energy controls splat intensity
Continuous gentle motion from energy
Colors chosen based on theme

Video Encoding (`src/rendering/video-encoder.ts`)

FFmpeg wrapper for final video creation:

export async function encodeVideo(options: EncodeOptions): Promise<void>

FFmpeg command:

ffmpeg -y -framerate 30 -i frame_%06d.png -i audio.mp3 \
  -c:v libx264 -preset medium -crf 18 \
  -pix_fmt yuv420p -c:a aac -b:a 192k \
  -shortest -movflags +faststart output.mp4

Web Interface

Server (`web/server.js`)

Node.js HTTP server providing:

Static file serving:

/ → index.html
/audio/* → files from audio/ folder
/preview/* → rendered preview videos

API endpoints:

GET /api/audio - List audio files
POST /api/preview - Start preview render
POST /api/preview/cancel - Cancel in-progress render
GET /api/preview/status - Check render status

Web UI (`web/index.html`)

Simple interface for:

Selecting songs from audio/ folder
Choosing themes
Selecting preview duration (10s, 15s, 30s)
Generating preview videos
Copying export command

No live preview - Uses actual render pipeline for accurate preview.

Preview Rendering (`src/preview-render.ts`)

Duration-limited render script:

Lower resolution (854×480)
Faster encoding (CRF 23)
Truncates audio/analysis to specified duration

Themes

Themes are defined in src/config/defaults.ts:

const themes = {
  'neon-water': {
    name: 'Neon Water',
    palette: {
      primary: '#00d4ff',
      secondary: '#0066ff',
      accent: '#ff00ff',
      background: '#000011'
    },
    fluid: {
      dissipation: 0.98,
      pressure: 0.8,
      splatRadius: 0.005
    },
    particles: {
      count: 500,
      size: 4,
      speed: 1
    }
  },
  // ... more themes
};

Adding a New Theme

Add theme definition to src/config/defaults.ts:

'my-theme': {
  name: 'My Theme',
  palette: {
    primary: '#ff0000',
    secondary: '#00ff00',
    accent: '#0000ff',
    background: '#000000'
  },
  fluid: {
    dissipation: 0.97,  // 0.9-1.0, higher = slower fade
    pressure: 0.8,      // Pressure solver strength
    splatRadius: 0.006  // Size of splats
  },
  particles: {
    count: 600,         // Number of particles
    size: 5,            // Particle size
    speed: 1.2          // Particle speed multiplier
  }
}

Add to web UI theme buttons in web/index.html

Adding Features

Adding a New Audio Feature

Extract in analyzer (src/audio/analyzer.ts):

// Add to feature extraction loop
const myFeature = Meyda.extract(['myFeature'], buffer);

Map in musical-features (src/audio/musical-features.ts):

// Add to MusicalFeatures interface and mapping function
myVisualParam: smoothing.myParam.update(raw.myFeature)

Use in renderer (web/render-template.html):

// In renderFrame function
if (features.myVisualParam > threshold) {
  // Add visual effect
}

Adding a New Visual Effect

Add shader (if needed) in render-template.html
Create material/geometry in initScene()
Update in renderFrame() based on features

Adding a New CLI Option

Add to commander definition in src/index.ts:

.option('--my-option <value>', 'Description', 'default')

Pass through to relevant functions

Performance Considerations

Rendering Speed

Bottlenecks:

Screenshot capture (Playwright)
PNG encoding/writing
Video encoding (FFmpeg)

Optimizations:

Lower resolution for preview (480p vs 1080p)
Lower quality for preview (CRF 23 vs 18)
Parallel frame writing could help but adds complexity

Memory Usage

Audio samples kept in memory during analysis
Frames written to disk immediately (not buffered)
Browser process uses significant memory for WebGL

For long tracks:

Consider chunked processing
Clean up frames during encoding

Fluid Simulation

Simulation runs at 128×72 regardless of output resolution
20 pressure iterations per frame
Could reduce for faster rendering (quality tradeoff)

Debugging

Browser Console

Add to renderer.ts to see browser logs:

this.page.on('console', msg => console.log('Browser:', msg.text()));
this.page.on('pageerror', err => console.error('Page error:', err));

Audio Analysis

Use npm run analyze to inspect features without rendering.

Frame Inspection

Use --keep-frames flag to preserve PNG frames for inspection.

WebGL Debugging

Open render-template.html directly in browser with mock data for shader debugging.

Future Improvements

Potential enhancements:

Live preview using WebGL (compromise on accuracy)
More themes / theme editor
Custom shader support
Multiple visualization styles (not just fluid)
Audio segment detection (verse/chorus)
Batch processing multiple tracks
GPU-accelerated encoding (NVENC)
Web-based full rendering (WebCodecs API)

Dependencies

Package	Purpose
`commander`	CLI argument parsing
`meyda`	Audio feature extraction
`playwright`	Headless browser rendering
`three`	3D graphics (in browser)
`fluent-ffmpeg`	FFmpeg wrapper (unused, direct spawn used)
`tsx`	TypeScript execution

External requirements:

FFmpeg (audio decode + video encode)
Chromium (installed by Playwright)

Contact / Contributing

This project was built as a tool for creating music visualizations. Feel free to fork, modify, and extend. Key areas for contribution:

New themes
Performance improvements
Additional visualization styles
Better beat detection algorithms
Web UI improvements

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History