VoiceFlow: Local-First macOS Voice Agent
Turn spoken intent into verifiable Mac workflows with local Whisper, confirmation-gated execution, and encrypted audit trails.
01 · Problem Space & Threat Model
Autonomous desktop agents that execute system actions via voice face two critical hazards. First, streaming ambient microphone audio to third-party cloud LLM endpoints poses grave corporate and personal confidentiality risks. Second, natural language is inherently ambiguous: a misinterpreted speech recognition token or an errant agent plan can silently delete critical files, send unvetted communications, or execute unauthorized terminal scripts.
VoiceFlow solves this with a strict local-first architecture coupled with a deterministic confirmation state machine. All audio ingestion, transcription, and intent routing run 100% on-device on CPU hardware. Furthermore, any action with external side effects is intercepted at an immutable human confirmation barrier.
02 · System Constraints & Non-Negotiables
Capture, STT, intent classification, and workflow planning run locally on macOS with zero cloud API dependencies.
External side effects (terminal execution, browser submissions, file deletion) strictly require physical user confirmation.
Every voice command, intent resolution, and execution trace is encrypted at rest using CryptoKit AES-GCM in local SQLite.
03 · Architecture & Side-Effect Confirmation Barrier
Audio captured via Carbon hotkey passes through local faster-whisper STT and regex normalization into the intent classifier. Read-only queries execute directly, while side-effect commands halt at the confirmation gate.
Local Speech Intent to Verified macOS Execution
01. Push-to-Talk Capture intercepts audio via Carbon hotkey and feeds 16kHz PCM to on-device memory without cloud exposure.
02. Local Whisper STT (faster-whisper int8 CPU) generates transcript in p50 2,003ms with zero external network connectivity.
03. Rule-based zh-TW Text Normalizer sanitizes transcripts against domain dictionaries, eliminating STT hallucinatory drift.
04. Intent Classifier partitions commands into read-only queries versus state-mutating actions (B7: 1.0000 success rate).
05. Safety Gate intercepts all external side effects. Destructive actions require explicit user confirmation (0 auto-submits over 160 soak tests).
06. Sandboxed adapter executes confirmed command. Watchdog restarts crashed workers in 0.43s. Audits are AES-GCM encrypted in local SQLite.
04 · Three Key Architectural Decisions
1. Confirmation-Gated Side Effects Over Fully Autonomous Execution
SAFETY GATEContext: Voice commands can be easily misinterpreted in noisy rooms or when discussing technical code with colleagues.
Decision: Hardcoded the executor to intercept any irreversible operation (file modifications, POST requests, system script executions) and require explicit user approval. In social post automation, the demo stops completely before publish.
Outcome: In an 8-hour soak test across 160 tasks, recorded exactly 0 high-risk auto-submit actions.
2. Multi-Process Watchdog with 0.43s Recovery
RELIABILITY LAYERContext: Native CTranslate2 audio processes or headless browser adapters can hang, segfault, or leak memory under extended continuous operation.
Decision: Decoupled the SwiftUI menu-bar process from backend worker daemons via UNIX domain sockets and heartbeat pings. Implemented a self-healing watchdog with circuit breakers and worker pool recycling.
Outcome: Validated in stability runs: worker crashes recovered automatically in 0.43 seconds without terminating the menu-bar UI or dropping task state.
3. Encrypted Audit Logs with Redaction Pipeline
PRIVACY LAYERContext: Debugging voice agent failures requires inspectable logs, but persisting raw transcripts and screen grabs risks exposing credentials or sensitive user data.
Decision: Built an automated regex redaction filter that masks API keys, bearer tokens, passwords, and PII before writing. Encrypted all persistent database rows using Apple CryptoKit AES-GCM with keys anchored in the macOS Keychain.
Outcome: Zero unredacted secret leaks detected across all 160 soak task audit inspections.
05 · The Hardest Technical Challenge
Eliminating Hanging States in Route Selection During Monitor Failures
Root Cause: Under specific macOS permission states, the Carbon global hotkey monitor could fail during background initialization, leaving the UI state machine suspended indefinitely in a waiting_confirmation state where user keystrokes were never received.
Engineering Approach: Refactored the state machine to decouple hotkey listener registration from action evaluation. Added an automatic fallback timeout: if the key monitor fails to report a valid listener within 500ms, the system falls back to a windowed macOS dialog and executes the default safe route automatically.
Verification: Simulated 100 consecutive keyboard listener crash events in the test harness; 100/100 gracefully diverted to dialog fallbacks with zero orphaned or unrecoverable hanging states.
06 · Negative Results: STT Fine-Tuning Rejection
Engineering discipline means refusing to ship models that fail task-level validation, even when training and technical conversion complete without error.
07 · Soak Testing & Measured Evidence
8-hour continuous soak test executing automated voice workflows; zero main-window crashes or memory fragmentation.
Virtually flat memory slope across 8 hours of background operation, proving zero CoreAudio or Swift buffer leaks.
Sub-second process restart time when worker threads are intentionally killed via SIGKILL in fault injection tests.
Benchmark B3: 20/20 audio samples transcribed within acceptable desktop interactive latency on CPU hardware.
08 · Retrospective & Learnings
Engineering Takeaway: Engineering Takeaway: In voice-driven desktop agents, model benchmarks on static audio clips do not translate to reliable system execution. Fine-tuning a speech model on domain audio gave minimal real-world command execution improvements (+0.0727), whereas deterministic regex normalization and state-machine confirmation barriers provided 100% safety guarantees against destructive actions.
What I Would Do Differently: What I Would Do Differently: Decouple the browser automation adapters from direct DOM manipulation earlier. Transitioning to standard accessibility APIs (macOS AXUIElement) provides far greater resilience across third-party web application updates than scraping web selectors.