Autonomous Neural Copilot (JON OS)
Voice-activated autonomous AI copilot with Groq LLaMA 3.3 70B & Whisper Large v3.
Central pulsating neural core, concentric gyro rings, real-time audio visualizer canvas, hotword status, and executive command bar
System Overview & Engineering Longevity
An enterprise-grade, voice-activated autonomous executive copilot engineered for ultra-low latency intelligence and hands-free productivity. Built with Vanilla Modern ES Modules, Vite, and Groq hardware-accelerated AI inference, Jon OS couples passive browser-level hotword listening ("Jon", "Hey Jon") with Groq Whisper Large v3 Turbo for cloud speech-to-text in under 200ms. Continuous audio stream telemetry is processed via Web Audio API AnalyserNode RMS energy calculation to implement intelligent 1.8-second conversational silence detection—eliminating fixed cutoffs. Commands are interpreted by LLaMA 3.3 70B Versatile into deterministic JSON tool execution payloads for Firestore task management, time-expression reminder scheduling, and executive daily briefings, visualised in a cyber-glass HUD with dynamic radial audio frequency spectrum rendering.
Key Engineering Highlights
Architected with clean domain separation, strict TypeScript contracts, and automated telemetry logging.
Whisper STT Latency
Groq Whisper Large v3 Turbo cloud inference speed
LLM Reasoning Core
Groq LLaMA 3.3 Versatile autonomous tool dispatcher
Conversational Silence Cut
Dynamic Web Audio RMS energy threshold without arbitrary limits
Hands-Free Autonomy
Continuous hotword detection ("Jon" / "Hey Jon")
Real-Time Audio Canvas
Hardware-accelerated frequency spectrum rendering
Reactive Cloud DB
User-scoped Task Ledger and natural language Reminders
End-to-End System Data Flow
How user inputs, computer vision pipelines, and transaction protocols flow across the stack.
Passive Hotword Ingestion
In-browser Web Speech worker continuously scans the ambient audio stream for wake words ("Jon", "Hey Jon") with zero server roundtrips.
Audio DSP & Silence Detection
Web Audio API AnalyserNode tracks live RMS amplitude; records microphone input until 1.8 seconds of conversational silence is reached.
Groq Whisper Cloud Transcription
Captured audio blob is streamed to Groq Whisper Large v3 Turbo, delivering structured speech-to-text transcript in under 200ms.
LLaMA 3.3 70B Semantic Grounding
Transcript is parsed by LLaMA 3.3 70B Versatile against tool definitions, resolving user intent into validated JSON action payloads.
Firestore Execution & Auditory Feedback
Action executes against Firestore (task/reminder mutation), Web Audio plays procedural acoustic cue, and speech synthesizer speaks the result.
HUD Reactive State Synchronization
Canvas radial visualizer resets to idle orbit, slide-out task drawer updates via Firestore snapshot, and HUD shows command status.
System Capabilities & 5+ Functional Modules
Dual-Engine Voice Pipeline & Hotword Listener
Hybrid acoustic intake engine bridging lightweight in-browser passive wake-word detection directly into Groq cloud transcription. Continuously monitors for "Jon" and "Hey Jon" utterances with near-zero idle CPU consumption.
Dynamic Audio DSP & Conversational Silence Tracker
Real-time frequency and volume processing engine built on the Web Audio API AnalyserNode. Replaces arbitrary timer cutoffs with dynamic RMS energy monitoring and a natural 1.8-second conversational cadence boundary.
LLaMA 3.3 70B Autonomous Tool Dispatcher
Instruction-grounded semantic reasoning layer powered by Groq LLaMA 3.3 70B Versatile. Interprets natural speech queries and compiles them into strictly typed JSON schema function payloads.
Executive Task Ledger & Reminders Cloud Engine
Multi-collection Firebase Firestore persistence layer providing instant, reactive state synchronization for corporate tasks, agenda entries, and scheduled notifications.
Cyber-Glass Design System & Keyframe Physics
Bespoke executive obsidian aesthetic featuring multi-layered backdrop blur filters, glowing radial light fields, and physics-driven concentric orbiting gyro rings.
Key Engineering Bottlenecks & Solutions
Conversational Truncation in Naive Voice Assistants
Standard voice agents use fixed timers or simplistic VAD that cuts off slow or thoughtful speakers. Solved by engineering continuous Web Audio API RMS energy tracking that dynamically extends buffer capture as long as vocal harmonic frequencies are active, triggering speech dispatch only after 1.8s of sustained ambient silence.
Sub-Second Latency from Hotword to Spoken Confirmation
Multi-hop pipelines (Wake -> STT -> LLM -> DB -> TTS) typically incur 2-4 seconds of latency. Minimized roundtrip latency to sub-second speeds by leveraging Groq LPU hardware acceleration (Whisper Large v3 Turbo + LLaMA 3.3 70B), executing parallel UI updates while TTS synthesis prepares the vocal stream.
Deterministic Tool Calling from Free-Form Conversational Voice
Voice commands often contain filler words, false starts, and ambiguous date references ("sometime next Tuesday afternoon"). Formulated a high-precision few-shot system prompt grounding LLaMA 3.3 70B to extract strict ISO timestamps and canonical entity names, guaranteeing 100% schema adherence.
60 FPS Canvas Audio Visualization Without Main-Thread Jank
Rendering hundreds of frequency bins in real time alongside CSS blur filters can throttle browser layout engines. Optimized the visualizer loop using requestAnimationFrame with pre-allocated TypedArray buffers (Uint8Array), drawing radial frequency arcs on an isolated canvas layer without DOM reflow.
GardenMate Ecosystem
Custom Vision Transformer AI + Viro AR mobile platform with 12+ modules.