Fully local AI assistant
Baron — a JARVIS-style personal assistant
Like JARVIS in Iron Man: talk to it and it answers right away, give it a task and it combines tools to get it done, and it speaks up before you ask. Describe a song and it writes the lyrics, composes it and can even sing it in your own voice. Runs entirely on local models across two Mac minis, with no external AI API.
Research question
Without any cloud API, can two small servers deliver an everyday assistant with conversational response speed, safe tool execution and long-term memory all at once?
Approach
Not every request goes to the big model. A small model (gemma-4) classifies intent first and answers greetings and simple lookups itself within seconds; only multi-step work goes to the 27B reasoning loop. Just before that, a rule-based planner loads only the tool group the request needs, keeping the prompt cache warm. Tools are defined as capabilities, not use cases, so the loop composes them, and risky tools run only after an approval card. Code does the collecting, computing and checking; the LLM does the interpreting and writing.
Repositories
| baron / primary | Inference machine. 27B agent loop and planner, RAG memory, vision, REST·WebSocket API (runs detached from connections), morning briefing, self-review |
|---|---|
| baron / secondary | Worker machine. Intent router and fast paths, tool runtime·policy·audit, market data and strategy validation, position monitoring, business management, mail digests |
| baron / voice | Real-time voice conversation. Voice activity detection, turn-taking and barge-in, speaker verification, adaptive pronunciation correction, lookup-tool bridge |
| baron / music | Music production. Song design and lyrics, ACE-Step generation with GPU-swap scheduling, reference-track analysis, vocal separation·voice conversion·remix |
| baron / pwa | Smartphone client (React PWA). Chat, voice, photos and music, approval cards, pairing-code auth |
Key capabilities
- Three response paths — the small router answers directly or runs a single lookup tool only when it is confident (0.8+), in a few seconds to about 20 seconds; everything else goes to the 27B ReAct loop (up to 5 steps). An answer written without tools for a request that needed them is discarded and retried.
- Button-free real-time voice — it listens continuously, answers when you finish, and stops when you cut in while it is speaking. Backchannels like "uh-huh" are told apart from interruptions so it keeps talking, and an interrupted answer is remembered only up to the part actually heard. Measured: first audio about 2 seconds after you stop, and it stops within about 0.5 seconds when interrupted.
- Listens only to you — enroll about 45 seconds of voice and it accepts only registered speakers. Profiles are encrypted locally, and it learns your voice and pronunciation habits (e.g. terms it mishears) over time. Speaker verification is never used as authorization, so voice mode is lookup-only — no orders or sending.
- Says only what it verified — for live questions such as weather, stock prices or calendar it covers the wait with "let me check", then answers only from the lookup result; if the lookup fails it does not make up numbers. Stock questions all go to one briefing tool where the server collects, computes and drafts, reducing tool-selection mistakes.
- An answer within the time limit — when time runs short it stops calling tools and writes from what it has, and if that is empty it writes a partial answer from successful lookups. Runs are detached from the connection, so they finish even if the phone disconnects, and reconnecting resumes them.
- Long-term memory — durable facts (names, preferences, schedules) are extracted into a vector DB, and only those above an empirically tuned similarity threshold are recalled. Near-duplicate memories are not stored twice.
- Speaks first — a morning briefing of mail, calendar, tax deadlines, reminders and holdings (date and weekday phrases verified and corrected in code), an hourly digest of new mail, and D-day reminders synced with Google Tasks.
- Business assistant — record transactions by voice, receipt photo or card-statement file (VAT back-calculated), issue quotes and invoices with receivable alerts, store and semantically search documents, net worth and monthly reports.
- Market data and strategy validation — collects daily prices for every listed stock for screening and backtests. Backtests use only information up to the previous day and assume the unfavorable case when intraday order is unknown. A new setting must beat the current one in all four test windows to become a candidate, accumulates out-of-sample results through daily shadow runs with no real orders, and is adopted only by a human. Orders placed through conversation always require direct human approval, regardless of auto-approve settings.
- Lyrics, composition and audio — describe the song you want and the 27B model drafts a song blueprint (JSON) with sections, chord progressions, BPM and key, writes the lyrics, and the music model ACE-Step 1.5 renders a track with vocals. Blueprints are schema-validated, and truncated or malformed responses are re-requested with the reason attached. Measured: the audio-generation step takes about 45 seconds for a 30-second track.
- Sharing one GPU by swapping — 32 GB of memory cannot hold the 27B model and the music model together, so the 27B is unloaded only while music jobs run and restored afterwards; chat keeps working on the small-model path in the meantime. Generation runs in a night batch (22:00–05:30) by default; daytime requests get their design and lyrics done, then step out of the queue so they don't block other jobs. [Generate now] runs one immediately.
- Composing from reference tracks — give several YouTube links with weights and it analyzes each one's BPM, key, chord progression, structure, energy and timbre into a target profile, then designs a new chord progression and structure to match it, without reusing the references' melodies or progressions. The finished track is re-analyzed with the same analyzer and flagged if it is too close to a reference (a screening aid for possible copying, not a legal judgment). References are kept for personal analysis only.
- Sung in your own voice — after recording consent that the voice is yours or used with permission, enroll about 30 seconds of voice; it separates the vocals from a generated track or a song you own (Demucs), converts the timbre (Seed-VC) and remixes them with the instrumental, with key shifts up to ±12 semitones. Enrolled voices are encrypted and stay local, and results are for personal listening.
- Gets faster on its own — every night it diagnoses the previous day's latency, tool failure rate and negative feedback, and learns router examples from simple questions that were handled slowly so they take the fast path next time. Changes roll back automatically if eval-set accuracy drops; it never edits its own code.
Processing flow · chat
- Inputtext · photo · audio file
- Intent routingsmall router · fast answers
- Plannerrules · tool-group selection
- Memory searchbge-m3 · pgvector
- Agent loop27B ReAct · up to 5 steps
- Tool policySSRF · budget · approval · audit
- Answermemory extraction · speech
Processing flow · real-time voice
- Mic stream16 kHz · 20 ms frames
- Voice detectionsilero VAD
- Speaker checkenrolled voices only
- RecognitionWhisper · pronunciation fix
- Turn detectionendings · silence rules
- Responsegemma-4 · lookup tools
- Sentence-level TTSplays from first sentence
Processing flow · music production
- Requestdescription · references · voice
- Reference analysislibrosa · tempo·key·chords
- Song design27B · sections·chords·BPM
- Lyricsper-section lyrics
- GPU swapunload 27B · night batch
- Audio generationACE-Step 1.5
- Voice conversionDemucs · Seed-VC
- Output checksfile checks · originality
Code · statisticsLLM judgmentValidation gate
Python · asyncioFastAPI · WebSocket · SSEMLX local LLMs (Qwen3.8-27B · gemma-4)Whisper · silero VADsherpa-onnx speaker verificationACE-Step 1.5 · Demucs · Seed-VClibrosa · yt-dlpbge-m3 RAGPostgreSQL · pgvector · RedisReact · Vite PWA
※ The finance features are a technical reference, not investment advice or a solicitation to trade, and no returns are guaranteed.