A Local AI That Keeps Its Own Memory, and a Voice to Talk to It

Every assistant I use forgets me when the window closes. CogniCore is my experiment in the opposite arrangement: a companion that runs entirely on one 24 GB graphics card in my house, keeps a long-term memory, and retrains part of itself every night on what it learned that day. Two related projects give it a voice. None of this is a product, and most of it is unfinished.

The architecture

The current version uses one large local model as the reasoning core, which the code calls the cortex: a 27-billion-parameter Qwen model that occupies about 17 GB of GPU memory. Around it sit three supporting pieces.

• A memory store. Each memory is embedded and retrieved by a blend of relevance to the current conversation, recency, and importance.
• A self-description document the system writes and revises about itself.
• A small model built on a 0.6-billion-parameter embedding backbone with five prediction heads: importance, appraisal, reaction, relevance, and interest. It is fully retrained each night around 3 a.m. on the day's exchanges.

The trust rule

The small model makes predictions the cortex can use, such as which of two draft replies the person is more likely to welcome. A freshly trained head has no business steering anything, so each head is used only after it beats a trivial baseline on held-out lessons, and its measured advantage over that baseline is its trust. A nightly retrain that scores worse on the held-out set is discarded, and the previous version stays in place.

That rule is the same discipline I apply to production models at work. A model earns influence by beating a baseline on data it has not seen, and a regression rolls back automatically. It also sets expectations correctly: the project notes say the first days feel like the cortex with a memory, because the heads have not earned trust yet. The reaction head needs a few hundred exchanges before it is used.

An earlier version is still in the repository for comparison: a from-scratch recurrent core of about 200 million parameters, with small frozen language models as translators and an action selector trained by reinforcement learning.

Giving it a voice

AIConvos is an always-on, interruptible voice pipeline built on the open-source Pipecat framework. Voice activity detection and an end-of-turn model decide when you have finished speaking, Whisper transcribes, a local model answers, and Orpheus or Kokoro speaks the reply. Expected voice-to-voice latency is around 0.8 seconds on an M3 Pro laptop and around one second on a desktop with an RTX 3090. Those are design targets built from component benchmarks, and I have not published measured end-to-end numbers.

Interruption was the hardest part to get right. When you start talking over it, the speaker queue is flushed, the text-to-speech stream is closed so generation stops, the language model request is cancelled, and the conversation history keeps only the words that were actually spoken aloud. Without that last step the model believes it said things you never heard.

Roobit ports that design onto a Kinect v2, so the assistant can also see. Depth, 25-joint skeletons, and the color camera are tools the model can call. Two rules there came from watching it fail. A benchmark replays tasks that went wrong in real use against a simulated PC and scores them, so a fix has to prove itself. And any reply that claims an action is done without a tool call having done it is held back from speech.

Status

The Kinect integration and the Orpheus voice path in Roobit have not been run against real hardware in the build environment. They are verified by automated tests and interface checks only. CogniCore's nightly retraining runs, and whether a long-running companion develops anything like initiative is the open question the project exists to explore.