Running LLMs Where the Data Can't Leave the Building

The most common objection I hear to generative AI has nothing to do with capability: the data cannot leave the building. For defense programs that is law. For healthcare, legal, and finance it is regulation or client obligation. The engineering question underneath the objection is what fraction of useful LLM work can run entirely inside a boundary you control, and for the bounded tasks that dominate real enterprise workloads, that fraction has been growing for years.

Most enterprise workflows are bounded

Decompose the actual work: retrieval over internal documents, structured extraction, drafting inside a known domain, workflow control. These tasks have narrow inputs, checkable outputs, and a known vocabulary. I have designed, trained, and deployed LLMs for locally secure workflow control in environments where no commercial API was ever an option, and the ceiling on those deployments was set less by model quality than by how well the surrounding system fed the model what it needed.

Retrieval sets the ceiling

The pattern that holds up in practice: run a model sized for hardware you control, and spend the savings on retrieval quality, prompt orchestration, and evaluation. On grounded tasks, output quality tracks the relevance of the retrieved context more tightly than it tracks model scale, and retrieval is the component you fully control. The allocation is also cheaper to iterate on, since reindexing a corpus takes hours while new inference hardware takes a budget cycle.

Design from the boundary inward

Teams get into trouble by building the workflow first and presenting it to security review afterward. In a secure environment the boundary determines the architecture: where embeddings live, what gets logged, who can query what, and which data may share an index. Starting from those constraints and designing inward is consistently faster than retrofitting, because a retrofit reopens every component the review rejects.

The figure walks one request through an internal proposal assistant of the kind defense and consulting firms now build: a locally hosted model drafting from the company's own documents. The design choice that matters most is when the access check runs.

If entitlements are checked before the model reads anything, restricted documents never enter the prompt, and the audit record shows exactly what shaped the draft. If the check runs on the finished draft instead, the model has already read restricted text. A filter can strip sentences that cite a restricted document, but a paraphrase with no citation passes through. Run the proposal manager through both settings to see the difference. The program engineer is entitled to everything the search returns, which is why a system that only fails for some users can pass a demo and still fail a security review.

Training inside the boundary

The same constraint applies upstream of deployment. On the defense programs I led, model training and experimentation ran on premises inside controlled boundaries because the data could not move to cloud compute. Deployment then followed the consuming workflow, on premises or into an approved government cloud environment, depending on where the users and their data already lived.

Training inside the boundary changes the economics of experimentation. Every GPU, dataset version, and experiment log has to be justified and maintained locally, so the evaluation harness and the data pipeline deserve the same investment as the model. In my experience several of the better architectural decisions on those programs came directly from that constraint.

The tradeoffs

• The operations burden is yours: updates, monitoring, and the hardware lifecycle.
• Evaluation is entirely yours as well. No vendor is measuring output quality for you.
• Some tasks genuinely require frontier-scale capability, and identifying which ones before committing to an architecture prevents an expensive discovery afterward.

An organization that has been told it cannot use AI for security reasons usually has a narrower constraint underneath: no data egress. That requirement has engineering answers, and the sections above cover most of them.