The Dynamic Inference Boundary: AI Coding Is Moving Into the Enterprise

AI developer tools are simple to deploy when the architecture looks like this:

developer → SaaS API → model

At enterprise scale, the story is different. The question is no longer which model produces the best results. Data flows and data residency become increasingly important, especially as models become more interchangeable. The deployment of those models needs to follow suit.

Security and infrastructure teams need to know where source code is processed, which network boundaries it crosses, who operates the infrastructure, and whether the system can function without public network access. Regulatory and compliance requirements become non-negotiable in certain environments. The challenge is that workloads evolve over time - while managed infrastructure is a good start, moving to a private network or disconnected environment might be required later and this might not be obvious from the get-go.

The agility of deployment therefore becomes a first-class property of an AI development platform.

"Self-hosted" hides several different boundaries

Self-hosted is a familiar, but misleading term. Similar to a decade ago, where on-premise vs cloud was too broad a comparison, we cannot just say "just self-host it!" and call it a day. There are at least three relevant boundaries to define:

• Execution boundary. Where do repository checkouts, shell commands, builds, tests, MCP servers, and other tools execute?
• Agent boundary. Where does the agent loop select context, orchestrate model calls, and process tool outputs?
• Inference boundary. Where does the model itself execute?

These are not equivalent.

Moving execution into a customer's Kubernetes cluster keeps builds and tools inside the customer's network while source code and context still leave the perimeter for agent processing or inference.

Cursor's self-hosted workers illustrate the distinction. Repository checkout and tool execution can run on customer-operated infrastructure, while the agent loop and inference remain in Cursor's cloud.

The execution is local and can operate within a safe area, inference and agent loops are outsourced to the vendor. A good compromise but very little is actually won in terms of data control.

Data residency moves another boundary

Keeping execution inside the customer environment is one level of control. Another is constraining where external processing can happen.

GitHub Copilot provides a useful example. GitHub Enterprise Cloud customers with data residency can enforce regional processing for Copilot. With the policy enabled, GitHub routes inference to model endpoints within the enterprise's designated region, and code, prompts, responses, logs, and telemetry associated with Copilot remain in-region. Requests processed under this policy consume 10% more AI credits.

This addresses an important class of regulatory and data-sovereignty requirements without requiring the customer to become an expert in inference infrastructure. It adds control over where processing occurs. It also keeps the operational simplicity of a SaaS while ensuring data does not randomly change country jurisdictions. But it still keeps the inference boundary intact - data still crosses into the vendor's perimeter.

So what if geography is not enough? What if inference itself must happen inside customer-controlled infrastructure?

That moves us from:

application → regional endpoint → model

to:

application → internal endpoint → customer-hosted model

And that changes the game entirely.

The disconnect-my-internet test

Let's reduce the question for customer-hosted models to another problem: the whole system must operate without connectivity to a vendor or the public internet.

There is a simple question to test this:

Remove any access to outside networks. Can the application, agent, and model still perform the required workflow?

If not, the system may provide strong privacy or residency guarantees, but it still depends on external infrastructure. Even a seemingly harmless external dependency, such as license verification or telemetry, breaks the isolation guarantee.

A fully isolated deployment requires the components needed to process requests, including inference, to exist inside the customer's network boundary. This is the real self-hosted solution and it removes a whole category of problems and regulatory checklist headaches.

Sounds too good to be true? That's because it is. Running inference effectively is hard. Fine-tuning model serving, deployments and caches requires specialized skillset and knowledge, which often requires additional hiring and training. The smart architect postpones this decision until it actually matters and provides some benefit, choosing the simplest possible solution until then.

Make the inference boundary movable

The deployment requirement an application has today may not be the one it has two years from now.

A team might start with managed inference because it requires no GPU infrastructure. Later, the same workload may need to run inside a regulated environment. Another deployment may need to operate without external connectivity because it is now trying to be compliant with health or federal laws.

The difficult part is not supporting each environment independently. It is moving between them without turning deployment policy into an application rewrite every 6 months.

This is the architectural constraint Kimchi is designed around.

Kimchi keeps the inference interface consistent while allowing the infrastructure behind it to change:

• Serverless runs inference on Kimchi-operated GPU infrastructure, with the added benefit of choosing regional endpoints interchangeably, while agents can use local harnesses for private execution or keeping the loop local.
• Self-hosted moves inference into the customer's Kubernetes environment, for a self-contained deployment.

The important property is what remains intact between those two. The application continues to use the same API and SDK contract. The Kimchi harness operates the same way. Moving from managed to self-hosted inference primarily changes where that API endpoint resolves, rather than how the application integrates with inference.

Choose the best infrastructure fit, but don't tailor the whole application architecture around it. That is the real enterprise AI solution.