What a Private AI Workload Taught Us About Compute, Cost, and Control

Private AI discussions often start with the model. The first business decision is usually elsewhere: where the workload runs, who controls the data, what the monthly cost looks like, and how much operational work the team wants to own.

We measured those questions on ZCP. The workload used Ollama and Open WebUI on a 16 vCPU, 64 GB Intel VM in YUL-1. The test included model serving, a browser experience, secure access controls, performance monitoring, and complete cleanup.

The result gave us a clear decision framework for teams evaluating private AI.

The executive takeaway

A 64 GB CPU-only VM is a useful private AI evaluation platform. It supports smaller local models for internal chat, prototypes, controlled API workloads, and offline processing.

A 70B model fits in the memory envelope, but fit does not mean fast interaction. Our 70B CPU test used about 43 GB of resident memory, drove CPU usage up to 99.5 percent, and produced response times too slow for a good live chat experience.

The smaller model produced the better chat experience for this VM. The 8B model responded in 10.96 seconds during the reference run. The first short 70B response took 117 seconds. The 8B model used less memory, left more room for the operating system and other services, and gave the user a faster path to an answer.

This was the most useful product lesson from the test: the best model is the one matched to the use case. A smaller model is the better choice for interactive chat when it meets the quality target. A larger model belongs in a workload where its quality advantage justifies the added latency and infrastructure cost.

For decision-makers, the measurements support these starting decisions:

Public-cloud GPU VM inference is not currently available in YOW-1 or YUL-1. Larger responsive models therefore require private or dedicated GPU capacity, a specialist partner, or a managed inference service. The correct answer depends on the workload, the sensitivity of the data, the response-time target, and the team operating the service.

What we built

The test used a two-service pattern:

Ollama and Open WebUI ran on the same VM. Open WebUI provided the user experience. Ollama loaded the model and performed inference.

That separation matters when you evaluate a vendor proposal or internal architecture. A browser interface improves access for users. It does not add compute and it does not accelerate inference.

We deleted the test VM and its public resources after validation. No customer data or production service touched the environment.

Why run this workload in a Canadian cloud region?

Business requirements often drive a private AI deployment before technical preference:

• Keep sensitive prompts and responses in a chosen jurisdiction.
• Keep control of the machine, storage, network, and access policy.
• Use a predictable Canadian-dollar infrastructure bill.
• Start with a contained pilot before buying dedicated hardware.
• Keep a path toward private GPU infrastructure as usage grows.
• Avoid sending every internal request to a public model endpoint.

ZCP YUL-1 gave us a regional environment for this test. The value was control and a clear cost boundary. It was not a promise that every model would run quickly on every VM size.

What the test cost

On 2026-08-16, the YUL catalog priced the reference resources as follows. Check the current ZCP pricing before a paid deployment:

A 24-hour validation run at those rates is about CA$19.40 before tax.

The estimate covers one VM, one isolated network, and one public IPv4 address in YUL-1. It excludes tax, optional block volumes, snapshots, backups, and discounts. Open WebUI did not create a second VM charge because it ran on the same machine.

The VM bill is only one part of the operating cost. Budget for model downloads and storage, engineering time, monitoring, backups, access controls, patching, incident response, and either private GPU capacity or managed inference fees when the user experience requires it.

The hourly model is useful for pilots. A team gets a real workload measurement for a day or a week before committing to a larger architecture. Still delete the deployment when the test ends.

Prices and availability change. The ZCP catalog remains the source of truth before a paid deployment.

What the performance results mean

The test compared a smaller 8B model with a larger 70B model. Both ran without a GPU.

The measurements used local Q4-quantized model artifacts, a short prompt, num_ctx set to 2048, and num_predict set to 128 for chat. The first request included model loading, so these are cold-load measurements from one validation run, not a production SLA. The technical tutorial contains the reproduction settings.

These results show why model selection belongs in the business case.

The smaller model offered a workable starting point for private interactive use. The larger model offered a quality test, but the user experience was too slow for a responsive assistant on this CPU-only VM.

A 70B model loading successfully proves capacity. It does not prove readiness for production chat.

What worked well

The Marketplace image shortened the path from account to working model service. The team did not need to build the base operating system and Ollama installation from scratch.

The regional VM made the deployment boundary clear. Compute, storage, public access, and cleanup were visible as separate resources.

Open WebUI made the service easier to demonstrate to non-technical users. It gave the test a browser workflow without changing the inference path.

The access design also exposed a useful operational lesson. Use an encrypted SSH tunnel during validation. If users need shared public access, terminate TLS at an authenticated reverse proxy and publish only HTTPS. Keep the Ollama API private. A service running inside a VM is not automatically a public service.

What did not work as a production default

The largest model was the wrong choice for responsive CPU chat. Memory pressure left little room for the operating system and other services. Additional concurrent requests increased waiting time instead of improving throughput.

Exposing the Ollama API directly would have created an avoidable security risk. Ollama does not provide built-in authentication. The safer test pattern kept Ollama private and exposed only the browser port.

Running the browser interface and inference service on one VM worked for a demonstration. A production design should assess isolation, authentication, TLS, backups, observability, and failure recovery. A separate UI layer and inference layer often make those controls easier to manage.

The decision framework

Use a CPU VM when the goal is a private pilot, a small model, low request volume, offline work, evaluation, or a controlled internal tool.

Use private or dedicated GPU compute, or a specialist GPU partner, when users expect fast responses from larger models. The same applies when concurrency matters or the assistant sits in a customer-facing workflow. Public-cloud GPU VM inference is not currently available in YOW-1 or YUL-1.

Use managed inference when the priority is a fast launch with limited infrastructure ownership. The tradeoff is less control over the runtime and a different data and cost model.

Use dedicated private infrastructure when the workload has sustained load, strict residency requirements, specialized hardware needs, or a long operating horizon. The initial design should leave a path from VM validation to dedicated capacity.

What leaders should ask before approving a private AI pilot

1. What data enters the model service?
2. Which jurisdiction must hold the data?
3. What response time do users expect?
4. Which model size supports the quality target?
5. Is CPU capacity enough, or is GPU capacity required?
6. Who owns patching, monitoring, backups, access, and incident response?
7. What is the hourly pilot budget and the expected monthly production budget?
8. How will the team delete or retain the model and conversation data?
9. Which network ports are public, and why?
10. What evidence will move the workload from pilot to production?

These questions prevent a large model from becoming the architecture by default.

From evaluation to delivery

The deployment path is repeatable:

1. Choose the region and data boundary.
2. Select a VM based on model size and response-time needs.
3. Start with a smaller model and measure it.
4. Add a browser interface only after direct inference works.
5. Use an encrypted SSH tunnel for validation. For shared access, publish only HTTPS behind TLS and authentication.
6. Record cost, latency, memory, concurrency, and errors.
7. Decide whether the next step is a larger CPU VM, GPU compute, managed inference, or dedicated infrastructure.
8. Delete the test resources or harden them for an approved production service.

For the developer implementation, see Run Ollama Chat and Inference on ZCP and Run Open WebUI With Ollama on ZCP.

The decision follows from the measurements. Private AI is a workload decision, not a model-size contest. Start with a measured pilot, price the full resource boundary, and choose hardware based on the user experience the business needs.