Sovereign AI Perspectives | ClydAI

Sizing an on-premises GPU workstation for Gemma 4 26B

By Khyati Nayak, Founder, ClydAI Technologies

Key takeaway: In an offline deployment, the GPU is the decision that drives everything else. Size it for the model's total parameters, the precision you want to run at, and the realistic context and concurrency of your users, then verify the result against what the serving engine actually reports.

The sizing sequence for an offline Gemma 4 26B workstation.

When an AI system has to run fully offline, there is no scaling up to a bigger cloud instance later. Whatever hardware you specify is what the system lives with. So I went to the base metal: start with the GPU, because GPU memory determines which model you can run, at what precision, how much context it can hold, and how many people can use it at once. For this configuration, GPU memory is the primary inference constraint; system RAM still needed to support model loading and the surrounding application.

ClydAI is developing a natural-language interface for an aerospace client that will allow users to ask questions about complex operational data without navigating multiple systems or understanding the underlying data structures. The solution is designed to run entirely within the client's air-gapped environment, making context capacity, concurrent usage, and model performance important considerations in sizing the GPU.

For this deployment, we specified an NVIDIA RTX PRO 6000 Blackwell with 96 GB of VRAM, serving Gemma 4 26B with vLLM to support ten concurrent users.

Precision came first. With 96 GB, the model runs in BF16, its released precision. Google's published figure for loading Gemma 4 26B at BF16 is about 57.7 GB, which includes a 20% allowance for loading overhead. Running the original BF16 weights gives us a useful baseline: we can investigate incorrect outputs without introducing weight quantization as another variable. Lower-precision versions can then be evaluated against that baseline.

Google's BF16 loading figure is about 57.7 GB, leaving roughly 38 GB for the KV cache and vLLM's runtime overhead. The exact split depends on the serving configuration, which is why the KV cache capacity vLLM reports at startup is the number to check.

Gemma 4 26B adds a twist. It is a mixture-of-experts model: only about 4 billion parameters are active per token, but every expert must still be loaded in GPU memory. You get the computational advantage of sparse activation while still budgeting memory for the full model. Size for the total, not the active.

Context and concurrency come next. We configured a 70K-token maximum sequence length, with an expected working range of 10K to 20K tokens per request, including input and generated output. Reserving the maximum for every concurrent request can lead to overprovisioning. Ten concurrent requests in that working range represent 100K to 200K tokens to accommodate in the KV cache. We check that against the capacity vLLM reports at startup, allowing headroom for longer requests. If capacity falls short, we revisit precision, context limits, concurrency, or hardware before deployment. That establishes the memory budget. Load testing then tells us whether the system performs well enough, measuring time to first token, generation speed, and response times under concurrent requests.

The biggest lesson wasn't about hardware alone. In our testing, Gemma sometimes generated scripts that referenced libraries without importing them. Explicit instructions helped, but the experience illustrated the tradeoff: a model that fits comfortably on the workstation may still require more application support to deliver the quality the workflow needs. Hardware capacity and model capability have to be considered together

Offline AI isn't a smaller version of cloud AI. It is a different engineering problem, and it starts with the GPU.

#SovereignAI #AirGappedAI #AIArchitecture #EnterpriseAI

Contact us at info@clydai.ai