Sovereign AI, Served Faster — How Elice Cloud Accelerates Open Models with NVIDIA NIM

How Elice Cloud Accelerates Open Models with NVIDIA NIM

Every month brings new open models, and every new model restarts the same work for an inference team: pick a checkpoint and precision, then tune KV cache, batching, CUDA graphs, kernels, and speculative decoding until latency and throughput targets hold on each GPU. A sovereign AI cloud cannot skip this work, because there is no external API to fall back on when its own serving stack is slow.

At Elice, we run that stack in Korea end to end: modular AI data centers (AI PMDC), the Elice Cloud Infrastructure (ECI) software that schedules GPUs and NPUs, and Elice ML API, which turns open models into managed endpoints. Education, public-sector, and enterprise AI services run on top of it, so customer data, models, and safety controls stay in the country.

Over the past three months, we tested NVIDIA NIM on our own hardware and traffic in three experiments: a replay of our production peak hour, a 100-case service-level benchmark, and a long-context agent evaluation with NVIDIA Dynamo. This post walks through the setup, the results, and what we plan next.

Key takeaways

• On a replay of our production peak hour (GLM-5.3-Flash, 4× B200), NIM cut median time to first token from 17.3 s to 0.58 s and delivered 17% higher output throughput than our production-tuned SGLang settings in the same run.
• On a 100-case benchmark built from real requests (Nemotron 3.5 Super VL, 1× B200), NIM delivered 1.44× peak throughput and 1.72× per-user speed over vLLM with multi-token prediction, about 30% less GPU time per output token.
• With Dynamo, the advantage grows with input length, reaching 4.3 to 4.4× vLLM throughput at 128K-token inputs on Nemotron 3.

Why serving efficiency matters for a sovereign AI cloud

A sovereign AI cloud cannot hand its workloads to external APIs. Elice Cloud runs on CSAP-certified infrastructure, and we serve open models on our own GPUs so that student, citizen, and enterprise data never leave the country. That makes the efficiency of our own serving stack a direct cost for every customer: each token our GPUs produce faster is capacity we can offer at a lower price.

The workloads on Elice ML API pull in different directions:

• Education AI. AI tutors and teacher assistants see thousands of users arrive in the same minute when a class starts. Answers must stay within the curriculum and the student's grade level.
• Document intelligence. Enterprises and government agencies send long, multimodal documents that must be turned into schema-constrained records with field-level validation.
• Private coding agents. Repository-scale context, frequent tool calls, and strict data boundaries.

Safety adds to the load. In education and public-sector services, one answer is a pipeline of several model calls: content safety, a scope check that decides whether the question should be answered at all, retrieval, generation, and a grounding and policy check against Korean curriculum and institutional rules. Running these guardrails on the same infrastructure keeps the whole pipeline in Korea, and each extra call takes a share of the latency budget.

Tuning each new model by hand took our engineers weeks per model, and with new models arriving every month, that pace was hard to sustain.

How we test

We hold three rules in every comparison.

• Change one thing. Model weights, precision, GPUs, and tensor parallelism stay fixed; only the serving runtime changes.
• Measure against service objectives, not peak numbers alone. Mean throughput rises with queue depth, so we also report latency percentiles and the throughput that still meets a service-level objective (SLO).
• Respect the noise floor. Repeated runs of an identical configuration varied in throughput by up to 29% at high concurrency, so we rely on differences that are large or consistent across concurrency levels, not on a single run.

Result 1: our production peak hour, replayed

We replayed one real peak hour from an Elice production service, with the same request stream sent to both configurations: GLM-5.3-Flash in FP8 on 4× B200 (TP4), once with our production-tuned SGLang 0.5.20 settings and once with the NIM 2.1.4 container and its shipped GLM-5.3-Flash profile. Both runs used the same FP8 checkpoint and the same SGLang version; only the serving configuration differed. Requests arrived at 3.5 per second with up to 64 in flight. Prompts ranged from 460 to 154,000 tokens, half of them short and a tail of very long documents, and 45 to 75% of each prompt was a shared prefix, as is typical for system prompts and tool schemas.

▲ Figure 1: Peak-hour replay, GLM-5.3-Flash FP8 on 4× B200. Same request stream, one run per configuration.

Median time to first token fell from 17.3 s to 0.58 s and the 99th percentile from 32.6 s to 4.5 s, a difference far beyond run-to-run variation. Output throughput was 17% higher in this run. Most of our production latency was queue wait: our own tuning had leaned toward decode, and requests waited behind each other at peak. The NIM profile schedules prefill more aggressively, which clears the queue and keeps the median first token under a second.

Result 2: a 100-case service-level benchmark

For model-by-model comparisons we use a test set of 100 cases built from real service requests. It is the same set we use to compare GPUs and NPUs in a national AI semiconductor benchmarking project, so results are comparable across hardware. We ran Nemotron 3.5 Super VL (NVFP4) on a single B200 per model copy and swept concurrency from 1 to 512 for every case, comparing NIM 2.0.13 with its default profile against vLLM 0.31.0 with the same multi-token prediction (MTP, 3 tokens). Prefix caching was off and GPU memory utilization was 0.92 in both.

▲ Figure 2: Nemotron 3.5 Super VL NVFP4 on 1× B200, 100 SLO cases. Latency metrics are shown as baseline divided by NIM, so higher is better everywhere.

In token economics, 1.44× peak throughput means that a workload needing ten GPUs on our previous configuration needs about seven with NIM, or roughly 30% less GPU time per output token at peak load. For a provider that owns its GPUs, this is a direct reduction in serving cost.

Result 3: long-context agents with NIM and Dynamo

In August, NVIDIA asked us to evaluate Nemotron 3 Ultra (550B-A55B) and Nemotron 3 Super (120B-A12B) with NIM and NVIDIA Dynamo on a single 4× B200 node. Over 385 measurements across seven serving configurations, seven workload profiles, and concurrency from 1 to 256, three findings stood out.

▲ Figure 3: NIM with Dynamo relative to vLLM on 4× B200, tool-calling and long-history profiles at 64 concurrent requests, and 128K-token inputs at 16 concurrent requests.

• A 550B model fits on one node. Nemotron 3 Ultra in NVFP4 occupies 352 GB, leaving 364 GB of headroom across four B200s for KV cache. The full 256K context was usable for batch document processing.
• Dynamo pays off as inputs grow. At 4K input tokens results were mixed, parity on Ultra within the noise floor and 1.64× on Super; at 16K it delivered 1.38× (Ultra) and 3.31× (Super) the throughput of vLLM; at 128K the gap reached 4.3 to 4.4×, with time to first token reduced by up to 37×. Non-Dynamo configurations reached 38 to 52 s to first token at 16 concurrent 128K requests, while Dynamo stayed within 1 to 5 s.
• KV-aware routing works as designed. With a 15K-token shared prefix, a typical agent preamble of system prompt and tool schemas, Dynamo gained 27 to 30% (Ultra) and 39 to 119% (Super) over the same load with distinct prefixes. No non-Dynamo configuration showed a gain beyond the noise floor.

We also found that enabling MTP raised throughput by 23 to 34% on reasoning workloads with long outputs, and lowered it on short tool-calling turns. The NIM profile we tested in October ships with MTP on, and its speedup over a vLLM baseline that also used MTP shows that the gains come from more than speculative decoding alone. We shared detailed findings from this evaluation with NVIDIA's engineering team.

Adapting models to Korean documents with NeMo AutoModel

Serving speed matters only if the model is accurate on our customers' data, and much of that data is Korean documents. We moved our post-training pipeline to NVIDIA NeMo AutoModel and post-trained three models on 8× B200 with the same data and method, on early 20th-century Korean documents that mix Hangul, archaic letters, Hanja, and Japanese in vertical and horizontal layouts. Our data is annotated together with literature scholars. On an earlier test set, frontier models scored only 73 to 75% on this task.

▲ Figure 4: OCR character accuracy on 593 test pages, post-trained with NeMo AutoModel on 8× B200.

Nemotron 3.5 Super VL started lowest and gained the most, 46.1 points, ending within 0.4 points of our production model. The same post-trained checkpoints can then be served on a tuned runtime inside our own infrastructure, which is the workflow we plan to use for other domain models.

What comes next

• Integrate NIM into Elice ML API endpoints, starting with Helpy Chat, our AI assistant for students and teachers, and Helpy Code, our private coding agent, measured against our tuned SGLang deployment on latency, throughput, GPU efficiency, Korean response quality, and tool-calling reliability.
• Localize safety guardrails for Korean education and public-sector use, including curriculum scope, grade-level boundaries, child safety, and institutional policy, and run them on the same sovereign stack.
• Bring NVIDIA Dynamo into our serving environment for long-context, agent-style workloads, and evaluate disaggregated serving on 8× B200.
• Evaluate Nemotron for Helpy Vision, our document and table understanding service, with accuracy and serving performance measured separately.

Conclusion

Inference efficiency decides how much capacity a sovereign AI cloud can offer from the GPUs it owns, and how much latency budget remains for the safety checks that education and public-sector services require. On our own traffic, NIM gave us a tuned starting point that beat our hand-tuned configuration on time to first token and throughput. We will keep publishing what we measure as we bring it into Elice ML API.

Acknowledgments

We thank the NVIDIA NIM, NVIDIA Dynamo, and Nemotron teams for early access and for their engineering support throughout these evaluations.