Building a Small Korean Foundation Model on B200 GPUs, Part 2 — Training and Evaluation

This post covers the architecture of the model we trained the Part 1 dataset on, the problems we worked through while training it in a B200-based multi-node environment, and the initial evaluation results.

→ Read Part 1

Model architecture and tokenizer

The goal of this experiment was not to build a huge model. It was to confirm that a pretraining pipeline runs stably even at small scale, and to actually put a Korean-centric dataset to work. So for architecture we prioritized structures with proven stability and reproducibility, and good training efficiency.

Choosing the architecture

We took Qwen3 0.6B as the base. It includes the components common to recent model designs — SwiGLU, GQA, RoPE, RMSNorm — and despite being small, it has reported good results across several public benchmarks.

We also initialized all model weights and trained our dataset from scratch. The aim was to validate three things at once: the stability of the B200 cluster under a load as heavy as pretraining, the quality of the dataset we had collected, and the efficiency of the training framework.

Choosing the tokenizer

Because the tokenizer directly affects training efficiency, we compared several public tokenizers first. The criterion was how stably compression efficiency holds up in an environment handling Korean and English together.

The Midm-2.0 tokenizer showed a high average compression ratio across both English and Korean, and the model it comes from has reported strong Korean and English performance, so we adopted it.

For cases needing broad multilingual support, Llama-family tokenizers produced better results in places. For this project, though, we judged it more appropriate to stay inside the scope of Korean-English bilingual training. Rather than over-expanding our choices at the starting line, the priority was a stable training environment inside a clearly defined target range.

Figures are compression ratios — higher means more characters packed per token.

Training infrastructure and the multi-node environment

If architecture and data are the core of model performance, the core of actually running the training is managing a large number of nodes successfully. That was one of this project's main objectives. Unlike single-node experiments, a multi-node environment can stop training over small differences in network configuration, file system, or container execution, so we spent considerable time on initial stabilization.

Cluster configuration

Training ran on 120 NVIDIA B200 GPUs (15 nodes) built out on Elice Cloud. All nodes are connected over InfiniBand, and training jobs were managed through Slurm on top of NVIDIA NeMo.

We deployed training with NeMo's Slurm executor, which meant installing the enroot and pyxis plugins on the Slurm cluster. On Ubuntu 24.04 the permission settings had to be adjusted separately, and we used an automation script (Pyinfra) to keep every node's configuration consistent.

Training configuration and stabilization

Since the purpose was to stand up a pretraining pipeline stably in a new B200 environment, we chose a conservative start based on already-validated hyperparameter combinations.

Warm-up steps were set to 2,000 with BF16-mixed precision, and at sequence length 4,096 and micro batch size 8, the 15-node / 120-GPU setup gave a global batch size of 960. The optimizer was Distributed Fused Adam.

Early in training we ran into bottlenecks we had not expected. First, NCCL timeouts while processing large volumes of data kept halting training; we resolved that by adjusting TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC. Then a slowdown appeared, at 20–30 seconds per step. We initially suspected a shared storage (NFS) bottleneck, so we replicated the 1T dataset directly onto every node's NVMe and modified and redeployed the NeMo image to support non-shared directories. Speed did not improve.

Isolating and verifying each stage of the training process, we found the cause was the NCCL heartbeat timeout setting we had adjusted earlier. After optimizing that value and applying NVIDIA's NeMo tuning guide, we were processing 4,096 tokens per GPU in about 0.45 seconds — performance on par with a single node. That was roughly a 30× speedup, and it confirmed that Elice's B200 cluster scales stably in a distributed training environment with no additional bottleneck.

Evaluation

The original targets were a 2T-token dataset and 2 epochs of training; to finish inside the available time we ran 1T tokens and 0.5 epochs. That is about one-eighth of the planned training, so this evaluation was aimed at checking whether the model had acquired a minimum level of comprehension in both Korean and English.

For consistent, reproducible evaluation we used lm-evaluation-harness, measuring HAERAE, KMMLU, and MMLU under both 0-shot and 5-shot conditions.

Given the purpose of the experiment, what matters here is that the initial model reached a state where it operates stably.

How to use it

The model we trained loads and runs easily through the transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "eliceai/eliceai-0.6B-Base"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto").to(device)

prompt = "대한민국의 수도는"
model_inputs = tokenizer(prompt, return_tensors="pt").to(device)

generated_ids = model.generate(**model_inputs, max_new_tokens=100)
output_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output_text)

Sample output

The prompt above reads "The capital of South Korea is". The model continues:

대한민국의 수도는 서울이다. 서울은 대한민국의 중심 도시로서, 대한민국의 정치, 경제, 문화, 교육 등 모든 분야에서 중요한 역할을 하고 있다. 서울은 대한민국의 역사와 문화를 대표하는 도시로서, 대한민국의 발전과 번영에

In English: "The capital of South Korea is Seoul. As the country's central city, Seoul plays an important role across politics, economy, culture, education, and every other field. As the city that represents South Korea's history and culture, Seoul — in the nation's development and prosperity —"

Wrapping up, and what comes next

In this project we used Elice's new B200 cluster to build the groundwork and the operational experience for training a Korean-centric LLM. From data preparation and quality filtering through distributed-training stabilization and bottleneck resolution, we went through and validated the core stages of assembling a pretraining pipeline in practice.

Next we plan to expand the data scale and apply multi-stage pretraining. After extending the training structure to handle longer context, we are also weighing a path through SFT and RL stages toward a conversational model or an embedding model. Beyond that, applying the structure we built here to a larger model is a goal as well.

It was a short experiment, but the trial and error and the experience of pretraining a Korean foundation model directly will be the base that makes the next experiments faster.

Elice Cloud offers the same B200-based training environment used in this project. If you want to try research or model development on a B200 cluster in Asia's first liquid-cooled data center, get in touch with Elice Cloud. Research collaboration proposals are welcome any time.