Building a Small Korean Foundation Model on B200 GPUs, Part 1 — The Dataset

Elice recently became the first in Asia to deploy a liquid-cooled NVIDIA B200 cluster, and we are now testing directly what research this new class of AI infrastructure makes possible.

This project was the first step: training a small foundation model strong in both Korean and English from scratch. The point was not to compete on the finished model's benchmark scores — it was the process itself, of pretraining an LLM from zero on large-scale GPU infrastructure in Korea. Along the way we built a bilingual dataset of one trillion tokens and validated a multi-node training pipeline across 15 B200 nodes / 120 GPUs.

This post shares that experience, in the hope that it is useful to researchers and engineers who want to develop Korean-language LLMs.

Building the data: a Korean-centric 1T-token bilingual dataset

Data preparation is where most of the time and cost in foundation model training goes. Korean in particular still has far less publicly available high-quality data than English. Given that constraint, we designed the data with one aim: get the best quality we could actually reach.

We started by assembling a corpus from proven public English data and Korean datasets released both domestically and abroad. To address the shortage of Korean data, we then built a large-scale translation pipeline and added Korean translations of a portion of the English web data. Translation ran on Qwen3-Next-80B, and produced roughly 190B tokens of translated data.

We also applied quality scoring with the same LLM to both the translated data and the public Korean data, filtering out samples judged low in educational value and language quality. That cut the noise and gave us more stable training data.

Because current language models are expected to handle code generation and mathematical reasoning as well as language, we included an appropriate share of code and math data. The result is a bilingual dataset of about one trillion (1T) tokens spanning everyday language, technical text, math, and code.

The detailed composition:

Dataset composition

Data composition

Data is the single most important resource in training a large model. How much high-quality data you can secure decides early performance and stability more than anything else. Korean has relatively little large-scale public training data available, so this project combined existing public data with our own translation and cleaning pipeline. Following recent research, we aimed at collecting high-quality data rather than simply filling a volume target.

Because of that, we did not reach the 2T tokens we originally planned — time and cost were the limits — and finished with a corpus of roughly 1T tokens, centered on English and Korean web text and reinforced with math and code data.

Public English data

To establish the model's baseline language ability, we used a proven English dataset: fineweb-edu, which filters Common Crawl by learning value.

Public Korean data

High-quality public Korean corpora are relatively scarce, so our strategy was to combine data from as many sources as possible. We gathered text of varied kinds from the National Institute of Korean Language's Modu Corpus (newspaper and written-language sets), AI-Hub (news, web text, administrative and legal documents), and community-built datasets such as HAERAE and korean-web-collection. Formats varied widely and quality was uneven, so we collected first and then cleaned via model-based quality scoring.

That gave us a corpus covering a range of registers and subjects rather than one skewed toward a single domain.

Math and code data

Handling natural language well is no longer enough for a language model. Basic reasoning ability over code and math problems is expected too, so instead of deferring that we built it into pretraining from the start. Math data came from OpenWebMath and AlgebraicStack; code data was StackExchange-based text.

Rather than iterating through many mixture experiments, we set the ratios by referring to recipes from already-validated small-model builds such as SmolLM3 and Kanana. Following several research findings, we prioritized stable language learning at this early experimental stage over loading in large amounts of code and math data.

Data translation

The absolute shortage of Korean data was clear from the outset. Our approach was to translate the higher-quality portions of public English data into Korean and include them in training. Because the translations had to be good enough for a high-quality training set, we used Qwen3-Next-80B.

For translation we deployed the model with sglang across 15 nodes of the B200 cluster, at data parallel size = 4 and tensor parallel size = 2. Parallel configuration on B200 was not yet fully settled at the time, so we ran several LLM servers inside each node directly and put an external load balancer in front of them. That produced 17K tokens/sec per node and 200–300K tokens/sec overall, yielding roughly 190B tokens of translated data.

The experience made clear how many variables initial setup introduces when new hardware arrives. On the other hand, it let us build up experience with model inference environments and cluster operations quickly, which helped secure stable inference performance in later experiments.

Data filtering

Web-based data has volume on its side but very uneven quality, so rather than using what we collected as-is, cleaning through model-based quality scoring was necessary. We wrote a Korean-specific prompt modeled on fineweb-edu's evaluation prompt, scored text for learning value with it, and removed samples below a threshold.

Not all data evaluated the same way, though. News data in particular has clean sentences yet mostly scored low on fineweb's educational-value scale. We judged that a mismatch between the source data's characteristics and the model's evaluation criteria, so for news documents we switched to a binary classification (high quality / low quality) and re-applied it.

Preprocessing

Once collection and filtering were done, we preprocessed the data into a format usable directly in the NVIDIA NeMo framework. We organized the text as JSONL, tokenized it with the Midm-2.0 tokenizer used in this project, then converted it into .bin (text body) and .idx (index) files to match NeMo's training format, completing the dataset.

Part 1 has covered collecting, translating, cleaning, and preprocessing a dataset for Korean-centric pretraining. In the next part we will share the model architecture we trained this dataset on, the process of actually training it in a B200 multi-node environment, and the evaluation results.

→ Read Part 2