Scaling data mixing for language and DNA pretraining

We scale pretraining data mixing from a handful of predefined sources to thousands of sources derived automatically from the behavior of individual documents, and from a single mixture to tens of thousands that trace the Pareto frontier of multi-objective performance. We apply it to both language and genome models.

Language model performance is dictated by the quality and composition of its training data. A pretraining corpus is assembled from many sources. Language models train on web crawls, code repositories, books, and scientific papers. Genome models, such as our Omnii models¹, train on DNA from thousands of species. In practice, these sources are not sampled in equal proportion. Each source gets a weight, and training batches are drawn in proportion to those weights. That per-source weighting, called the data mixture, shapes which capabilities a model develops. A well-chosen data mixture yields a better model for the same compute, while a poor one leaves important capabilities undertrained.

Choosing the weights automatically is a well-studied problem. Most methods use small proxy models or small-scale training runs to predict how a mixture will perform^2–⁶. But nearly all of this work leaves two things fixed.

The first is the input: the sources themselves. They are usually fixed in advance, either as the original datasets the corpus was built from (a web crawl, books, code) or as a small set of topic categories (science, health, finance). A weight can only move a whole source, so if a topic holds both text that helps and text that doesn't, the mixture has to take or leave them together. The second is the output. A mixing method returns one mixture, which encodes one fixed trade-off between capabilities. Optimizing for a different trade-off means starting over.

So we asked: what if the sources came from the data itself, and the mixture were a map rather than a single point?

Languageweb documents

mixture fit to
weightlowhigh

DNAgenomic sequences

mixture fit to
weightlowhigh

Every point is a document, embedded so that documents with similar effects on the target evaluations are positioned close together. Points are grouped into derived sources, and color shows each source's weight in the data mixture fit to each evaluation in turn; hover over a point to light up its source. We explain these maps in detail later in the post (Figures 7 and 10).

In this post, we scale automated data mixing at both ends: the input sources and the output mixtures. Our method, Tessera, works as follows. For each of a few thousand documents, we train a nano proxy model on that document alone and measure how it changes performance on a set of target evaluations. We then learn to predict this eval performance for every document in the training corpus. Clustering all documents in this eval "behavior space" allows us to construct thousands of sources, each holding documents that behave alike when trained on. Mixing over these derived sources delivers four times the gain of mixing over a topic taxonomy, and the advantage holds as we scale up model size and training tokens.

Because fitting a mixture over the derived sources can be made to run efficiently, we solve it tens of thousands of times to map the Pareto frontier between evaluations, and then select the mixture that matches a desired trade-off. And because everything rests on a per-document measurement, the same pipeline doubles as a lens on which data is worth training on. We show this on language and on DNA (Figure 1).

Scale the input: thousands (or more) of derived sources

Scale the output

1Documents

Text, DNA, or any other modality

2Measure

Train nano proxy models for each document and score on a set of evaluations

scale the input

3Behavior space

Represent each document by a set of behavior directions

scale the input

4Thousands (or more) of sources

Cluster documents into thousands of sources by behavior; most end up with no weight

scale the input

5Explore the Pareto frontier of mixtures

Fit tens of thousands of mixtures, one per weighting of the evaluations

scale the output

Figure 1. Tessera scales data mixing at both ends: on the input side, from a handful of predefined sources to thousands derived from document behavior; on the output side, from a single optimized mixture to tens of thousands of mixtures on a Pareto frontier.

Highlights

Sources derived from document behavior. Instead of drawing sources by topic or origin, Tessera groups documents by their measured effect on training. It groups by what a document does to a model.

Four times the gain compared to an optimized mixture on hand-defined sources. Mixing over Tessera-derived sources improves held-out loss across six evaluations about four times as much as mixing over a hand-built topic taxonomy, measured against uniform sampling, while a random partition into the same number of sources gives no improvement at all.

Mapping the Pareto frontier. Each fit can be run efficiently. For illustrative purposes, we ran 48,000 of them here (16,000 weightings of 70 evaluations, at three proxy scales) to map the Pareto frontier across evaluations. For any desired balance between the evaluations, we can then read the matching mixture off the frontier.

Language to genomes. On a pool of DNA from OpenGenome2²³, the same pipeline organizes sequences by what they teach a genome model, with no biological taxonomy needed. It shows at a fine grain which DNA helps each evaluation, and the sources it up-weights combine sequences from several of OpenGenome2's original subsets.

The problem with hand-defined sources

Pretraining data comes in sources: the datasets a corpus was assembled from, or slices of one corpus such as topics. A mixture is a weighting over those sources, and each training batch is drawn from them in proportion to their mixing weights. When the pool holds more tokens than the training horizon, the weights determine which data is used. When it holds fewer, they determine how many times each source is repeated^a A source with weight $\lambda_j$ and size $t_j$ tokens is read $E_j = \lambda_j B / t_j$ times over a training horizon of $B$ tokens. A concentrated weight on a small source therefore means many passes over the same text..

How to identify optimal mixing weights. A family of methods picks the weights cheaply using proxies: train one small model per source, score it on examples drawn from the evaluations you care about, and read the weights off those scores^2–⁶.

Every method in this family takes the partition as given. What a mixture really decides is each document's exposure: how much of it the model sees over the training horizon. Fitting one weight per source ties the exposures of all its documents together, so a document can only be read more if everything grouped with it is read more too. A "home and hobbies" topic, for example, holds step-by-step guides as well as product listings, and a single weight has to cover both.

Tying exposures removes choices, so it never lowers the best achievable loss. We frame the best loss reachable over a set of allowed mixtures as a generalized entropy, known as H-entropy^7–¹², and we call the increase caused by the partition its H-entropy gap. The gap is zero when the documents tied together have the same effect on training, small when their effects are similar, and large when one source holds documents that pull the evaluations in different directions.

An intuitive view of the H-entropy gap

A mixture decides each document's exposure $a_d$ , the fraction of that document the model trains on, from 0 to 1. The training horizon fixes the total exposure, so giving one document more means giving others less.

Let $L(a)$ be the target loss after training with exposures $a$ . If every document could get its own exposure, the best reachable loss would be $L^\star$ . Grouping documents into sources ties their exposures. A source has a single weight, so every document in it gets the same exposure. With a partition $P$ into sources, the best reachable loss is $L_P$ , and the H-entropy gap of $P$ is

$\Delta_P = L_P - L^\star \ge 0.$

The gap is zero when the best exposures are already equal within every source, and it grows the more they differ within a source. $L^\star$ and $L_P$ are the H-entropies of the two problems, which gives the gap its name.

A small example. Take three documents, as in the figure below. Each split of the training budget among them is a point in the left triangle. Suppose the loss is lowest at a document optimum $a^\star$ and rises with squared distance from it, $L(a) = c + \|a - a^\star\|^2$ . Tying two documents allows only splits that give them equal exposure, the orange line. The best tied split is the point on this line closest to $a^\star$ , and the gap is the squared distance between the two points. For this loss, the gap equals half the squared difference between the optimal exposures of the two tied documents. Tying documents with similar optimal exposures costs little, and tying very different ones costs a lot. This is the intuition behind grouping documents that behave alike.

In the figure below, drag the document optimum or choose a pair to tie. The right panel sketches validation loss over training for the document optimum and the best tied split, with the gap as the difference at the end.

tie

allocations · drag the optimum

validation loss over training · illustrative

optimum (0.65, 0.20, 0.15)best with the tie (0.43, 0.43, 0.15)

For the formal setup, a bound on the gap, and a proof sketch, see the theory section at the end of the post.

So what would an ideal partition look like? Two properties matter:

Distinct. Sources should affect the target evaluations differently, so that different weights move different parts of performance.
Focused. Documents inside a source should contribute in the same way, so that a weight isn't diluted by unrelated text.

Neither property requires topics or other human-defined categories. Both depend on how documents behave when you train on them, which can be measured (Figure 2).

Topic-based sources

Ideal sources · distinct and focused

Figure 2. Each point is a document, placed by how training on it would change two evaluations. Topic-based sources (left) cut across the dense regions, so one weight moves documents with opposite effects; ideal sources (right) follow them, distinct from each other and focused within.

Both properties are ways of keeping the H-entropy gap small. At the end of the post we show that the gap is bounded by how far documents within each source sit from one another in behavior space, which is what clustering by behavior minimizes.

Grouping a corpus before mixing isn't new in itself. Recent work clusters documents by their content, using embeddings or trained classifiers, and mixes over the clusters^13,^14,¹⁵. Tessera instead groups documents by their measured effect on training against the evaluations you choose. This is most useful for data that is hard to interpret, such as DNA. For example, we rarely know the biological function of a given DNA sequence. We can still measure its effect on any evaluations we choose.

The method: Tessera

Tessera has three stages: measure a few thousand documents, generalize that measurement to the whole corpus, then cluster the corpus into sources and fit a mixture over them (Figure 3). The mixture fit is a standard proxy-based mixing method⁶, run unchanged on the new sources: Tessera works with any such method, because it changes only the sources the weights are fit over, not how they are fit.

new: derive the sourcesstandard proxy-based mixing

Stage 1

Measure

1Sample documents
Collect a few thousand documents, treated as individual datasets for the proxy models
2Document proxies
Fine-tune a nano proxy model on each document and score it on the target examples

Stage 2

Generalize

3Behavior axes
Reduce the scores to their top principal directions per evaluation
4Predict for every document
Train a regressor to predict these axes for every document in the corpus

Stage 3

Group and mix

5Cluster
Cluster the corpus in behavior space into thousands of sources
6Fit the mixture
Fit the weights with any proxy-based method; here, MixMin⁶
7Train and score
Train a model on the mixture and evaluate it

Figure 3. The Tessera pipeline. For a few thousand documents we directly measure each one's effect on the target evaluations, a cheap predictor carries the measurement to every document in the corpus, and the corpus is cut into sources that go into an ordinary proxy-based mixing method. The document measurements are paid for once and reused for every clustering and every mixture fit that follows.

Measure. We sample thousands of documents and train a separate nano proxy model on each. We start from a shared sub-100M-parameter base model that has already been trained on a uniform mix of the corpus, and continue training it for a few steps on that one document alone^b We start from a pretrained base model so that the measurement reflects how a document changes a model that already knows the language. The base model is trained on the whole corpus.. The proxy then scores held-out examples drawn from a set of chosen evaluations. The resulting vector of log-likelihoods is the document's label: a fingerprint of what that document teaches, example by example.

Generalize. For each evaluation, the labels form a matrix of documents by target examples. A singular value decomposition reduces it to its top directions of variation, so each document is summarized by a few coefficients per evaluation, which we call behavior axes. Only a few thousand documents have proxies, so we train a lightweight regressor to predict these coefficients from a document's content (hashed word counts plus length), then apply it to every document in the corpus.

Group and mix. We cluster the corpus in behavior space into k sources. From there, any proxy-based mixing method can fit the weights. We use MixMin⁶, which chooses the weights that maximize the likelihood of the target examples under a mixture of small proxy models, one per source; the problem is convex, so it solves in seconds on CPUs. We minimize the average loss across all target evaluations rather than on just one, with a cap on how often any source can be repeated (details in Methods below).

Method details

The label of document $d$ is the vector of log-likelihoods its proxy $p_d$ assigns to the $N$ target examples:

$\ell_d = \bigl[\log p_d(x_1),\ \ldots,\ \log p_d(x_N)\bigr] \in \mathbb{R}^N$

Behavior axes. For each evaluation $e$ , we stack the labels of the measured documents into a matrix $Y_e$ , with one row per target example of $e$ and one column per document, subtract each row's mean, and take the singular value decomposition $Y_e = U \Sigma V^\top$ . We drop the first component, which captures overall loss and does not distinguish between target examples, and keep the next $r$ components. Document $d$ 's behavior axes for $e$ are its coordinates on these components,

$z_{d,e} = \bigl[\sigma_2 V_{d,2},\ \ldots,\ \sigma_{r+1} V_{d,r+1}\bigr],$

standardized per evaluation so that no evaluation dominates the geometry.

Predicting the axes. Each document is represented by its word counts, hashed into a fixed-size vector, plus its log length. A ridge regression from these features to each axis is fit on the measured documents and then applied to every document in the corpus. Its 5-fold cross-validated correlation on an axis is that axis's reliability, which weights the axis in the clustering.

Clustering. We run k-means on the predicted, reliability-weighted axes, under an equal-token constraint. Proxy quality depends on how much data a proxy sees, and equal sources make the repetition cap identical across sources.

Repetition cap. With hundreds of small sources, an unconstrained fit tends to concentrate on a few of them and read them many times over, and repeated data is costly. We therefore cap each source at one pass over the training budget. The mixture over $K$ sources solves MixMin with this cap applied inside the optimization:

$\lambda^\star = \arg\max_{\lambda \in \Delta^K} \sum_{n=1}^{N} \log \sum_{j=1}^{K} \lambda_j\, p_j(x_n) \quad \text{s.t.} \quad \lambda_j \le t_j / B$

where $t_j / B$ is the weight at which source $j$ is read exactly once over a budget of $B$ tokens. Clipping the free solution afterwards does not work, because renormalizing pushes the clipped weight back over the bound.

Thousands of sources

To demonstrate the effectiveness of Tessera in a straightforward setting, we start with small language models trained on a standard web corpus^c WebOrganizer's Corpus-200B¹³, a public web corpus whose documents are labeled with 24 topic and 24 format domains. We take ten topic domains (57B tokens) and keep documents of at least 1,000 tokens (15.3M documents, 40B tokens). The 400M-parameter comparisons draw a fixed 20B-token pool from these; the scale-up uses all 40B. The ten topic labels also define the topic domains baseline. and scored on standard language benchmarks. Later in the post, we extend the same analysis to genomics.

We compare ways of partitioning the same pool of web documents, drawn from ten topic domains of the WebOrganizer corpus¹³. The pool is the full set of documents a mixture can draw from. Every run trains the same model on the same token budget, is fit with the same joint six-evaluation MixMin, and is scored on held-out examples from six evaluations^d HellaSwag¹⁶, PIQA¹⁷, WinoGrande¹⁸, OpenBookQA¹⁹, Natural Questions (open)²⁰ and MMLU²¹, each as 5-shot held-out loss. MMLU is scored on its auxiliary training set. Examples used to fit the mixture are disjoint from those used to score it. Error bars are one standard error over four training seeds.. Runs differ only in how documents are split into sources. The reference point is uniform sampling, with no mixture.

We start with 400M-parameter models. Here, derived sources deliver about four times the gain of the topic mixture (Figure 4, top). Gains are measured as reductions in held-out loss, averaged over the six evaluations^e Held-out loss is the cross-entropy a model assigns to text it was not trained on, here the evaluation examples, in nats per token. Lower is better; gains are measured against uniform sampling.; the derived mixture's 0.110 nats per token over uniform sampling is about a 3% reduction in loss. Against the topic mixture directly, derived sources win on five of six evaluations^f Here a single mixture targets the six-evaluation mean. Later in the post we target the evaluations explicitly, fitting tens of thousands of mixtures that weight them differently and mapping the trade-offs between them..

This scale is small enough to afford the sweep over source counts and the random-partition null that follow below. To test whether the advantage holds as models grow, we re-cut the full corpus into 1,280 sources, refit the mixture, and trained a 1B-parameter model from scratch, against the same three baselines (Figure 4, bottom). At this scale, the topic mixture roughly ties uniform sampling (+0.001) and random sources barely beat it (+0.010), while the derived mixture keeps a clear gain (+0.079), beats uniform on all six evaluations, and gains 1.4 to 3.5 accuracy points over uniform sampling on every task above chance.

derived

topics

random (null)

400M parameters · 4 seeds

Uniform

no mixture

Random sources

640, membership shuffled

Topic domains

10, hand-built taxonomy

Derived sources

640, by document behavior

0 (reference)

+0.004 ± 0.009

+0.026 ± 0.007

+0.110 ± 0.006

0.025

0.05

0.075

0.1

0.125

1B parameters · 2 seeds

Uniform

no mixture

Random sources

1,280, membership shuffled

Topic domains

10, hand-built taxonomy

Derived sources

1,280, by document behavior

0 (reference)

+0.010 ± 0.002

+0.001 ± 0.002

+0.079 ± 0.005

0.025

0.05

0.075

0.1

0.125

gain over uniform (nats/token)six-evaluation mean gain over uniform (nats per token, higher is better)

Figure 4. At 400M parameters, derived sources deliver about four times the gain of a topic mixture (top). At 1B parameters, only derived sources clearly beat uniform sampling (bottom). Bars show the drop in held-out loss versus uniform, averaged over six evaluations; whiskers show one standard error.

Does the clustering matter?

Derived sources differ from topics in two ways. There are many more of them, and they are grouped by behavior. Could the number of sources alone explain the gain? To test this, we keep everything from the 640-source derived run (the same pool, the same number of equal-sized sources and the same mixture fit) and change only one thing: documents are assigned to sources at random instead of by behavior. As Figure 4 shows, a random partition does not meaningfully improve on uniform sampling (+0.004 ± 0.009, within noise), while derived sources beat it by +0.106 ± 0.009, more than ten standard errors.

The mechanism is easy to see from the proxies' side. Random slices of a large corpus all look like the corpus, so their proxies are near-identical and the mixing method has nothing to choose between. The fit collapses toward uniform weights, which is why its result matches uniform sampling. Behavior clusters differ from each other by construction, so the fit has something to choose. Scaling the number of sources pays off only when each source groups documents with similar effects on the evaluations.

How finely to cut

More sources help, up to a point (Figure 5). Going from 160 to 640 sources raises the gain over uniform sampling from +0.090 to +0.110, because finer sources let the fit take part of a region instead of all of it. Going further to 1,280 sources maintains similar performance, so in this setting the gain plateaus between 640 and 1,280 sources. The fitting target matters too. In an ablation, fitting the mixture to the mean of all six evaluations, rather than to a single one, improves the six-evaluation mean by 0.034 nats per token. In the next section we go further, fitting mixtures to many different weightings of the evaluations.

400M parameters · 4 seeds per point

Figure 5. Finer sources help, and the gain plateaus between 640 and 1,280 sources. Each point is a full rerun of the Tessera pipeline (clustering, source proxies, mixture fit and training) at that number of sources, shown as the six-evaluation mean gain over uniform sampling, with one standard error over four seeds. Dashed lines mark the topic-domain and random-source mixtures, shown below a break in the y-axis; even the coarsest derived partition is well above both.

Methods: controls

Each row below tests an alternative explanation for the gain of derived sources, by changing one part of the setup and checking the result.

If the gain were due to…	Check	Result
balanced clustering, not the measurements	cluster the same pool on content features alone, with no document labels	ties the topic mixture
a bigger proxy budget	give the topic baseline 8× the proxy compute	within noise (+0.008 ± 0.022)
a preference for short documents	constrain the mixture to the pool's length profile and refit	the mixture does not change
a lucky split of target examples	refit on eight different example splits	the mixture does not change
the base model behind the labels	label documents with proxies from random initialization	keeps about a third of the gain
the extra fit parameters	randomize cluster membership, keep k and the fit	derived beats random by +0.106

So derived sources make for a better mixture. But a mixing method still returns a single mixture, and a single mixture is one fixed trade-off between evaluations. The joint fit here, for example, gives up some ground on NQ (open) to win the six-evaluation average. What if we wanted a different trade-off?

Tens of thousands of mixtures

The expensive part of Tessera, training source proxies, is paid once, and fitting a mixture over the resulting sources is a cheap, convex problem. So instead of fitting one mixture to one objective, we can fit many, and map the trade-offs between evaluations.

We do this with random scalarizations. Each draws a random weighting of the evaluations, and MixMin finds the best mixture for that weighted objective. Because the problem is convex, each such mixture is Pareto optimal^g A mixture is Pareto optimal if no other mixture can improve one evaluation without worsening another. The Pareto front is the set of all such mixtures.. Repeating this over many weightings traces out the Pareto front^h The weightings are drawn from Dirichlet distributions. Most spread their weight across all of the evaluations; a smaller set concentrates it on a handful, which reaches the edges of the front..

To show this at a broader scale, we use a suite of 70 evaluations: the six above, all 57 MMLU subjects, and benchmarks for code, math, science and commonsense reasoning. We fit 16,000 random weightings with proxies at three scales, 48,000 mixtures in allⁱ One proxy per source at each scale: 134M, 365M, and 941M parameters, trained on 5B, 14B, and 35B tokens of that source. The fitted weights correlate at 0.99 across scales, so Figure 6 shows the 16,000 mixtures from the smallest. Losses are the proxies' predictions for each mixture on held-out evaluation examples..

Figure 6 shows the result, with one point per mixture fit. Each point is placed by the first two principal components of its 70 evaluation losses, which roughly measure performance on STEM and code evaluations (horizontal) and on MMLU's humanities and social-science subjects (vertical). Together the points trace the Pareto front. The two menus pick a single mixture out of it: the first keeps the mixtures in the top 5% on one evaluation, and the second selects, among those, the one that is best on another. Hovering over any point shows that mixture's weights over the sources.

~16,000 mixtures · each fit to a random weighting of 70 evaluations

Every dot is a Pareto-optimal data mixture. The front lives in 70 dimensions (this is a 2D projection of it), so many optimal mixtures land inside the cloud rather than on its border.

Keep the top 5% onthen pick the best on

better on humanities & social-science MMLU →

Loading 16,000 mixtures…

better on STEM & code evaluations →

top 5% selected

Selected mixture

Figure 6.^j For interpretability, Figure 6 uses the ten original topic domains as sources, so each mixture's weights are easy to read. The same analysis applies unchanged to the full set of 1,280 derived sources; only the number of weights in each mixture grows. About 16,000 Pareto-optimal mixtures, each the MixMin solution for a random weighting of 70 evaluations. Points are placed by the first two principal components of their predicted evaluation losses, with each component's sign chosen so that moving right or up corresponds to lower predicted loss.

With the Pareto front in hand, choosing a mixture for an application becomes a lookup. Figure 6 opens on one example. Among the mixtures in the top 5% on HellaSwag (those with the lowest HellaSwag loss), the one with the best HumanEval loss is 69% science and technology and 11% health. Keeping the same HellaSwag requirement but asking for the best WinoGrande instead gives a mixture that is 76% social life, and asking for the best MMLU gives one that is mostly politics (42%) and home and hobbies (35%). The same constraint with a different second goal leads to very different mixtures, all without re-running the pipeline.

What is a document worth?

Every step of Tessera rests on a measurement of individual documents, which makes the pipeline useful in its own right as a way to look at a corpus. Related work on data attribution asks how individual training examples shape a model's predictions²². Here the question is narrower and cheaper: how does a document move the evaluations we chose, and what does the corpus look like when organized that way?

Figure 7 places every document in behavior space, on a 160-source partition of the pool (the results above use 640 and 1,280). Documents that are close together in this space have similar effects on their proxy models' evaluation performance. Selecting an evaluation highlights the sources the mixture up-weights when fit to that evaluation alone. The side panel shows the topic mix that results from the weights on the 160 derived sources. The topics themselves are never weighted, and each bar adds up sources that the fit treats very differently. Fit jointly to all six evaluations, for example, the mixture draws 12% of its data from health documents, close to their 11% share of the corpus, yet 37% of health documents sit in sources that get no weight at all.

40,000 documents · 160 sources · 2-D projection of 48 behavior axes

Loading 40,000 documents…

Figure 7. A map of the corpus, showing what each document teaches a language model. Pick an evaluation to color documents by source weight. Up-weighted documents are orange (darker for higher weight), and the rest are grey. The side panel shows topic shares of the mixture (bars) and corpus (ticks).

Although the pipeline never sees topic labels, the fitted weights line up with them in interpretable ways (Figure 8). Fitting to each evaluation separately and reading off the topic labels of the sampled tokens, MMLU leans toward politics and science, OpenBookQA toward science, WinoGrande toward social life and literature, and HellaSwag and PIQA toward health and home. Each evaluation draws on the topics closest to it.

share of sampled tokens by topic (%) · one fit per evaluation · 160 sources

Pool

HellaSwag

PIQA

WinoGrande

OpenBookQA

NQ (open)

MMLU

Health

10.5

23.5

27.1

4.8

13.6

6.4

8.1

Home & hobbies

8.4

22.9

25.8

16.2

17.4

11.4

4.5

Social life

10.0

12.5

13.3

30.3

8.2

12.0

5.8

Sports & fitness

9.6

8.5

8.6

10.8

5.2

11.0

10.2

Education & jobs

8.8

6.9

1.9

1.1

1.5

5.6

6.2

Finance & business

9.7

6.6

1.1

0.4

1.1

6.6

8.7

Science & tech

10.8

5.9

9.4

1.1

23.7

7.0

13.8

Literature

11.2

5.7

5.6

18.8

10.5

15.7

9.3

Politics

11.4

3.8

4.2

1.3

15.8

7.8

22.9

Entertainment

9.5

3.8

3.1

15.3

2.8

16.4

10.5

Figure 8. The derived sources and mixtures lead to interpretable topic shares, even though no step of the pipeline sees a topic label. Share of sampled tokens by topic, one fit per evaluation, on the 160-source partition, with the pool's own topic shares in grey for reference.

Within a topic, the measurement separates documents a topic label would lump together. The most heavily weighted sources for HellaSwag hold step-by-step instructional writing, such as "Aquaponics DIY Tilapia: Step-By-Step Advice", while the sources it down-weights hold news and recaps. Both kinds sit under the same few topic labels. This is exactly the selection a topic weight cannot make. The contrast is starkest in the 1,280-source partition used at larger scale (Figure 9). It shows three documents from an up-weighted source, which holds natural-history and field-guide writing, and three from a down-weighted source, which is almost entirely privacy policies and other legal boilerplate.

Up-weighted source

Natural history and field guides

Meet the different types of orcas… as wild orca research has expanded, researchers have described different forms or types…

Bee Flies. About 800 species in North America… Resembling bees, they are a family of true flies and are not bees at all…

Bobcat Facts. Bobcats are mainly carnivores… hunting rabbits, birds, bats, rodents and deer…

Down-weighted source

Privacy policies and other legal boilerplate

This Privacy Policy sets out how we collect, use and protect your personal information…

Our Commitment to Privacy… this notice explaining our online information practices…

…policies and practices regarding the collection, use and disclosure of personal information…

Figure 9. Excerpts from an up-weighted and a down-weighted source of the 1,280-source partition, each from a different document.

From language to genomes

The same questions matter even more for genome models. A genome model's corpus spans sequence from across the tree of life, and deciding what to train on is at least as consequential as it is for language. But the tools for drawing sources are much cruder. There is no topic taxonomy for DNA. The groupings that do exist (the datasets a corpus was assembled from, species, clade, annotation class) say what a sequence is, not what it would teach a model, and most of a genome can't be read by eye at all. Grouping by measured effect, as Tessera does, needs none of these labels.

We ran the same pipeline on a pool of DNA sequences drawn from OpenGenome2²³, the pretraining corpus behind Evo 2, which spans prokaryotic genomes, metagenomes, viral sequences, organelles and eukaryotic genic regions. Each sequence in the pool is a document. As example evaluations we use five held-out sets of human sequence by region (coding sequence, promoter, downstream, enhancer and ncRNA), and a Mendelian variant effect prediction (VEP) benchmark^k The region sets are held-out human sequences from coding sequence (CDS), promoter (upstream), downstream (the 256 bp after the coding sequence), enhancer (ENCODE cCREs) and ncRNA regions. The Mendelian VEP benchmark follows TraitGym²⁴, pairing pathogenic variants from ClinVar and OMIM with common gnomAD variants matched within each consequence class. As a mixing target, we use the reference sequence around each variant.. As with language, each measured sequence gets a nano proxy, its label is decomposed into behavior axes, a predictor carries those axes to the rest of the pool from simple sequence statistics (counts of short DNA words, plus sequence length), and the pool is clustered in behavior space into 160 sources. We use these mixtures as one input to how we pretrain Omnii¹. Here we focus on what the pipeline reveals about the pool itself (Figure 10).

40,000 DNA sequences · 160 sources · 2-D projection of behavior axes

Loading 40,000 sequences…

Figure 10. A map of the DNA pool, showing what each sequence teaches a genome model. Pick an evaluation to color sequences by source weight. Up-weighted sequences are blue (darker for higher weight), and the rest are grey. The side panel shows subset shares of the mixture (bars) and pool (ticks).

When fit to minimize the average loss over the six evaluations, the mixture up-weights 41 of the 160 sources, which hold about a quarter of the pool's tokens and concentrate in one region of the map (Figure 10). Relative to the pool, it shifts toward eukaryotic genic sequence (29% of the pool, 41% of the mixture) and away from prokaryotic and viral DNA: metagenomes fall from 14% to 4%, prokaryotic genomes from 7% to 1%, and viral sequence to almost nothing. As with language, these subset shares follow from the weights on the derived sources rather than being chosen per subset. Although 41% of the mixture is eukaryotic genic sequence, 70% of the eukaryotic genic sequences in the pool sit in sources that get no weight.

Two sources show the contrast up close (Figure 11). The most heavily up-weighted source holds vertebrate DNA, mostly birds and mammals, from kiwis and penguins to primates. Most of it lies between genes, with old repeat elements, but about four documents in ten contain genes with their exons and introns. The down-weighted source is far more uniform: soil and water bacteria, nearly all coding sequence, document after document. Bacterial DNA like this is abundant throughout the pool. Prokaryotic and viral sequence makes up about a fifth of it, and the mixture up-weights only one of the 18 sources made mostly of it. Each additional document of this kind likely adds little that the rest of the corpus does not already provide, so the fit learns to down-weight it.

Loading documents…

Figure 11. One hundred documents from a source the mixture up-weights (left) and one it down-weights (right), each drawn as an annotation track; choose a document to read it base by base below. Annotations are for reading only; the pipeline sees nothing but sequence.

As with language, sequences are grouped by their measured effect on the evaluations, and the annotations are used only afterward, to interpret the groups. For DNA, this kind of after-the-fact interpretation is often the only option, since the function of most sequences is unknown.

A theory of derived sources

Why should grouping documents by behavior give good sources? The opening section set mixing up as a decision problem: choose each document's exposure, with $L^\star$ the best loss when every exposure is free and $L_P$ the best when documents in the same source share one. The difference $\Delta_P = L_P - L^\star$ is the H-entropy gap, the cost of the partition $P$ ^7–¹². Treating every document as its own source makes the gap zero, but fitting the weights then needs a proxy per document, and fitting over millions of nano proxies is too expensive. The practical question is which partition into hundreds or thousands of sources keeps $\Delta_P$ small.

The gap can't be computed directly, but it can be bounded by what the pipeline measures. Let $D(P)$ be the within-source spread, how far documents sit from the center of their source in behavior space, which is exactly what k-means minimizes. Under assumptions stated in the box below, $\Delta_P$ is at most a sensitivity constant times $\sqrt{D(P)}$ , plus a reconstruction error for how much of each document's effect the behavior axes miss, plus a transfer error for how well proxy losses carry over to final training. Focused sources, whose documents behave alike (as in Figure 2), make $D(P)$ small. So clustering in behavior space tightens a bound on how much worse the derived sources can do than weighting every document individually. The box below gives the bound and a proof sketch.

Tessera from an H-entropy minimization perspective

A proxy for the loss. Keep the exposures $a$ , the loss $L$ and the gap $\Delta_P$ from the opening section. Each document $d$ has a nano proxy $p_d$ (step 2 of Figure 3), and the target examples are $x_1, \ldots, x_N$ . The document-level version of the MixMin objective mixes the proxies in proportion to exposure and scores the target examples under the mixture:

$q_a(x_n) = \frac{\sum_d a_d\, p_d(x_n)}{\sum_d a_d}, \qquad \hat L(a) = -\frac{1}{N}\sum_{n=1}^{N} \log q_a(x_n).$

$\hat L$ is the loss the proxies predict for allocation $a$ , and the mixture fit of the method minimizes it over tied allocations. Unequal document sizes weight the first sum by tokens, and a weighted evaluation suite weights the average over examples. If two documents have identical log-likelihood profiles $\ell_d$ and $\ell_e$ , tying them leaves $q_a$ unchanged, so the gap is zero. The bound covers profiles that are only similar.

The bound. The behavior axes of the Generalize step give each document coordinates $z_d$ , and the SVD basis is a linear decoder $T$ that maps coordinates back toward the full profile, $\ell_d \approx c + T z_d$ . Three quantities enter:

the within-source spread $D(P)$ , the mean squared distance from $z_d$ to the center of its source, which k-means minimizes in the Cluster step;
the reconstruction error $\epsilon_R$ , the root mean square of what the truncated axes and the regressor miss, with the decoder gain $\Vert T \Vert$ , the largest amplification of a coordinate difference;
the transfer error $\epsilon_T$ , how far the proxy loss can sit from the true loss, $|L(a) - \hat L(a)| \le \epsilon_T$ up to a constant offset, for every allocation.

If in addition no proxy outweighs the mixture by more than a ratio bound $\Gamma$ , that is $p_d(x_n) / q_a(x_n) \le \Gamma$ throughout, then

$\boxed{\;\Delta_P \;\le\; 2\epsilon_T + C \bigl(\Vert T \Vert \sqrt{D(P)} + \epsilon_R\bigr)\;}, \qquad C = \Gamma \sqrt{\tfrac{1-\rho}{\rho}},$

where $\rho$ is the training horizon as a fraction of the corpus and the sensitivity $C$ grows with the ratio bound and shrinks with the horizon. Read left to right: what the proxies cannot see, what the clustering controls, and what the axes miss.

Proof sketch. Take the best document-level allocation $a^\star$ and average it within each source. The average uses the same budget and is a valid tied allocation, so $\Delta_P \le L(\bar a^\star) - L(a^\star)$ . Moving from $L$ to $\hat L$ costs $\epsilon_T$ at each allocation. For $\hat L$ , the bound $\log x \le x - 1$ and Cauchy–Schwarz split the loss from averaging into two factors: how much exposure varied within sources, at most $\rho(1 - \rho)$ , and how much documents' marginal value varied within sources. The ratio bound $\Gamma$ controls the second factor by differences in profiles, and decoding coordinates into profiles gives the $D(P)$ and $\epsilon_R$ terms.

From the bound to the pipeline. With the coordinates and error terms fixed, the partition that minimizes $D(P)$ minimizes the bound, though not necessarily the true gap, and splitting a source never increases the true gap. The mixture fit adds its own term: if the source-proxy objective is within $\epsilon_F$ of $L$ on tied allocations and the solver is $\delta$ -optimal, the fitted allocation $\hat a$ satisfies

$L(\hat a) - L^\star \;\le\; 2\epsilon_T + C \bigl(\Vert T \Vert \sqrt{D(P)} + \epsilon_R\bigr) + 2\epsilon_F + \delta.$

Pipeline step	Object	Role in the bound
Document proxies (2)	$p_d$ , profiles $\ell_d$	bounded likelihood ratios give $\Gamma$
Behavior axes and prediction (3, 4)	$z_d$ , $T$	reconstruction error $\epsilon_R$ , decoder gain $\Vert T \Vert$
Cluster (5)	$P$ , $D(P)$	within-source spread, the term clustering controls
Fit the mixture (6)	$\hat a$	fit error $\epsilon_F$ and solver error $\delta$
Train and score (7)	$L$	transfer error $\epsilon_T$ , assumed rather than measured

The bound is only as useful as its assumptions. The behavior axes must reconstruct full profiles (small $\epsilon_R$ , moderate $\Vert T \Vert$ ), no proxy may dominate a mixture (moderate $\Gamma$ ), and proxy losses must track final training across all allocations (small $\epsilon_T$ ). The conditional theorem, from these assumptions to the bound, is machine-checked in Lean 4.

Closing

Data mixing is usually a single fit of a few weights over predefined sources. By estimating model behavior for each document, Tessera instead scales data mixing at both ends. On the input side, a corpus can be cut into thousands of sources that each behave consistently, and a mixture over them delivers four times the gain of a topic taxonomy, with the gain holding at larger scale. On the output side, fitting is cheap enough to map tens of thousands of mixtures and pick the desired trade-off. And the same measurement lets us view each document in a corpus by its behavior on the target evaluations. Since none of this is specific to text documents, we apply the same pipeline to DNA, where sources have meant taxonomic groups or the dataset a sequence came from. Grouping sequences by what each one teaches the model is one ingredient in how we mix pretraining data for Omnii, our multimodal genome model, across more than twenty modalities.

Building general biological intelligence means making decisions like these about data at every stage of training. If you want to work on problems like this, we'd like to hear from you.

We're hiring → join us

Citation

@misc{data_mixing_2026,
  title        = {Scaling data mixing for language and DNA pretraining},
  author       = {Neiswanger, Willie and Carter, Matthew and Thudi, Anvith and Poli, Michael},
  year         = {2026},
  month        = {Oct},
  url          = {https://www.radicalnumerics.ai/blog/tessera-data-mixing},
  organization = {Radical Numerics}
}

References

Radical Numerics. "A new frontier in generative genomics with Omnii." (2026).
Xie, S.M., Pham, H., Dong, X., et al. "DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining." Advances in Neural Information Processing Systems (NeurIPS) 36 (2023).
Liu, Q., Zheng, X., Muennighoff, N., et al. "RegMix: Data Mixture as Regression for Language Model Pre-training." International Conference on Learning Representations (ICLR) (2025).
Ye, J., Liu, P., Sun, T., et al. "Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance." International Conference on Learning Representations (ICLR) (2025).
Chen, M.F., Hu, M.Y., Lourie, N., et al. "Aioli: A Unified Optimization Framework for Language Model Data Mixing." International Conference on Learning Representations (ICLR) (2025).
Thudi, A., et al. "MixMin: Finding Data Mixtures via Convex Minimization." International Conference on Machine Learning (ICML) (2025).
Raiffa, H., Schlaifer, R. "Applied Statistical Decision Theory." Division of Research, Graduate School of Business Administration, Harvard University (1961).
DeGroot, M.H. "Uncertainty, Information, and Sequential Experiments." The Annals of Mathematical Statistics 33(2), 404–419 (1962).
Grünwald, P.D., Dawid, A.P. "Game Theory, Maximum Entropy, Minimum Discrepancy and Robust Bayesian Decision Theory." The Annals of Statistics 32(4), 1367–1433 (2004).
Neiswanger, W., Yu, L., Zhao, S., Meng, C., Ermon, S. "Generalizing Bayesian Optimization with Decision-theoretic Entropies." Advances in Neural Information Processing Systems (NeurIPS) 35 (2022).
Zhao, S., Sinha, A., He, Y., Perreault, A., Song, J., Ermon, S. "Comparing Distributions by Measuring Differences that Affect Decision Making." International Conference on Learning Representations (ICLR) (2022).
Poli, M., Massaroli, S., Ermon, S., Wilder, B., Horvitz, E. "Ideal Abstractions for Decision-Focused Learning." International Conference on Artificial Intelligence and Statistics (AISTATS) (2023).
Wettig, A., Lo, K., Min, S., et al. "Organize the Web: Constructing Domains Enhances Pre-Training Data Curation." International Conference on Machine Learning (ICML) (2025). Code: github.com/CodeCreator/WebOrganizer; corpus: WebOrganizer/Corpus-200B.
Diao, S., Yang, Y., Fu, Y., et al. "CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training." arXiv:2504.13161 (2025).
Ge, A., Huang, T.-H., Cooper, J., et al. "R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training." arXiv:2505.00358 (2025).
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., Choi, Y. "HellaSwag: Can a Machine Really Finish Your Sentence?" Annual Meeting of the Association for Computational Linguistics (ACL) (2019).
Bisk, Y., Zellers, R., Le Bras, R., Gao, J., Choi, Y. "PIQA: Reasoning about Physical Commonsense in Natural Language." AAAI Conference on Artificial Intelligence (2020).
Sakaguchi, K., Le Bras, R., Bhagavatula, C., Choi, Y. "WinoGrande: An Adversarial Winograd Schema Challenge at Scale." AAAI Conference on Artificial Intelligence (2020).
Mihaylov, T., Clark, P., Khot, T., Sabharwal, A. "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering." Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018).
Kwiatkowski, T., Palomaki, J., Redfield, O., et al. "Natural Questions: A Benchmark for Question Answering Research." Transactions of the Association for Computational Linguistics 7, 452–466 (2019).
Hendrycks, D., Burns, C., Basart, S., et al. "Measuring Massive Multitask Language Understanding." International Conference on Learning Representations (ICLR) (2021).
Ilyas, A., Park, S.M., Engstrom, L., Leclerc, G., Madry, A. "Datamodels: Predicting Predictions from Training Data." International Conference on Machine Learning (ICML) (2022).
Brixi, G., Durrant, M.G., et al. "Genome modelling and design across all domains of life with Evo 2." Nature (2026).
Benegas, G., Albors, C., Aw, A.J., Ye, C., Song, Y.S. "A DNA language model benchmark for the prediction of causal disease variants." bioRxiv (2025).