Text-to-image generative modeling without language pretraining

At Radical Numerics, we study the science of multimodal pretraining, motivated by the scale and complexity of biology. Our Omnii genome language models, for example, are trained on dozens of heterogeneous modalities, each with distinct characteristics. This raises two fundamental questions: how should modalities be combined, and how should pretraining compute be allocated across them? In this post, we share our latest work on multimodal pretraining, focusing on text-to-image (T2I) generative modeling to study these questions. T2I provides a well-established setting with public datasets, benchmarks, and strong baselines.

Text-to-image (T2I) generation maps natural-language descriptions to images. Modern T2I systems use internet-scale language pretraining^aa Language pretraining refers to pretraining on web-scale text-only corpora, excluding image-paired captions. for text-conditioned image generation. Language pretraining is integrated into T2I systems through two designs, distinguished by how and when text tokens interact with image tokens. In late fusion, the predominant design, a language model is pretrained separately on text-only data, and its output embeddings condition a separate image generator via cross-attention or joint-attention layers^1,2,3,4. Early fusion removes the separate text encoder but retains language pretraining within a single multimodal model trained on both text-only and image–text data, with text tokens serving as a prefix that conditions image generation^5,6,7,8. In both designs, language pretraining dominates the compute budget^8,9. We estimate the total pretraining compute of representative T2I systems, including the language model, image tokenizer and image generator, and break it down into language and vision compute in Figure 1.

Is language pretraining necessary for strong T2I generation? We introduce Halcyon, a family of early-fusion autoregressive models (0.1B–4B parameters) trained entirely from scratch on 100 million image–text pairs from GPIC¹⁰, without language pretraining. Halcyon outperforms both late- and early-fusion methods that use language pretraining, with substantially less total pretraining compute.

Highlights

• Strong T2I generation without language pretraining. Halcyon-0.52B outperforms LlamaGen-XL/8^bb LlamaGen⁹ is a popular autoregressive model for visual generation. The T2I variant uses a 1.22B FLAN-T5-XL text encoder and a 0.90B image generator. (48.0 vs. 63.3 gFD; lower is better) using 18.2× less total pretraining compute. Halcyon-1.6B outperforms JiT-T2I-XXL/8++^cc JiT-T2I¹² is a popular pixel-space diffusion model for T2I generation. It uses a 1.7B Qwen3 text encoder and a 1.1B image generator; "++" adds perceptual loss. (35.3 vs. 41.0 gFD) using 275× less. Halcyon models outperform LlamaGen-XL/8 and JiT-T2I-XXL/8++ on GenEval¹³ and DPG-Bench¹⁴.
• Language pretraining does not help T2I generation. We examine this through three experiments. Replacing 10–50% of a fixed budget with text-only pretraining worsens gFD (48.0 → 51.6–65.1 gFD). Adding text-only pretraining on top of a full T2I epoch (1.5× total compute) does not improve T2I generation (49.2 vs. 48.0 gFD). Continual pretraining from Qwen3-0.6B-Base¹⁵ (36T text tokens) gives no gain over the same architecture trained from scratch (47.8 vs. 46.6 gFD).
• Image captioning under the same pretraining budget. Classifier-free guidance¹⁶ combines text-conditioned and unconditional image predictions during sampling. The unconditional branch is typically trained by dropping captions from a fraction of T2I examples. Instead, we replace these caption-dropped examples with image-to-text (I2T) examples, where the model predicts image tokens unconditionally, followed by a caption conditioned on the image. This enables image captioning at no additional pretraining cost while maintaining competitive image generation quality.
• Open models and recipes. We release ten Halcyon models, from 0.1B to 4B parameters.

Halcyon: A Family of Early-Fusion Multimodal Autoregressive Models

Halcyon is a decoder-only Transformer that models text and images as one sequence of discrete tokens, trained with next-token prediction^5,6. Every component is trained from scratch on GPIC: the image tokenizer, the text tokenizer and the generative model.

Tokenizers. The image tokenizer follows the Cosmos encoder–decoder design with finite scalar quantization^17,18. It maps each 256×256 image to a 32×32 grid of 1,024 tokens drawn from 64,000 codes. The text tokenizer is a byte-level BPE¹⁹ with 16,384 tokens, trained on GPIC captions and OpenWebText.

One vocabulary, four sequence formats. Text, image and special tokens share a single vocabulary of 80,640 entries, with one embedding table and one output head. Figure 2 shows the four formats: text-only, image-only, image-to-text (I2T) and text-to-image (T2I). An I2T sequence starts with the same image prefix as a T2I sequence whose caption has been dropped. We return to this in Image Captioning at No Extra Cost.

Models. We train five sizes, from 0.1B to 4B parameters, using the Qwen3 architecture¹⁵ with random initialization. Each model sees GPIC once: 100 million image–text pairs at 256×256, over 97,632 steps at batch size 1,024.

A Controlled Comparison of T2I Models

T2I methods are routinely compared on common benchmarks such as GenEval and DPG-Bench, despite being pretrained on different datasets. This makes it difficult to determine whether performance differences come from model design or training data. To address this, we train all image generators on the same 100 million image–text pairs from GPIC at 256×256 resolution and evaluate them using GPIC metrics, GenEval, and DPG-Bench.

Late fusion. We retrain two representative late-fusion methods, LlamaGen^ee LlamaGen code: github.com/FoundationVision/LlamaGen. and JiT-T2I^ff JiT-T2I code: github.com/keshik6/gpic/baselines/PixelGen., with the authors' official code. Together they cover autoregressive and diffusion generators, encoder–decoder and decoder-only language models, and 256 and 1,024 image tokens. For each, we train the original configuration and two variants:

• LlamaGen. LlamaGen-XL/8 replaces the original image tokenizer with ours (256 → 1,024 image tokens). LlamaGen-XL-T5-Base/8 then replaces FLAN-T5-XL with FLAN-T5-Base, to measure the effect of language-model size on T2I generation.
• JiT-T2I. JiT-T2I-XXL/16++ adds a perceptual loss, which is used to train the image tokenizers of Halcyon and LlamaGen. JiT-T2I-XXL/8++ further reduces the patch size from 16×16 to 8×8, increasing the image sequence length from 256 to 1,024 tokens.

Configuration	Generator	Frozen language model	Text pretraining	Image tokens	Perceptual loss
LlamaGen-XL	AR, 0.78B	FLAN-T5-XL, 1.22B	~1T	256	✓
LlamaGen-XL/8	AR, 0.90B	FLAN-T5-XL, 1.22B	~1T	1,024	✓
LlamaGen-XL-T5-Base/8	AR, 0.90B	FLAN-T5-Base, 0.11B	~1T	1,024	✓
JiT-T2I-XXL/16	Diffusion, 1.10B	Qwen3-1.7B	36T	256	—
JiT-T2I-XXL/16++	Diffusion, 1.10B	Qwen3-1.7B	36T	256	✓
JiT-T2I-XXL/8++	Diffusion, 1.10B	Qwen3-1.7B	36T	1,024	✓
Halcyon	AR, 0.1–4B	—	—	1,024	✓

All configurations are trained for one epoch on GPIC at 256×256. We compare Halcyon against the strongest variant of each method: LlamaGen-XL/8 (63.3 gFD, from 147.0) and JiT-T2I-XXL/8++ (41.0 gFD, from 67.9).

Early fusion. Early-fusion systems add language pretraining in one of two ways. We reproduce both within Halcyon's training setup:

• Language-first pretraining. We allocate 10%, 20%, or 50% of Halcyon-0.52B's fixed pretraining budget to text-only pretraining, followed by T2I training. The text-only token allocations exceed the Chinchilla-optimal ratio of 20 training tokens per parameter²⁰. An additional variant performs text-only pretraining before a full T2I epoch (1.5× total compute).
• Continual pretraining. We initialize from Qwen3-0.6B-Base¹⁵, pretrained on 36T text tokens, extend its vocabulary with our 64,000 image tokens (216K total), and continue pretraining on GPIC. As a control, we train the same architecture and vocabulary from scratch under the same pretraining budget.

Halcyon vs. Late Fusion

Generation quality. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2× less total pretraining compute. Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275× less total pretraining compute. Both comparisons use the strongest variant of each baseline, with the same image sequence length of 1,024 tokens (Figure 1).

Prompt adherence. gFD measures how closely generated images match the real image distribution, not how well they follow text prompts. We therefore also evaluate on GenEval¹³, which tests compositional generation (object counts, colors, and positions), and DPG-Bench¹⁴, which tests adherence to long, detailed prompts. Halcyon-0.52B outperforms LlamaGen-XL/8 on both benchmarks (44.91 vs. 30.48 on GenEval; 76.84 vs. 67.08 on DPG-Bench). It also outperforms JiT-T2I-XXL/8++ on GenEval (44.91 vs. 43.17) and matches it on DPG-Bench (76.84 vs. 76.85). Halcyon-1.6B outperforms JiT-T2I-XXL/8++ on both benchmarks (53.48 on GenEval; 77.90 on DPG-Bench). At 4B, GenEval improves further to 56.39, while DPG-Bench remains at 77.90. Figure 3 compares these scores alongside images generated by each model from identical prompts.

Does Language Pretraining Help T2I Generation?

(1) A larger pretrained language encoder. In LlamaGen-XL/8, replacing FLAN-T5-Base (0.11B) with FLAN-T5-XL (1.22B)²¹ improves gFD only from 66.7 to 63.3, despite 13.5× more language-model pretraining compute.

(2) Language-first pretraining. Allocating 10%, 20%, or 50% of the budget to text-only pretraining worsens gFD from 48.0 to 51.6, 52.8, and 65.1. Running text-only pretraining before a full T2I epoch, at 1.5× compute, does not help either (49.2 vs. 48.0 gFD).

(3) Continual pretraining. Initializing from Qwen3-0.6B-Base gives no gain over training from scratch (47.8 vs. 46.6 gFD).

Across all three, language pretraining does not improve T2I generation in our setting. Scaling the language encoder buys a small gain at a large compute cost.

Model Scaling, Classifier-Free Guidance, and Caption Length

(1) Model scaling. Validation NLL decreases with model size for both image and text tokens, throughout pretraining. gFD drops from 144.2 at 0.1B to 28.4 at 4B.

(2) Classifier-free guidance. The optimal guidance scale decreases with model size, from 7.5 at 0.1B to 4.5 at 4B. Guidance improves gFD at every size.

(3) Caption length. On GPIC Test-1M, medium and long captions give lower gFD than short captions at every model size.

Image Captioning at No Extra Cost

Classifier-free guidance training discards text. To train the unconditional image branch, the caption is deleted from 10% of T2I examples: the model sees those images but never their captions.

We move the caption instead of deleting it. Placed after the image, the caption leaves the image prefix untouched, so the unconditional branch trains exactly as before (Figure 2). The caption becomes a prediction target. With the same data, the same pretraining budget and the same image tokens, the model now captions images.

The model learns to caption. The standard recipe never predicts a caption from an image; at 0.52B, its image-conditioned text NLL is 18.41. With I2T examples, it drops to 1.10. Caption quality grows with scale: Qwen3-VL-Reranker-8B alignment rises from 0.09 at 0.1B to 0.56 at 4B, against 0.89 for real GPIC captions (Figure 6).

T2I generation holds. Across all five sizes, gFD, GenEval and DPG-Bench show no consistent gain or loss against the standard recipe.

Closing Thoughts

Halcyon demonstrates that strong T2I generation does not require internet-scale language pretraining. Models trained entirely from scratch on image–text pairs can outperform modern T2I systems that rely on pretrained language models, while using substantially less total pretraining compute. Our controlled experiments further show that language pretraining does not improve T2I generation under the evaluated settings, while image captioning can be learned under the same pretraining budget.

At Radical Numerics, these findings connect to our broader interest in multimodal pretraining: understanding how to jointly model different modalities, which training objectives are effective, and how to allocate pretraining compute across modalities. Our Omnii models build on this broader direction by jointly modeling more than twenty biological modalities within a single sequence model.

We're hiring → join us

References

1. Chitwan Saharia, William Chan, Saurabh Saxena et al. "Photorealistic text-to-image diffusion models with deep language understanding." NeurIPS (2022).

2. Junsong Chen, Jincheng Yu, Chongjian Ge et al. "PixArt-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis." ICLR (2024).

3. Black Forest Labs. "FLUX.2." GitHub (2025).

4. Bing Zhao, Chenfei Wu, Deqing Li et al. "Qwen-image-2.0 technical report." arXiv:2605.10730 (2026).

5. Chameleon Team, Mingda Chen, Jacob Kahn et al. "Chameleon: Mixed-Modal Early-Fusion Foundation Models." arXiv:2405.09818 (2024).

6. Xinlong Wang, Yufeng Cui, Jinsheng Wang et al. "Multimodal learning with next-token prediction for large multimodal models." Nature 650(8101), 327–333 (2026).

7. Chunting Zhou, Lili Yu, Arun Babu et al. "Transfusion: Predict the next token and diffuse images with one multi-modal model." ICLR (2025).

8. Zhiheng Liu, Weiming Ren, Xiaoke Huang et al. "Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation." arXiv:2604.24763 (2026).

9. Peize Sun, Yi Jiang, Shoufa Chen et al. "Autoregressive model beats diffusion: Llama for scalable image generation." arXiv:2406.06525 (2024).

10. Keshigeyan Chandrasegaran, Kyle Sargent, Suchir Agarwal et al. "GPIC: A Giant Permissive Image Corpus for Visual Generation." arXiv:2605.30341 (2026).

11. Tianhong Li, Kaiming He. "Back to basics: Let denoising generative models denoise." CVPR (2026).

12. Zehong Ma, Ruihan Xu, Shiliang Zhang. "PixelGen: Improving pixel diffusion with perceptual supervision." arXiv:2602.02493 (2026).

13. Dhruba Ghosh, Hannaneh Hajishirzi, Ludwig Schmidt. "GenEval: An object-focused framework for evaluating text-to-image alignment." NeurIPS (2023).

14. Xiwei Hu, Rui Wang, Yixiao Fang et al. "ELLA: Equip diffusion models with llm for enhanced semantic alignment." arXiv:2403.05135 (2024).

15. An Yang, Anfeng Li, Baosong Yang et al. "Qwen3 technical report." arXiv:2505.09388 (2025).

16. Jonathan Ho, Tim Salimans. "Classifier-free diffusion guidance." arXiv:2207.12598 (2022).

17. Niket Agarwal, Arslan Ali, Maciej Bala et al. "Cosmos world foundation model platform for physical AI." arXiv:2501.03575 (2025).

18. Fabian Mentzer, David Minnen, Eirikur Agustsson, Michael Tschannen. "Finite scalar quantization: VQ-VAE made simple." ICLR (2024).

19. Rico Sennrich, Barry Haddow, Alexandra Birch. "Neural machine translation of rare words with subword units." ACL (2016).

20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. "Training compute-optimal large language models." NeurIPS (2022).

21. Hyung Won Chung, Le Hou, Shayne Longpre et al. "Scaling instruction-finetuned language models." JMLR 25(70), 1–53 (2024).