HuggingFace Papers
7/10 signal
Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
modelsresearch
Summary
This paper introduces SpectraReward, a training-free reward function that converts pretrained Multimodal Large Language Models (MLLMs) into zero-shot reward models for text-to-image generation. Instead of explicitly judging an image, the method calculates the log-likelihood of recovering the original prompt from the generated image in a single forward pass. The authors also propose Self-SpectraReward, a variant where a unified multimodal model uses its own understanding component to reward its generation component, enabling self-improvement without external models. Experiments show this approach consistently improves generation performance and outperforms prior MLLM-based reward methods.
Problem
Aligning text-to-image models with complex user prompts often requires reinforcement learning (RL), which in turn relies on reward models. These reward models are typically expensive to create, requiring extensive human preference data and fine-tuning. This work addresses the need for an efficient, training-free method to derive effective reward signals for text-to-image RL directly from existing pretrained MLLMs.
Method
The proposed method, SpectraReward, uses a pretrained MLLM as a reward model without any fine-tuning. The reward is calculated as the average image-conditioned log-likelihood of the original text prompt, measured via a single teacher-forced forward pass through the MLLM. A special case, Self-SpectraReward, is introduced for unified multimodal models, where the model's own vision-language understanding branch serves as the reward function for its image generation branch, creating a closed-loop self-improvement framework.
Details
- SpectraReward and Self-SpectraReward were shown to significantly and consistently improve the performance of text-to-image diffusion models.
- The proposed methods outperformed prior MLLM-derived reward training techniques in experimental comparisons.
- The framework was extensively validated across a wide range of configurations: 2 diffusion models, 3 RL algorithms, 9 MLLM backbones (from 4B to 235B parameters), and 5 out-of-distribution text-to-image benchmarks.
- Analysis revealed that simply using a larger MLLM as the reward model does not always lead to better results.
- The Self-SpectraReward variant was found to match or even surpass the performance of much larger external reward models, suggesting that alignment between the policy (generator) and the reward model is a critical factor for effective RL-based image generation.
What's new
The core contribution is a novel, training-free method for multimodal reward modeling that reframes the reward signal as the model's ability to 'read back' the prompt from the image. The introduction of Self-SpectraReward is particularly new, demonstrating a closed-loop, self-improvement mechanism for unified text-to-image models that does not require any external reward models, preference data, or knowledge.
Conclusion
Pretrained MLLMs can serve as effective, off-the-shelf, zero-shot reward models for text-to-image generation using the proposed SpectraReward function. This approach bypasses the need for preference labels and reward-model fine-tuning. The effectiveness of Self-SpectraReward highlights that strong alignment between the reward model and the generation policy is a key factor for successful reinforcement learning in this domain, sometimes being more important than the absolute size of the reward model.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.