Welcome FLUX.3 Image Klein in Diffusers

Hugging Face Blog1d ago

Black Forest Labs has officially unveiled FLUX.3 Image Klein, a groundbreaking model that arrives with full, day-one support for the Diffusers ecosystem. This release marks a significant milestone in generative AI, consolidating text-to-image, multi-reference image-to-image, and advanced layout control into a single, highly flexible architecture.

Whether you are looking for rapid inference via distilled checkpoints or deep, granular control over image composition, FLUX.3 Image Klein is designed to handle it all.

A Unified Approach to Image Generation

The core innovation of FLUX.3 is its versatility. By utilizing a single model for multiple modalities, Black Forest Labs has streamlined the workflow for creators and developers alike. The collection features two primary checkpoints:

black-forest-labs/flux-3-image-klein: A time-step distilled model optimized for high-speed inference, requiring only 16 steps and no classifier-free guidance (CFG).
black-forest-labs/flux-3-image-klein-base: The foundational model, designed for 50-step generation with CFG, serving as the primary target for fine-tuning.

For those working with hardware constraints, the model supports NVFP4 and FP8 weights, which can be loaded via from_single_file to ensure compatibility with smaller GPUs without sacrificing the model's core capabilities.

---

Getting Started with the Pipeline

Integrating FLUX.3 into your existing stack is straightforward. By installing the latest versions of diffusers and transformers (specifically for the Qwen3-VL text encoder), you can initialize the pipeline with minimal code:

import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

# Generate your first image
pipe(prompt="a red fox", output="images")[0].save("fox.png")

---

Advanced Capabilities: Beyond Text-to-Image

#### Text-to-Image Mastery FLUX.3 thrives on descriptive, scene-rich prompts. By providing details regarding the subject, lighting, composition, and artistic style, users can achieve highly specific results. The distilled checkpoint is particularly impressive, capable of producing a 1024 × 1024 image in approximately two seconds on an NVIDIA H100.

#### Multi-Reference Image-to-Image The model excels at maintaining consistency across multiple references. By passing a list of images and referencing them in your prompt using tags like <ref_0> or <ref_1>, you can instruct the model to blend specific elements—such as a character from one image and a setting from another—into a cohesive new output.

#### Precision Layout Control Perhaps the most exciting feature is the introduction of bounding-box layout control. By appending a JSON list of regions to your prompt, you can define exactly where objects should appear. Each region includes an id, a bbox (normalized on a 0–1000 scale), and a descriptive tag. This allows for complex compositions where the model respects the spatial constraints defined by the user.

Furthermore, this system supports spatial editing. By defining src_bbox and tgt_bbox, users can move specific objects within a reference image while keeping the background and other elements perfectly preserved.

---

Architecture and Technical Deep Dive

FLUX.3 is a latent-space flow model that processes three distinct types of tokens: text, the noisy image being generated, and optional reference images.

#### The VAE and Tokenization The model employs the FLUX.2 VAE architecture, which compresses images by a factor of 8 in each direction. Every 2 × 2 patch of latents is packed into a single token, meaning one token represents a 16 × 16 pixel patch. This results in a 64 × 64 grid of 4,096 tokens for a standard 1024 × 1024 output.

#### The Qwen3-VL Text Encoder Unlike models that rely solely on the final layer of a language model, FLUX.3 reads hidden states from eight different layers (4, 8, ..., 32) of the Qwen3-VL encoder. These are stacked and projected into the transformer’s 3072-width dimension, providing the model with a richer, multi-depth understanding of the prompt.

#### Two-Stage Processing The transformer operates in two distinct stages: 1. Per-mode blocks: Text, the noisy image, and reference images are processed independently in their own dedicated blocks. 2. Joint blocks: The streams are concatenated, allowing for cross-attention where the image learns directly from the text and reference inputs.

This separation ensures that each modality is refined on its own terms before the final integration, leading to higher fidelity and better adherence to complex instructions.

---

Performance and Optimization

For users concerned about hardware requirements, the Modular Diffusers framework offers significant flexibility.

Memory Management: By utilizing ComponentsManager, users can enable automatic CPU offloading, moving components to the GPU only when necessary.
Quantization: The use of fp8r and nvfp4 formats allows for substantial reductions in VRAM usage. For instance, the nvfp4 variant can run with a peak allocation of approximately 15.3 GiB post-inference on an RTX 6000.
Speed: On an H100, the distilled checkpoint delivers consistent performance, with text-to-image generation clocking in at under 2 seconds.

"FLUX.3 is a flow-matching model. Generation starts from pure noise, and at each of the 50 steps, the transformer predicts a velocity—the direction from noise toward a clean image."

---

LoRA Fine-Tuning

FLUX.3 is an ideal candidate for custom fine-tuning. The diffusers library provides dedicated DreamBooth LoRA training scripts for both text-to-image and image-to-image tasks. Whether you are training a specific artistic style or a character, the model's modular nature allows for efficient adaptation.

Key optimization flags like --gradient_checkpointing and --cache_latents are available to ensure that training remains accessible even on hardware with limited VRAM.

Final Thoughts

With its modular design, powerful layout control, and seamless integration into the Diffusers library, FLUX.3 Image Klein represents a significant leap forward for the generative AI community. Whether you are a developer building the next generation of creative tools or an artist looking for precise control, this model provides the architecture to bring your vision to life.

We encourage the community to explore the Hugging Face collection, experiment with the provided pipelines, and share your custom LoRAs on the Hub. The future of image generation is here, and it is more modular than ever.

Read Original Source