Exploring MLLM-Diffusion Information Transfer with
MetaCanvas

Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation.

Figure 1: Overview of the MetaCanvas framework. MetaCanvas tokenizes the text and encodes it using the MLLM’s text embedder, while user-provided images and videos are encoded using both the MLLM’s visual encoder and the VAE encoder. The text embeddings produced by the MLLM are passed through a lightweight MLP connector and used as conditioning for the DiT. In addition, we append a set of learnable multidimensional canvas tokens to the MLLM input, which are processed using multimodal RoPE (Bai et al., 2025b). The resulting canvas embeddings are then fused with the noisy latents through a lightweight transformer-based connector with two blocks. Connector details are illustrated below. Green tokens represent media context tokens, blue tokens represent text context tokens, and purple tokens represent the canvas tokens.

The canvas connector comprises a vanilla Transformer block and a Diffusion Transformer (DiT) block. The vanilla Transformer block transforms the learnable canvas tokens to align them with the DiT latent space. The second DiT block adopts a ControlNet-style design, where the transformed canvas tokens and the noisy latents are first combined and then passed through a DiT block with Adaptive LayerNorm (Perez et al., 2018). We adopt Linear-Attn and Mix-FFN design from (Xie et al., 2024a) to reduce memory usage. The outputs of both blocks are followed by a zero-initialized linear projection layer, ensuring that at the beginning of training, the learnable canvas tokens have no influence on the DiT’s latent inputs.

We modify the input layer of Wan2.2-5B (Wan et al., 2025) to concatenate reference and condition latents with noisy latents along the channel dimension. The resulting tensor is then passed through the patch embedding layer and combined with MetaCanvas keyframes after interpolation. Light purple tokens represent interpolated keyframe canvas. Note that we do not apply MetaCanvas keyframe latents to reference frames for video tasks.

We aim to validate two questions here:
Q1: Does MetaCanvas really help guide the generation process of diffusion models?
Q2: What connector design is most effective?

We address Q2 with an ablation study on the MetaCanvas connector design in Table 1. We find that conditioning on the timestep enables dynamic control over the influence of canvas tokens on the noisy latents, while the proposed DiT block and accompanying transformer blocks effectively transform and fuse canvas-token information with the latents. Moreover, avoiding early projection of canvas tokens into the low-dimensional VAE space yields additional gains.

We evaluate the fine-tuned image-editing model FLUX.1-Kontext [Dev] (Batifol et al., 2025) augmented with MetaCanvas against competing methods on ImgEdit-Bench (see Table 2) and GEdit-Bench (see Table 3). Equipping FLUX.1-Kontext [Dev] with MetaCanvas yields consistent improvements on both benchmarks.

Figure 6 further contrasts the vanilla model with its MetaCanvas-augmented counterpart under the same training setup, showing steady gains throughout training. Notably, these benefits come from adding only lightweight connector modules, incurring minimal parameter and computational overhead.

In Table 4, we compare videos generated by MetaCanvas with open-source models, including CogVideoX-5B (Yang et al., 2024), HunyuanVideo (Kong et al., 2024), Wan2.1-14B (Wan et al., 2025), and the baseline model Wan2.2-5B (Wan et al., 2025). Our method achieves comparable performance while being additionally equipped with strong video editing capabilities.

We compare MetaCanvas with recent SoTA models, including InsViE (Wu et al., 2025), Ditto (Bai et al., 2025), and Lucy-Edit-Dev (Decart et al., 2025), as well as a control setup of our method that excludes canvas tokens. As shown in Table 5, MetaCanvas achieves comparable video quality scores, as measured by VBench (Huang et al., 2023; Huang et al., 2024) and GPT-4o (OpenAI, 2024), while outperforming all baselines in editing accuracy (i.e., semantics) by a large margin. In addition, we conduct human evaluations comparing Lucy-Edit-Dev v1.1, Ditto, and MetaCanvas, and report the win rates for editing accuracy, spatio-temporal consistency, and overall user preference. MetaCanvas achieves the highest preference rate across all evaluation dimensions. Furthermore, the controlled variant without canvas tokens attains competitive or better performance relative to other baselines, demonstrating the effectiveness of replacing the text encoder with a MLLM-based multimodal condition encoder.

In Table 6, we compare MetaCanvas with Wan-VACE (Jiang et al., 2025) 1.3B/14B on video generation from reference images. MetaCanvas achieves competitive performance with these baselines, particularly on human-object interaction tasks (i.e., character + object under multiple ID categories).