Teaching Image and Video Models What "Better" Means: DiffusionART

Image and video generators are first trained to imitate a huge collection of pictures and clips. That teaches them to make plausible output, but not necessarily good output: sharp, well lit, with natural motion, and matching the prompt. To close that gap, researchers add a second stage that uses reinforcement learning (RL): the model generates, a scoring model rates the result, and the generator is nudged toward higher scores.

A paper posted to arXiv on 8 October 2026, DiffusionART (arXiv 2610.12572, from the HPC-AI Lab at the National University of Singapore, Kling AI and HKUST), is about doing that nudging faster and more reliably. It was among the most-upvoted papers on Hugging Face's daily list on 12 October.

Background: how diffusion models generate, and why scores are hard to use

A diffusion model starts from random noise and removes it over a series of steps until a picture or video appears. (The paper uses "diffusion" loosely and works with a close relative called rectified flow.) We covered the text version of this idea in our article on adaptively looped diffusion language models.

The score from a reward model arrives only after the last step. But the model made many decisions on the way. Deciding which of those steps deserves credit for a good score is the credit-assignment problem, and it is the difficulty this paper targets. The reward can be a "black box": any scorer will do, even one that cannot be differentiated, such as a quality or motion rater or a human-preference model.

Two existing families, each with a weakness

• Reverse-process methods such as Flow-GRPO train on the denoising steps the model actually took, using a group-relative reward (how a sample scored compared with others for the same prompt). The authors argue credit assignment is poor, because each step is updated along its own random noise and weighted by a score that the step only partly caused. This makes training slow.
• Forward-process methods such as DiffusionNFT add noise to the finished samples and train the model to predict the direction back, weighting by reward. They are efficient, but the authors say they are unstable unless two tricks are used: training on both good and bad samples together, and an exponential moving average (EMA), a slowly updated copy of the model used to generate samples. How slowly the copy updates has to be tuned for every task.

The authors' goal is a method that is both efficient and stable without per-task tuning.

What the authors found: a hidden force

The first half of the paper is analysis. The authors show that the forward-process objective quietly contains an extra pull they call a forward-reverse consistency force. It is not about getting higher rewards; it comes from a mismatch between the direction the model really samples in and the direction the training target assumes. Training on good and bad samples together cancels this force only partly, and what remains can destabilise learning. They also argue that the EMA copy helps because it acts as a restoring force that pulls the model back toward where it was, not because of the off-policy samples it generates.

The method: aggregate first, then restore

DiffusionART (short for Diffusion Reinforcement with Aggregation-First Reward Tilting) has two main ideas:

1. Aggregate each prompt's feedback into one direction. For one prompt, the model generates a group of samples and gets a reward for each. Instead of sending each sample's score back along its own path, the method first computes how the rewards relate to the final images across the whole group (a covariance) and normalises it into a single "reward-improving direction". That one direction is then used as the training target for every step the model took. Because the target never uses the forward direction, the hidden consistency force is removed by construction.
2. Restore automatically instead of tuning an EMA. Each batch is split in two and the model is updated once on each half. The second update adds a pull back toward the model's starting point that grows with how far a step has already drifted. A trust-region mask, inspired by the clipping used in the well-known PPO algorithm, ignores updates that would push a step further away while keeping those that pull it back. No EMA schedule has to be set.

The method does not need a differentiable reward model. The authors also describe a variant, DiffusionART-Pairwise, that learns from "which of these two is better" judgments instead of numeric scores.

What the results say

The authors test on image models (Stable Diffusion 3.5 Medium and the 12-billion-parameter FLUX.1-dev) and on video models (HunyuanVideo-13B for text-to-video and Wan2.1-I2V-14B for image-to-video), against Flow-GRPO, Flow-GRPO-Fast, DanceGRPO, Flow-DPPO and DiffusionNFT, with and without EMA. Headline numbers from the paper:

• On HunyuanVideo-13B with equal weights on visual and motion quality, DiffusionART reached the late-stage average reward of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, and with 28.8% and 53.0% higher average training rewards.
• After 400 training steps, the authors report evaluation scores of 8.970 for visual quality and 4.224 for motion quality, 31.4% and 57.4% above the strongest baseline.
• For image-to-video, it reached a motion-quality score of 6 in 45.8 GPU hours, against 220.3 for Flow-GRPO-Fast.
• With pairwise feedback, using one pair of videos per prompt gave a final visual-quality score of 12.235 in 260.8 GPU hours, compared with 12.092 in 883.3 GPU hours for a larger group.
• In ablations, removing the second update or the trust-region mask made training less stable, with dips in the reward curve.

Caveats worth keeping in mind

• The scores are from reward models. The evaluations use learned scorers (such as VideoReward, GenEval and PickScore), the same kind of signal the models are trained to maximise. In the main text we did not find a study with human raters, so a higher score does not by itself show that people prefer the output.
• The comparisons were run by the authors. By their own account, DiffusionNFT used its original per-task settings, and on FLUX.1-dev those were not retuned, which the authors say explains its unstable training there. A baseline tuned for each task might close part of the gap.
• "Fewer GPU hours" means to reach a given level. The savings depend on the target score chosen and on how the baselines were run.
• It is a preprint. It has not been peer reviewed, and the results come from a small number of models and reward settings.

Why it matters if you are learning AI

Reinforcement learning is no longer only for game-playing and chatbots. Fine-tuning image and video generators with scores is now a standard stage, and much of the progress is about the same hard questions: who gets the credit for a good result, and how to keep training stable. The paper is a good example of how researchers diagnose an unstable method before fixing it, and of ideas that carry over, such as group-relative rewards, a trust region and a restoring force. If you are new to the underlying learning ideas, our article on sigmoid, softmax and logistic regression covers the basics, and our look at Learn2Play Bench shows another case of an AI system learning from feedback.

What to watch next

• Independent reproductions on other models, and comparisons against retuned baselines.
• Human-preference studies that check whether the reward gains match what people see.
• Whether the code released on GitHub is adopted in open image and video training pipelines.

Source: DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models, arXiv 2610.12572, 8 October 2026 (code: github.com/vanzll/DiffusionART). We read the paper's full text. Hugging Face's daily papers list was used to pick the paper.