Semantic Guided Autoregressive Diffusion Based Data Augmentation Using Visual Instructions

Abstract

Recent breakthroughs in generative image models, especially those based on diffusion techniques, have radically transformed the landscape of text-guided image synthesis by delivering exceptional fidelity and detailed semantic control. In this study, we present an iterative editing framework that harnesses the inherent strengths of these generative models to progressively refine images with precision. Our approach begins by generating diverse textual descriptions from an initial image, from which the most effective prompt is selected to drive further refinement through a fine-tuned Stable Diffusion process. This pipeline, as detailed in our flow diagram, orchestrates a series of controlled image modifications that preserve the original context while accommodating deliberate stylistic and semantic adjustments. By cycling the augmented output back into the system, our method achieves a harmonious balance between innovation and consistency, paving the way for highquality, context-aware visual transformations. This dynamic, auto-regressive strategy underscores the transformative potential of modern image generation models for applications that require detailed, controlled creative expression. The code is available on Github. © 2025 Elsevier B.V., All rights reserved.