PAPER_TITLE

FIRST_AUTHOR_LAST, FIRST_AUTHOR_FIRST; SECOND_AUTHOR_LAST, SECOND_AUTHOR_FIRST

SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks

Jialiang Li, Wenzhen Wu, Gaojing Zhang, Yifan Han, Wenzhao Lian

HIROL Lab, School of Artificial Intelligence, Shanghai Jiao Tong University
ICRA 2026

arXiv Code

Abstract

Successfully solving long-horizon manipulation tasks remains a fundamental challenge. These tasks involve extended action sequences and complex object interactions, presenting a critical gap between high-level symbolic planning and low-level continuous control. To bridge this gap, two essential capabilities are required: robust long-horizon task planning and effective goal-conditioned manipulation. Existing task planning methods, including traditional and LLM-based approaches, often exhibit limited generalization or sparse semantic reasoning. Meanwhile, image-conditioned control methods struggle to adapt to unseen tasks. To tackle these problems, we propose SAGE, a novel framework for Scene Graph-Aware Guidance and Execution in Long-Horizon Manipulation Tasks. SAGE utilizes semantic scene graphs as a structural representation for scene states. A structural scene graph enables bridging task-level semantic reasoning and pixel-level visuo-motor control. This also facilitates the controllable synthesis of accurate, novel sub-goal images. SAGE consists of two key components: (1) a scene graph-based task planner that uses VLMs and LLMs to parse the environment and reason about physically-grounded scene state transition sequences, and (2) a decoupled structural image editing pipeline that controllably converts each target sub-goal graph into a corresponding image through image inpainting and composition. Extensive experiments have demonstrated that SAGE achieves state-of-the-art performance on distinct long-horizon tasks.

Method

We introduce a SAGE, a Scene Graph-Aware Guidance and Execution framework for long-horizon manipulation tasks. SAGE leverages LLMs for planning and generated images for goal guidance, while mitigating their limitations. At its core, SAGE uses semantic scene graphs as a structural representation of the scene state, effectively aligning task-level semantic reasoning with pixel-level visuo-motor control. A scene graph models the physical world by representing objects as nodes and their spatial relationships (e.g., On, In) as edges. SAGE consists of two key components: (1) a scene graph task planner that uses a VLM to parse the scene into a scene graph and a LLM to reason about a scene graph transition chain, which serves as a physically-grounded task plan; and (2) a decoupled structural image editing pipeline that controllably converts each scene graph of this plan into a corresponding image through image inpainting and composition. These generated images guide a visuo-motor policy to accomplish the entire long-horizon manipulation task by executing each sub-task iteratively.

The transitions between scene graphs could be viewed as object movements. We thus design a direct and effective way to realize these movements, i.e., first erasing the object from its original location and then inserting it into a new one. To this end, we propose a strutural image editing pipeline decoupled into three stages: layout prediction, image inpainting and image composition.

Experiments

We compare our method againts two categories of baselines: (1) End-to-End Execution, and (2) Sub-Goal Image Generation.

Qualitative comparison with other image generation methods. (a) Current observation. (b) Ground-truth next sub-goal image. (c) SuSIE (d) GR-MG. (e) Break-A-Scene. (f) SGEdit. (g) Our decoupled structural image editing pipeline.

Demo

More Visualization

We offer examples of generated scene graph transition chains and the corresponding edited sub-goal images.

BibTeX

@misc{li2025sagescenegraphawareguidance,
      title={SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks}, 
      author={Jialiang Li and Wenzheng Wu and Gaojing Zhang and Yifan Han and Wenzhao Lian},
      year={2025},
      eprint={2509.21928},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2509.21928}, 
}