LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

Renshan Zhang¹, Haoyang Meng², Yixiao He², Rui Shao^{1 3}†, April Hua Liu⁴, Liqiang Nie^{1 3}†

¹Harbin Institute of Technology, Shenzhen
²Beijing University of Post and Telecommunications
³Shenzhen Loop Area Institute
⁴Shanghai University of Finance and Economics
†Corresponding author

✨If you find this work useful for your research, please kindly cite our paper and star our repo.✨

Updates

• [08/2026] arXiv paper released.

Introduction

This is the GitHub repository of LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection. We introduce LookAgain, a novel closed-loop grounding paradigm that enables MLLM-based GUI agents to iteratively predict, inspect, and refine coordinates through visually grounded reflection. By combining reflective trajectory SFT with GRPO optimization, LookAgain achieves robust GUI grounding performance on various challenging scenarios.

Installation

1. Clone this repository and navigate to the folder

git clone lookagain.git
cd lookagain

2. Install the environment

bash scripts/install_uv_env.sh

Evaluation

Download Benchmarks

Download the benchmark data from:

osworld-g:       https://huggingface.co/datasets/MMInstruction/OSWorld-G
screenspot-pro:  https://huggingface.co/datasets/likaixin/ScreenSpot-Pro
ui-vision:       https://huggingface.co/datasets/ServiceNow/ui-vision
mmbench-gui:     https://huggingface.co/datasets/OpenGVLab/MMBench-GUI
venus-bench-gd:  https://huggingface.co/datasets/inclusionAI/VenusBench-GD

Deploy Model

MODEL_PATH=/path/to/lookagain-8b bash scripts/start_server.sh

OSWorld-G

# convert data format (for first run)
DATA_PATH=/path/to/osg bash scripts/osg/prepare_osworldg_inputs.sh
# run evaluation
bash scripts/osg/run_osworldg_eval.sh

UI-Vision

# convert data format (for first run)
DATA_PATH=/path/to/uivision bash scripts/ui_vision/prepare_ui_vision_inputs.sh
# run evaluation
bash scripts/ui_vision/run_ui_vision_eval.sh

Screenspot-Pro

# convert data format (for first run)
DATA_PATH=/path/to/ssp bash scripts/ssp/prepare_screenspot_pro_inputs.sh
# run evaluation
bash scripts/ssp/run_screenspot_pro_eval.sh

MMBench-GUI-L2

# convert data format (for first run)
DATA_PATH=/path/to/mmb bash scripts/mmbench/prepare_mmbench_gui_inputs.sh
# run evaluation
bash scripts/mmbench/run_mmbench_gui_eval.sh

VenusBench-GD

# convert data format (for first run)
DATA_PATH=/path/to/venus bash scripts/venus_bench_gd/prepare_venus_inputs.sh
DATA_PATH=/path/to/venus bash scripts/venus_bench_gd/prepare_venus_refusal_inputs.sh
# run evaluation
bash scripts/venus_bench_gd/run_venus_bench_gd_eval.sh
bash scripts/venus_bench_gd/run_venus_refusal_eval.sh

Citation

If you find this work useful for your research, please kindly cite our paper:

@misc{zhang2026lookagain,
      title={LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection}, 
      author={Renshan Zhang and Haoyang Meng and Yixiao He and Rui Shao and April Hua Liu and Liqiang Nie},
      year={2026},
      eprint={2608.09723},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.09723}, 
}