Yiheng Xu

I develop scalable methods to build autonomous agents that interact with digital environments as naturally as humans, advancing AI from task-specific automation to robust autonomy. My work focuses on advancing AI to:

• Understand and process human-created digital content, from unstructured, visually rich documents (e.g., LayoutLM, DiT, DocBank) to structured web pages and interfaces (e.g., MarkupLM).
• Autonomously perform complex tasks across diverse digital interfaces, including programming-language interactions (e.g., Lemur) and multimodal GUI control on computers and mobile devices (e.g., Aguvis, AgentTrek).

Selected Publications

• Qwen2.5-VL Technical Report
  Core Contributor (Responsible for Computer/Mobile Using Agent Capability)
  [PDF] [Code]

• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
  Yiheng Xu*, Zekun Wang*, Junli Wang*, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong
  [PDF] [Project]

• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
  Yiheng Xu*, Dunjie Lu*, Zhennan Shen*, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
  ICLR 2025 Spotlight (Top 5%)
  [PDF] [Project]

• Lemur: Harmonizing Natural Language and Code for Language Agents
  Yiheng Xu*, Hongjin Su*, Chen Xing*, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, Tao Yu
  ICLR 2024 Spotlight (Top 5%)
  [PDF] [Code] [Model] [Blog]

• DiT: Self-supervised Pre-training for Document Image Transformer
  Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
  ACM Multimedia 2022
  [PDF] [Code] [Demo]

• MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
  Junlong Li*, Yiheng Xu*, Lei Cui, Furu Wei
  ACL 2022
  [PDF] [Code] [Model] [Blog]

• XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
  Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
  ACL 2022 Findings
  [PDF] [Data]

• LayoutReader: Pre-training of Text and Layout for Reading Order Detection
  Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
  EMNLP 2021
  [PDF] [Code] [Blog]

• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
  Yang Xu*, Yiheng Xu*, Tengchao Lv*, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
  ACL 2021
  [PDF] [Code] [Model] [Blog]

• DocBank: A Benchmark Dataset for Document Layout Analysis
  Minghao Li*, Yiheng Xu*, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
  COLING 2020
  [PDF] [Code] [Blog]

• LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  Yiheng Xu*, Minghao Li*, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
  KDD 2020
  [PDF] [Code] [Model] [Blog] [PaperDigest Most Influential Papers] [ICBS 2024 Frontiers of Science Award]

Service

• Conference Reviewer: AAAI, ACL, COLING, EMNLP, ICLR, ICML, MM
• Journal Reviewer: IEEE Transactions on Multimedia, Neurocomputing, IJDAR

Undergraduate Collaborators

I enjoy working with passionate undergraduates who have strong coding/math skills and bring fresh inspiration. If you’re interested in collaborating, please feel free to email me.

Here are the undergraduate collaborators I’ve been fortunate to work with:

• Junlong Li: SJTU -> HKUST (MarkupLM; DiT)
• Junli Wang: THU (AGUVIS; AgentTrek)
• Dunjie Lu: SYSU -> HKU (AgentTrek; AGUVIS)
• Zhennan Shen: SJTU (AgentTrek)
• Haoyuan Wu: HKU
• Yitao Liu: HKU (Lemur)
• Boyu Mi: ZJU (Lemur)

Powered by Jekyll and Minimal Light theme.