inclusionAI/AReaL

AReaL: A Large-Scale Asynchronous Reinforcement Learning System

| Paper | Documentation | 中文文档 | Ask DeepWiki | 🤗 Models & Data | WeChat (微信) Group |

AReaL is an open-source fully asynchronous reinforcement learning training system for large reasoning and agentic models, developed by members from Tsinghua IIIS and the AReaL Team at Ant Group. Built upon the open-source project ReaLHF, we are fully committed to open-source principles by providing the training details, data, and infrastructure required to reproduce our results, along with the models themselves. AReaL aims to help everyone build their own AI agents easily and affordably. Our team loves milk tea because it’s delicious, customizable, and affordable—we hope you enjoy our project just as much as you’d enjoy real milk tea. Cheers!

AReaL Highlights

• ⚡ Flexibility: Seamless customization for agentic RL and online RL training by simply replacing the base_url.
• 📈 Scalability: Stable fully asynchronous RL training with industry-leading speed.
• ✨ Cutting-Edge Performance: State-of-the-art math, coding, search, and customer service agents.

📰 News

\[2026/03/02\] We provide a complete example to train your own 🦞 OpenClaw agent by simply replacing the base_url and api_key with AReaL’s RL service - no complicated dependencies, no code changes, works with any agentic runtime!

\[2026/02/06\] We are delighted to introduce AReaL-SEA, a self-evolving data synthesis engine. Combined with RL training on AReaL, the 235B MoE model surpasses GPT 5 and achieves comparable performance with Gemini 3.0 Pro on $\tau^2$-bench! Check out the paper, model, data, and code.

\[2026/01/15\] Congrats to our friends at CAMEL-AI for open-sourcing SETA, their terminal agent RL project trained with AReaL! Check out their training workflow and the announcement on X.

🚀 Getting Started

First, install the package:

1
2
3
4

git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install uv
uv sync --extra cuda

Our training scripts automatically download the required dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). To run on a single node:

1

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

To run on a Ray cluster with 2 nodes and 8 GPUs per node (remember to update paths in the YAML file to point to your shared storage):

1
2
3

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  scheduler.type=ray

For comprehensive setup instructions, see our quickstart guide.

📚 Examples

Math & Reasoning

Agentic RL

Vision-Language Models

Alignment & Infrastructure

🔧 Support Matrix

🧠 Algorithms

All RL algorithms support both asynchronous and synchronous versions by setting max_head_offpolicyness=0. See Asynchronous RL Guide.

Models

Check the AI Coding Assistant Guide and Archon Reference for how to integrate new models into AReaL.

Training Backends

Inference Backends

📖 Resources

Tutorial

• Installation
• Quickstart
• Agentic RL
• Evaluation
• Large MoE with Megatron
• Large MoE with PyTorch Archon

Code Walkthrough

• Running GRPO on GSM8K dataset

Best Practices

• Improving Algorithm Performance
• Agent Workflow Best Practices
• Debugging
• Handling OOM Issues
• Performance Profiling

Customization

• Customize Dataset
• Customize Agentic/RVLR Rollout Workflows

Algorithms

• Asynchronous RL Explained
• PPO, GRPO, and Related Algorithms
• M2PO

Reference

• CLI Configurations
• Checkpointing
• Metrics Tracking
• Allocation Mode
• Rollout Workflow
• Agent Workflow
• AI-Assisted Development

🤝 Contributing

We warmly welcome contributions from the community! Whether you’re fixing bugs, adding features, improving documentation, or helping others, your contribution is valued. Please check our Contributing Guide for detailed information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Fork and clone the repository
git clone https://github.com/YOUR-USERNAME/AReaL
cd AReaL

# Install uv and sync dependencies
pip install uv
# Use `--extra cuda` on Linux with CUDA for full functionality
uv sync --extra cuda --group dev
# Or without CUDA support
# uv sync --group dev

# Set up pre-commit hooks (formatting, linting, commit message checks)
pre-commit install --install-hooks

# Make changes
git checkout -b feat/gpt-o5
git add .
# `git commit` will automatically check your files and commit messages
git commit -m "feat: implement gpt-o5 training loop"
git push

🗺️ Future Roadmap

• Full Roadmap
• 2025 Q4 Roadmap

AReaL is under active development with planned minor releases weekly and major releases monthly. We warmly welcome community engagement and contributions. We are also actively hiring interns and full-time employees with open positions in both the US and China.

🙏 Acknowledgments

We gratefully acknowledge that major contributors are from the AReaL Team at the Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University and Ant Group.

We have also received invaluable assistance from the following groups (listed alphabetically):

• The Data Intelligence Lab at Ant Research for their data support

• @HwVanICI for support on vLLM, LoRA, NPU integration, and more

• The Relaxed System Lab at HKUST for seamless collaboration on numerous system-related aspects

• The SGLang team for supporting custom weight update features and their contributions during AReaL-lite development

• The Super Computing Technology (SCT) team at Ant Group for their expertise in large-scale cluster operations and maintenance

• Special thanks to @Lyken17 for providing valuable suggestions throughout the API design process

We also deeply appreciate all pioneering work from the community, particularly the ReaLHF project from OpenPsi Inc. and other outstanding projects, including but not limited to DeepScaleR, Open-Reasoner-Zero, OpenRLHF, VeRL, SGLang, QwQ, Light-R1, and DAPO.

📄 Citation

1
2
3
4
5
6
7
8

@inproceedings{mei2025real,
  author       = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
  title        = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
  booktitle    = {Proceedings of the Eighth Conference on Machine Learning and Systems,
                  MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
  publisher    = {mlsys.org},
  year         = {2025},
}

1
2
3
4
5
6
7
8
9

@misc{fu2025areal,
      title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},
      author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},
      year={2025},
      eprint={2505.24298},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.24298},
}