Runze Liu, Haoran He, Jiashun Liu, Zijie Geng, Wenrui Zhou, Yuxiao Ye, Jiale Ding, Ziqi Wang, Wang You, Chen Hu, Ling Pan**†**

Hong Kong University of Science and Technology, StepFun

† Corresponding Author: Ling Pan ([email protected])

Blog: https://ryanliu112.notion.site/sleep-time-evolving

<aside> 💭

Large-scale pre-training and post-training produce capable but mostly static agents. Existing deployment-time improvement mainly focuses on test-time scaling, which spends more compute while the user is waiting and usually does not internalize agents’ capabilities into model parameters. In this work, we propose Sleep-Time Evolving (STE), a new deployment-time learning paradigm for LLM agents. During user-facing time, the agent collects natural interaction trajectories. During sleep-time, STE abstracts these trajectories into user-grounded world models, synthesizes verifiable training environments, and performs Sleep-Time Reinforcement Learning (STRL), leveraging sleep-time compute to improve future user task performance. Across longitudinal user simulations, Sleep-Time Evolving improves next-day task success, improves interaction efficiency, and transfers to related tasks.

</aside>

1. What is sleep time and why do agents need sleep-time evolving?

Large language model (LLM)-based agents have shown great progress in recent years. For example, we can use Codex or Claude Code for complex coding tasks, and OpenClaw for common daily tasks involving tool use and real environment interaction.

The life cycle of LLM agents generally contains two phases: training-time and deployment-time. Training-time improvement mainly includes large-scale pre-training and post-training. After this stage, however, most deployed agents are still largely static. When an agent fails on a user task today, the same failure may still happen tomorrow unless the system is manually updated, the user provides explicit feedback, or more compute is spent again during the next interaction.

For deployment-time improvement, test-time scaling is a common technique. It improves the current response by increasing inference compute, such as sampling more reasoning paths or performing more search. However, test-time scaling significantly increases user latency on complex tasks. More importantly, the high-quality reasoning or responses produced during test time are usually discarded after the session, so the agent does not really internalize the improvement into its own parameters.

Figure 1: LLM agents’ life cycle.

<aside> 💡

So is there any new paradigm that can improve LLM agents with internalization, while not increasing user latency?

</aside>

Besides test-time scaling, sleep-time scaling is another way to use deployment-time compute. Sleep time mainly refers to the time when the user is not interacting with the agent, such as night-time or other user-idle periods. Prior sleep-time compute mainly focuses on pre-computing answers. However, this approach is typically limited to mathematical reasoning tasks and do not generalize well to complex, interactive agent tasks involving tool use and dynamic environments.

In this work, we propose Sleep-Time Evolving (STE). STE is motivated by a simple observation: personal agents have repeated idle periods, but most current agents do not use this time for evolving. During day-time, the user interacts with the agent and the agent collects interaction trajectories. During sleep-time, instead of staying silent, STE analyzes these trajectories, builds a user-grounded world model, imagines and synthesizes verifiable environments, and performs Sleep-Time Reinforcement Learning (STRL). In this way, STE turns the idle periods into opportunities for agent evolving and the user can get a stronger and more personalized agent the next day, and this process can continue over multiple days. The comparison of STE with other deployment-time methods is shown in Table 1.

Table 1: Comparison of different deployment-time improvement paradigms for LLM agents.

Our main contributions are:

1. We introduce Sleep-Time Evolving, a new deployment-time learning paradigm where personal agents improve during user-idle periods rather than only during training time or test time.
2. We propose a closed-loop STE pipeline that transforms fragmented day-time trajectories into user-grounded world models and verifiable imagined environments, then performs targeted sleep-time RL.