Bei Liu

AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization’s purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal–assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.

Long-document understanding requires models to find and combine evidence across many pages, layouts, tables, figures, and charts. Existing retrieval-augmented systems usually select evidence from a static index before generation, while recent agentic systems add multi-turn tool use but often rely on frozen proprietary backbones whose behavior is set by prompts. We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents. With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7%, compared with a 54.4% direct-input baseline, showing that mutable document-harness design can improve compact document agents by a large margin.

Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high – yet most existing benchmarks still measure short-context or single-page QA. We introduce XL-DocBench, a fully human-verified benchmark for extra-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6%) use multiple evidence pages. The final set also includes 556 questions (36.6%) that use tables, charts, or figures, and 165 questions (10.9%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 None-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.

Robotic motor control necessitates the ability to predict the dynamics of interactions with environment and objects. However, existing self-supervised pre-trained visual representations in robotic motor control, leveraging large-scale egocentric videos, often focus solely on learning the static content features. Thus it neglects the crucial motion clues in human videos, which contain key knowledge about interaction with the environments and objects. In this paper, we present a simple yet effective visual pre-training framework termed as STP for robotic motor control, which jointly performs spatial and temporal prediction with different masking ratios and a dual-decoder design by utilizing large-scale egocentric video data. Specifically, given paired frames in a video clip, we first employ a spatial decoder to perform spatial prediction on the masked current frame to learn the content features. Second, we employ a separate temporal decoder to conduct temporal prediction, conditioned on the masked current frame and an extremely highly-masked future frame, for capturing motion features. This asymmetric masking and decoupled decoding design differentiate our STP from the traditional image or video masked autoencoders by explicitly ensuring the learned image representation to focus more on motion information while still capture spatial details . Due to its lightweight temporal decoder and highly masked future frame, STP incurs manageable extra training overhead over standard image MAE. Extensive simulation and real-world experiments demonstrate the effectiveness and generalization abilities of STP, especially in generalizing to unseen environments with more distractors. Additionally, we extend this framework with post-training and hybrid pre-training to demonstrate the generality and data efficiency of STP. The source code and models will be available at https://github.com/MCG-NJU/STP.

Improving the generalization capabilities of general-purpose robotic manipulation in real world has long been a significant challenge. Existing approaches often rely on collecting large-scale robotic data, which is costly and time-consuming. However, due to insufficient diversity of data, they typically suffer from limiting their capability in open-domain scenarios with new objects and diverse environments. In this paper, we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models, to condition robot manipulation tasks. By integrating the mask modality, which incorporates semantic, geometric, and temporal correlation priors derived from vision foundation models, into the end-to-end policy model, our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning, including new object instances, semantic categories, and unseen backgrounds. We first introduce a series of foundation models to ground natural language demands across multiple tasks. Secondly, we develop a two-stream 2D policy model based on imitation learning, which processes raw images and object masks to predict robot actions with a local-global perception manner. Extensive real-world experiments conducted on a Franka Emika robot and a low-cost dual-arm robot demonstrate the effectiveness of our proposed paradigm and policy.

We propose RoLD, a novel approach that decouples robot action trajectory encoding and control policy generation by leveraging latent action trajectory spaces. We pre-train a task-agnostic auto-encoder to project action trajectories into a latent space, then learn a diffusion model to generate actions. Experiments demonstrate 7%–29% improvement over baselines across eight tasks.

We propose CLIP-ViP, an Omnisource Cross-modal Learning method equipped with a Video Proxy mechanism, adapting the image-text pre-trained CLIP model for video-language post-pretraining. Our approach significantly improves CLIP’s performance on video-text retrieval and achieves state-of-the-art results on MSR-VTT, DiDeMo, LSMDC, and ActivityNet.

Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on short-form videos (i.e., within 30 seconds) and sentences, leaving long-form video-language pre-training rarely explored. Directly learning representation from long-form videos and language may benefit many long-form video-language understanding tasks. However, it is challenging due to the difficulty of modeling long-range relationships and the heavy computational burden caused by more frames. In this paper, we introduce a Long-Form VIdeo-LAnguage pre-training model (LF-VILA) and train it on a large-scale long-form video and paragraph dataset constructed from an existing public dataset. To effectively capture the rich temporal dynamics and to better align video and language in an efficient end-to-end manner, we introduce two novel designs in our LF-VILA model. We first propose a Multimodal Temporal Contrastive (MTC) loss to learn the temporal relation across different modalities by encouraging fine-grained alignment between long-form videos and paragraphs. Second, we propose a Hierarchical Temporal Window Attention (HTWA) mechanism to effectively capture long-range dependency while reducing computational cost in Transformer. We fine-tune the pre-trained LF-VILA model on seven downstream long-form video-language understanding tasks of paragraph-to-video retrieval and long-form video question-answering, and achieve new state-of-the-art performances.

We propose HD-VILA, a High-resolution and Diversified VIdeo-LAnguage pre-training model. We collect 371.5k hours of 720p videos across 15 YouTube categories. HD-VILA achieves new state-of-the-art on 10 VL understanding tasks, improving SOTA by 40.4% R@1 on zero-shot MSR-VTT text-to-video retrieval.

We propose SOHO to "See Out of tHe bOx", an end-to-end vision-language pre-training framework that takes a whole image as input without requiring bounding box annotations, enabling 10x faster inference than region-based approaches. SOHO uses a dynamic visual dictionary for cross-modal understanding.

We propose a fully Transformer visual embedding for Vision-Language Pre-training (VLP), introducing a metric named Inter-Modality Flow (IMF) to measure vision-language interaction, and a Masked Feature Regression (MFR) mechanism to promote inter-modality learning.

We propose ViCo with three novel designs for generating engaging video comments. We use the number of "likes" as a proxy for human preference, train a reward model to align with this proxy, and optimize a comment generator via reward feedback. We collect a large video comment dataset ViCo-20k to facilitate research.

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.

We propose a multi-adversarial training framework via policy gradient to generate poems from images, jointly learning cross-modal relevance and poetic language style. We introduce a deep coupled visual-poetic embedding and two discriminative networks, and release two poem datasets including the first human-annotated image-poem pair dataset.