Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos

This is the official code repository for the EMNLP 2026 paper Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos.

Project page: https://github.com/HKUST-KnowComp/PACE

1. Framework Pipeline

The framework consists of two stages:

1.1 Question-Conditioned Evidence Indexing

The indexing stage builds an evidence database from question-derived factors. Key files:

• Factor extraction: ./pace/frame_caption_query.py (extract_priors_from_question)
• Clip-level captioning: ./pace/frame_caption_query.py (process_video)
• Database construction: ./pace/build_database_query.py (init_single_video_db)

The video is uniformly partitioned into clips. For each clip, a vision-language model produces a textual record that explicitly annotates which question-derived factors are observed.

1.2 Option-Aware Verification

The verification stage uses candidate answers to derive contrastive factors and queries the index. Key files:

• Core agent: ./pace/pace_query.py (PACECoreAgent)
• Factor-guided agent: ./pace/pace_factor.py (PACECoreAgent with additional tools)
• Tool definitions: ./pace/build_database_query.py (evidence_find_tool, frame_inspect_tool, clip_search_tool, global_browse_tool)
• Factor decomposition: ./pace/pace_factor.py (find_factor)

The agent follows a THINK → ACT → OBSERVE loop using the following tools:

• global_browse_tool: Get global overview of events and main subjects
• find_factor: Analyze options to identify decisive factors
• clip_search_tool: Search video clips by event description
• frame_inspect_tool: Inspect frames in specific time ranges
• evidence_find_tool: Retrieve pre-analyzed factor evidence
• finish: Commit to final answer

2. Required Packages

Required packages are listed in requirements.txt. Install them by running:

pip install -r requirements.txt

Copy .env.example to .env, fill in only the services you use, and load the variables in your shell before running PACE. Credentials are never required in tracked source files. Do not commit .env.

3. Running the Code

After installing dependencies, you can run the PACE agent on a video with a specific question using the provided local_run.py script.

3.1 Basic Command

python local_run.py <video_url_or_path> "<question>" <log_file.json>

Arguments:

Argument	Description
video_url	Path to the local video file (e.g., /path/to/video.mp4) or a YouTube URL (if download is enabled).
question	The question you want to ask about the video. Enclose in quotes if it contains spaces.
log_file	The output JSON file name (under LOG_FOLDER) where the agent's reasoning history will be saved.

3.2 Optional Flags

Flag	Type	Default	Description
--port	int	32001	Port for the vLLM serve (used for LLM inference).
--caption_only	flag	False	If set, the script performs only video captioning (indexing) and exits without running the agent.
--run_name	str	'default'	Subdirectory name under VIDEO_DATABASE_FOLDER for storing cached video data (frames, captions, database).
--num_factors	float	None	Number of factors to extract per question (if not set, uses the default from config).

4. Downstream Evaluation

We evaluate PACE primarily on MMR-V, a benchmark curated to assess multimodal deep reasoning in videos. Consistent gains are also demonstrated on LVBench, VideoMME, EgoSchema, and LongVideoBench. For further information, please refer to the paper.

5. Data Download

Should you want to download the benchmark data, please refer to the respective dataset repositories:

• MMR-V
• LVBench
• VideoMME
• EgoSchema
• LongVideoBench

6. Citing this Work

Please use the bibtex below for citing our work.

@inproceedings{xu2026pace,
  title={Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos},
  author={Xu, Baixuan and Xu, Yinyui and Zheng, Tianshi and Wang, Zhaowei and Wang, Weiqi and Shi, Haochen and Liu, Jiayu and Zong, Qing and Ren, Xiyu and Geng, Xinyu and He, Zhitao and Song, Yangqiu},
  booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year={2026}
}