The Cognitive Bandwidth Bottleneck

This repository contains the compact implementation used for the paper's Planning with Actions (PwA) and Planning with Schemas (PwS) experiments. The release covers TextCraft, WebShop, ALFWorld, and SciWorld, including the schema-variation and ALFWorld distractor stress-test conditions.

Project page: https://github.com/HKUST-KnowComp/Cognitive-Bandwidth

What is included

• The AgentGym controller changes for grounded actions, schemas, schema variation, and distractor injection.
• Minimal source wrappers and launch instructions for all four environments.
• Reproducible trajectory metrics for invalid/repetitive actions and context token growth.
• Configuration templates with no embedded credentials or machine paths.

Raw trajectories, result archives, plots, private experiment notes, model checkpoints, WebShop crawled data, and the later Cognitive Inertia project are intentionally excluded.

Installation

Create an environment with Python 3.10 or newer, then install the controller:

python -m pip install -e .
python -m pip install -e agentenv

Install each environment in its own environment as required by its upstream benchmark. The four wrapper directories contain the corresponding setup files and launch entry points. Benchmark datasets should be downloaded from their official sources; they are not redistributed here.

Copy .env.example to .env or export the variables in your shell. The API key is read only at runtime:

export OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
export OPENROUTER_API_KEY=YOUR_KEY

Do not commit .env.

Running an evaluation

Start one environment server and pass its local URL to the runner. The task indices should match the selected benchmark split; the default is the first 100 indices for a smoke run.

python run_experiment.py \
  --environment alfworld \
  --paradigm pws \
  --model moonshotai/kimi-k2 \
  --server-url http://127.0.0.1:36001 \
  --indices 2420-2519 \
  --output outputs/alfworld_pws.json

The accepted paradigms are pwa, pws, schema-variation, and distractor. The latter two are intended for ALFWorld/SciWorld and ALFWorld, respectively. scripts/run_*.sh provide equivalent small wrappers.

Analysis

analysis/token_stats.py computes cumulative GPT-4o-compatible token counts from saved conversations. analysis/action_metrics.py counts environment rejections and can optionally call the configured judge model for repetitive action labels. Both scripts accept explicit input/output paths.

Attribution

The environment clients are adapted from AgentGym and the benchmark projects. See LICENSE and THIRD_PARTY.md for attribution and redistribution notes.

Citation

@inproceedings{xu2026cognitive,
  title={The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agents from Planning with Actions to Planning with Schemas},
  author={Xu, Baixuan and Zheng, Tianshi and Wang, Zhaowei and Tsang, Hong Ting and Wang, Weiqi and Fang, Tianqing and Song, Yangqiu},
  booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year={2026}
}