[📃 Paper] ｜ [🤗 Dataset] ｜ [💻 Code]

Features

• 🔁 Static + Dynamic referral
• 🏥 Real-hospital settings
• 📈 Information gain

Overview

• Motivation: Outpatient referral is often simplified as static department classification, but real clinical referral is an interactive process under incomplete information.
• Key Challenge: Patients may describe symptoms vaguely or gradually, so a useful referral system must ask discriminative follow-up questions before making a reliable decision.
• IOR-Static: Evaluates whether LLMs can select the right department from fixed patient information.
• IOR-Dynamic: Evaluates whether LLMs can reduce uncertainty through multi-turn questioning and improve referral decisions over time.

Figure 1. Overview of IOR-Bench. The benchmark evaluates outpatient referral under both static and dynamic decision-making settings.

Benchmark Settings

Setting	Input	Model Behavior	Evaluation Target
IOR-Static	Complete patient case	Predict the target department directly	Final referral accuracy
IOR-Dynamic	Partial patient information	Ask follow-up questions and update referral decisions	Turn-level and final referral performance

Repository Structure

IOR-Bench/
├── assets/                     # Figures and project images
├── data/                       # Benchmark data
├── scripts/                    # Inference and evaluation scripts
│   ├── static_evaluation.sh
│   ├── dynamic_evaluation.sh
│   ├── entropy_evaluation.sh
│   ├── static_calculation.sh
│   ├── dynamic_calculation.sh
│   └── entropy_calculation.sh
├── src/                        # Core implementation
│   ├── evaluation/
│   │   ├── static.py
│   │   ├── dynamic.py
│   │   └── entropy_metric.py
│   ├── post_processing/
│       ├── static_calc.py
│       ├── dynamic_calc.py
│       └── entropy_calc.py
│   └── bert-sft/
│       └── bert-training.py
├── environment.yml             # Conda environment
└── README.md

Quick Start

1 · Install environment

conda env create -f environment.yml
conda activate ior-bench

2 · Run inference

Run IOR-Static inference:

sh ./scripts/static_evaluation.sh

Run IOR-Dynamic inference:

sh ./scripts/dynamic_evaluation.sh

Run entropy-based dynamic inference:

sh ./scripts/entropy_evaluation.sh

3 · Train BERT baseline

Train a bert-base-chinese classifier for the static referral task:

python src/bert-sft/bert-training.py

The script expects a prepared train.csv with input and lable columns, where input is the dialogue text and lable is the target department.

4 · Compute metrics

Compute IOR-Static metrics:

sh ./scripts/static_calculation.sh

Compute IOR-Dynamic metrics:

sh ./scripts/dynamic_calculation.sh

Compute entropy and information-gain metrics:

sh ./scripts/entropy_calculation.sh

License

This project is released under the MIT License. See LICENSE for details.

Contact

For questions or collaborations, please open an issue on GitHub.