Skip to content

WebArena Verified #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
NicolasAG wants to merge 65 commits into
base: main
Choose a base branch
from
Open

WebArena Verified #377

NicolasAG wants to merge 65 commits into from

Conversation

@NicolasAG
Copy link
Collaborator

@NicolasAG NicolasAG commented Nov 6, 2025 •
edited
Loading

Introduce the webarena_verified benchmark.

  • tasks are registered with this template: webarena_verified.{intent_template_id}.{task_id}
  • new WebArenaVerifiedTask class overrides the setup() function of GenericWebArenaTask to:
    • use the webarena_verified evaluator
    • load extra html headers if PW_EXTRA_HEADERS is set -- used to store secret keys to access self-hosted webarena instances
    • append a hint to the goal for the model to return the expected response format for the webarena_verified evaluator
  • new WebArenaVerifiedEvaluator class that calls the webarena_verified.api.WebArenaVerifiedEvaluator from platform-labs-webarena-verified

NicolasAG and others added 30 commits September 16, 2025 20:03
@NicolasAG
init commit for webarena verified
e55db81
@NicolasAG
upd Makefile
e8a5594
@NicolasAG
adding the basic files
8f33d10
@NicolasAG
update dependencies
02dfc57
@NicolasAG
start adding integration with wa_verified
48acaeb
@NicolasAG
upd readme
5be792d
@NicolasAG
use custom backend for webarena_verified
aae906c
@NicolasAG
pass the wa instance to the evaluator
2b04c7d
@NicolasAG
pass the wa instance to the evaluator
0d7e8dc
@NicolasAG
cleanup evaluator
b57c0f8
@NicolasAG
remove custom webarena verified instance
0330f72
@NicolasAG
update requirements to latest wav code
ab0437b
@NicolasAG
use simpler and cleaner wav eval
bd43467
@NicolasAG
enable tracing
fecedb1
@NicolasAG
fix wav
8fdebe6
@NicolasAG
update to new webarena verified version
4bdfa7e
@NicolasAG
update task name template to webarena_verified.templateID.taskID
e59f754
@NicolasAG
fix config
5b05044
@NicolasAG
fix csv file
56574eb
@NicolasAG
add webarena_verified backend
81f930c
@NicolasAG
fix wav tasks
cbca5a2
@NicolasAG
do not check reachable if url is todo
8d4381b
@NicolasAG
fix tmp trace creation, update goal to prompt model to satisfy wav re…
63b4b07 
…turn format/
@NicolasAG
create webarena_verified action space with special submit function to…
b7f847a 
… match the benchmark expected agent response format
@NicolasAG
look for extra header file path in environment variable
3e6b5b7
@NicolasAG
undo special action set for webarena_verified
525fd3b
@NicolasAG
remove wav actions
4272b5e
@NicolasAG
load extra context headers for webarena(+lite)
fea25ed
@NicolasAG
update README
fc090f0
@NicolasAG
update requirements
377dcca
amanjaiswal73892
amanjaiswal73892 reviewed Dec 4, 2025
View reviewed changes
amanjaiswal73892
amanjaiswal73892 reviewed Dec 8, 2025
View reviewed changes
amanjaiswal73892
amanjaiswal73892 reviewed Dec 8, 2025
View reviewed changes
@NicolasAG NicolasAG mentioned this pull request Dec 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amanjaiswal73892 amanjaiswal73892 amanjaiswal73892 left review comments

@ollmer ollmer Awaiting requested review from ollmer

@recursix recursix Awaiting requested review from recursix

At least 1 approving review is required to merge this pull request.

Assignees

@NicolasAG NicolasAG

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NicolasAG @amanjaiswal73892