Skip to content

feat: add Tavily Extract as configurable WebFetch backend alongside Jina #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
manisrinivasan2k1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add Tavily Extract as configurable WebFetch backend alongside Jina #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,5 @@ DEBUG_AGENT=false # Print verbose agent-l
DEBUG_SEARCH=false # Print verbose WebSearch debug logs.
DEBUG_SCHOLAR=false # Print verbose ScholarSearch debug logs.
DEBUG_VISIT=false # Print verbose WebFetch debug logs.
WEBFETCH_PROVIDER=jina # WebFetch backend: "jina" (default) or "tavily".
TAVILY_API_KEY="your_tavily_key" # https://tavily.com/ — required when WEBFETCH_PROVIDER=tavily.
52 changes: 45 additions & 7 deletions agent_base/tools/tool_web.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,13 +355,22 @@ def call(self, params: Union[str, dict], **kwargs) -> str:
except ValueError as exc:
return f"[WebFetch] {exc}"

response = self.readpage_jina(
url,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
runtime_deadline=runtime_deadline,
)
provider = os.getenv("WEBFETCH_PROVIDER", "jina").strip().lower()
if provider == "tavily":
response = self.readpage_tavily(
url,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
)
Comment on lines +360 to +365

@gemini-code-assist gemini-code-assist Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pass the runtime_deadline parameter to readpage_tavily to ensure consistency with the Jina backend and respect the agent's runtime budget.

Suggested change
response = self.readpage_tavily(
url,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
)
response = self.readpage_tavily(
url,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
runtime_deadline=runtime_deadline,
)

else:
response = self.readpage_jina(
url,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
runtime_deadline=runtime_deadline,
)

if visit_debug_enabled():
print(f"WebFetch Length {len(response)}")
Expand Down Expand Up @@ -435,6 +444,35 @@ def readpage_jina(
max_chars=max_chars,
)

def readpage_tavily(
self,
url: str,
*,
start_line: int = 1,
end_line: Optional[int] = None,
max_chars: int = DEFAULT_WEBFETCH_MAX_CHARS,
) -> str:
tavily_api_key = os.getenv("TAVILY_API_KEY", "").strip()
Comment on lines +447 to +455

@gemini-code-assist gemini-code-assist Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add the runtime_deadline parameter to readpage_tavily and check the remaining budget before proceeding with the Tavily API call. This ensures the tool respects the agent's runtime limit, matching the behavior of the Jina backend.

    def readpage_tavily(
        self,
        url: str,
        *,
        start_line: int = 1,
        end_line: Optional[int] = None,
        max_chars: int = DEFAULT_WEBFETCH_MAX_CHARS,
        runtime_deadline: Optional[float] = None,
    ) -> str:
        remaining = self._remaining_budget_seconds(runtime_deadline)
        if remaining is not None and remaining <= 0:
            return "[WebFetch] Failed to read page: agent runtime limit reached."
        tavily_api_key = os.getenv("TAVILY_API_KEY", "").strip()

if not tavily_api_key:
return "[WebFetch] TAVILY_API_KEY is not set."
try:
from tavily import TavilyClient
client = TavilyClient(api_key=tavily_api_key)
result = client.extract(urls=[url])
except Exception as exc:
return f"[WebFetch] Failed to read page: {exc}"
results = result.get("results", [])
if not results or not results[0].get("raw_content"):
return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format."
Comment on lines +464 to +466

@gemini-code-assist gemini-code-assist Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a check to ensure results[0] is a dictionary before calling .get() on it. If the API returns an unexpected structure or None in the results list, this prevents a potential AttributeError.

Suggested change
results = result.get("results", [])
if not results or not results[0].get("raw_content"):
return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format."
results = result.get("results", [])
if not results or not isinstance(results[0], dict) or not results[0].get("raw_content"):
return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format."

content = results[0]["raw_content"]
return self._format_page_content(
url=url,
content=content,
start_line=start_line,
end_line=end_line,
max_chars=max_chars,
)


def main(argv: Optional[list[str]] = None) -> int:
parser = argparse.ArgumentParser(description="Run web tools directly.")
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ openai>=2.3.0
Pillow>=11.3.0
requests>=2.32.5
structai>=0.1.23
tavily-python>=0.5.0
tiktoken>=0.12.0
uvicorn>=0.34.0